Title: Laplacian Canonization: A Minimalist Approach to Sign and Basis Invariant Spectral Embedding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Positional Encoding

Spectral Embedding

Laplacian Eigenvectors
Abstract: Spectral embedding is a powerful graph embedding technique that has received a lot of attention recently due to its effectiveness on Graph Transformers. However, from a theoretical perspective, the universal expressive power of spectral embedding comes at the price of losing two important invariance properties of graphs, sign and basis invariance, which also limits its effectiveness on graph data. To remedy this issue, many previous methods developed costly approaches to learn new invariants and suffer from high computation complexity. In this work, we explore a minimal approach that resolves the ambiguity issues by directly finding canonical directions for the eigenvectors, named Laplacian Canonization (LC). As a pure pre-processing method, LC is light-weighted and can be applied to any existing GNNs. We provide a thorough investigation, from theory to algorithm, on this approach, and discover an efficient algorithm named Maximal Axis Projection (MAP) that works for both sign and basis invariance and successfully canonizes more than 90\% of all eigenvectors. Experiments on real-world benchmark datasets like ZINC, MOLTOX21, and MOLPCBA show that MAP consistently outperforms existing methods while bringing minimal computation overhead. Code is available at https://github.com/PKU-ML/LaplacianCanonization.

Laplacian Canonization: A Minimalist Approach to Sign and Basis Invariant Spectral Embedding

Abstract

Spectral embedding is a powerful graph embedding technique that has received a lot of attention recently due to its effectiveness on Graph Transformers. However, from a theoretical perspective, the universal expressive power of spectral embedding comes at the price of losing two important invariance properties of graphs, sign and basis invariance, which also limits its effectiveness on graph data. To remedy this issue, many previous methods developed costly approaches to learn new invariants and suffer from high computation complexity. In this work, we explore a minimal approach that resolves the ambiguity issues by directly finding canonical directions for the eigenvectors, named Laplacian Canonization (LC). As a pure pre-processing method, LC is light-weighted and can be applied to any existing GNNs. We provide a thorough investigation, from theory to algorithm, on this approach, and discover an efficient algorithm named Maximal Axis Projection (MAP) that works for both sign and basis invariance and successfully canonizes more than 90% of all eigenvectors. Experiments on real-world benchmark datasets like ZINC, MOLTOX21, and MOLPCBA show that MAP consistently outperforms existing methods while bringing minimal computation overhead. Code is available at https://github.com/PKU-ML/LaplacianCanonization .

1 Introduction

Despite the popularity of Graph Neural Networks (GNNs) for graph representation learning [55, 56, 54, 27, 13], it is found that many existing GNNs have limited expressive power and cannot tell the difference between many non-isomorphic graphs [57, 35]. Existing approaches to improve expressive power, such as high-order GNNs [35] and subgraph aggregation [9], often incur significant computation costs over vanilla GNNs, limiting their use in practice. In comparison, a simple but effective strategy is to use discriminative node identifiers. For example, GNNs with random features can lead to universal expressive power [44, 1]. However, these unique node identifiers often lead to the loss of the permutation invariance property of GNNs, which is an important inductive bias of graph data that matters for sample complexity and generalization [24]. Therefore, during the pursuit of more expressive GNNs, we should also maintain the invariance properties of graph data. Balancing these two conflicting demands presents a significant challenge for the development of advanced GNNs.

Spectral embedding (SE) is a classical approach to encode node positions using eigenvectors U of the Laplacian matrix L , which has the advantage of being expressive and permutation equivariant.

GNNs using SE can attain universal expressive power (distinguishing any pair of non-isomorphic graphs) even under simple architectures (results in Section 2). However, spectral embedding faces two additional challenges in preserving graph invariance: sign ambiguity and basis ambiguity, due to the non-uniqueness of eigendecomposition. These ambiguities could lead to inconsistent predictions for the same graph under different eigendecompositions.

Many methods have been proposed to address sign and basis ambiguities of spectral embedding. A popular heuristic is RandSign [17] which randomly flips the signs of the eigenvectors during training. Although it is simple to use, this data augmentation approach does not offer any formal invariance guarantees and can lead to slower convergence due to all possible 2 n sign flips. An alternative involves using sign-invariant eigenfunctions to attain sign invariance [25], which significantly increases the time complexity to O ( n 4 ) . Another solution is to design extra GNN modules for sign and basis invariant embeddings, e.g. , SignNet and BasisNet [29], which can also add a substantial computational burden. Therefore, as summarized in Table 1, existing spectral embedding methods are all detrimental in a certain way that either hampers sign and basis invariance, or induces large computational overhead. More discussions about the related work can be found in Appendix B.

In this work, we explore a new approach called Laplacian Canonization (LC) that resolves the ambiguities by identifying a unique canonical direction for each eigenvector, amongst all its sign and basis equivalents. Although it is relatively easy to find a canonization rule that work for certain vectors, up to now, there still lacks a rigorous understanding of what kinds of vectors are canonizable and whether we could find a complete algorithm for all canonizable features. In this paper, we systematically answer this problem by developing a general theory for Laplacian canonization and characterizing the sufficient and necessary conditions for sign and basis canonizable features.

Based on these theoretical properties, we propose a practical canonization algorithm for sign and basis invariance, called Maximal Axis Projection (MAP), that adopts the permutation-invariant axis projection functions to determine the canonical directions. We theoretically characterize the conditions under which MAP can guarantee sign and basis canonization, and empirically verify that this condition holds for most synthetic and real-world graphs. It is worth noting that LC is a lightweight approach since it is only a pre-processing method and does not alter the dimension of the spectral embedding. Empirically, we show that employing the MAP-canonized spectral embedding yields significant improvements over the vanilla RandSign approach, and even matches SignNet on large-scale benchmark datasets like OGBG [20]. We summarize our contributions as follows:

We explore Laplacian Canonization (LC), a new approach to restoring the sign and basis invariance of spectral embeddings via determining the canonical direction of eigenvectors in the pre-processing stage. We develop a general theoretical framework for LC and characterize the canonizability of sign and basis invariance.

We propose an efficient algorithm for Laplacian Canonization, named Maximal Axis Projection (MAP), that works well for both sign and basis invariance. In particular, we show that MAP-sign is capable of canonizing all sign canonizable eigenvectors and thus is complete. The assessment of its feasibility shows that MAP can effectively canonize almost all eigenvectors on random graphs and more than 90% eigenvectors on real-world datasets.

We evaluate the MAP-canonized spectral embeddings on graph classification benchmarks including ZINC, MOLTOX21 and MOLPCBA, and obtain consistent improvements over previous spectral embedding methods while inducing the smallest computational overhead.

2 Benefits and Challenges of Spectral Embedding for GNNs

Denote a graph as G = ( V , E , X ) where V is the vertex set of size n , E is the edge set, and X ∈ R n × d are the input node features. We denote A as the adjacency matrix, and let ˆ A = D -1 2 ˜ AD -1 2 = D -1 2 ( I + A ) D -1 2 be the normalized adjacency matrix, where I denotes the augmented self-loop, D is the diagonal degree matrix of ˜ A defined by D i,i = ∑ n j =1 ˜ A i,j . A graph function f ([ X , ˆ A ]) is permutation invariant if for all permutation matrix P ∈ R n × n , we have f ([ PX , P ˆ AP ⊤ ]) = f ([ X , ˆ A ]) . Similarly, f is permutation equivariant if f ([ PX , P ˆ AP ⊤ ]) = P f ([ X , ˆ A ]) .

Spectral Embedding (SE). Considering the limited expressive power of MP-GNNs, recent works explore more flexible GNNs like graph Transformers [58, 62, 16, 25, 42]. These models bypass the explicit structural inductive bias in MP-GNNs while encoding graph structures via positional embedding (PE). A popular graph PE is spectral embedding (SE), which uses the eigenvectors U of the Laplacian matrix L = I -ˆ A with eigendecomposition L = UΛU ⊤ , where Λ = diag( λ ) is the diagonal matrix of ascending eigenvalues λ 1 ≤ · · · ≤ λ n , and the i -th column of U is the eigenvector corresponding to λ i . It is easy to see that spectral embedding is permutation equivariant: for any node permutation P of the graph, PU is the spectral embedding of the new Laplacian since PLP ⊤ = ( PU ) Λ ( PU ) ⊤ . Therefore, a permutation-invariant GNN ( e.g. , DeepSets [63], GIN [57], Graph Transformer [16]) using the SE-augmented input features ˜ X = [ X , U ] remains permutation invariant.

Reweighted Spectral Embedding (RSE). Previous works have shown that using spectral embedding improves the expressive power of MP-GNNs [6]. Nonetheless, we find that SE alone is insufficient to approximate an arbitrary graph function, as it does not contain all information about the graph, in particular, the eigenvalues. Consider two non-isomorphic graphs whose Laplacian matrices have identical eigenvectors but different eigenvalues. As their SE is the same, a network only using SE cannot tell them apart. To resolve this issue, we propose reweighted spectral embedding (RSE) that additionally reweights each eigenvector u i with the square-root of its corresponding eigenvalue λ i , i.e. , U RSE = UΛ 1 2 . With the reweighting technique, RSE incorporates eigenvalue information without need extra dimensions.

Universality of RSE. In the following theorem, we prove that with RSE, any universal network on sets ( e.g. , DeepSets [63] and Transformer [61]) is universal on graphs while preserving permutation invariance. All proofs are deferred to Appendix K.

Theorem 1. Let Ω ⊂ R n × d × R n × n be a compact set of graphs, [ X , ˆ A ] ∈ Ω . Let NN be a universal neural network on sets. Given any continuous invariant graph function f defined over Ω and arbitrary ε > 0 , there exist a set of NN parameters such that for all graphs [ X , ˆ A ] ∈ Ω ,

As far as we know, this theorem is the first to show that, with the help of graph embeddings like RSE, even an MLP network like DeepSets [63] (composed of a node-wise MLP, a global pooling layer, and a graph-level MLP) can achieve universal expressive power to distinguish any pair of non-isomorphic graphs. Notably, it does not violate the NP-hardness of graph isomorphism testing, since training NN itself is known to be NP-hard [10]. Actually, it is not always necessary to use all spectra of the graph. Existing studies find that high-frequency components are often unhelpful, or even harmful for representation learning [5, 19]. Thus, in practice, we only use the first k low-frequency components of RSE. An upper bound on the approximation error of truncated RSE can be found in Appendix K.10.

Ambiguities of eigenvectors. Although RSE enables universal GNNs, there exist two types of ambiguity in eigenvectors that hinder their applications. The first one, known as sign ambiguity , arises when an eigenvector u λ i corresponding to eigenvalue λ i is equally valid with its sign flipped, i.e. , -u λ i is also an eigenvector corresponding to λ i . The second one, termed basis ambiguity , occurs when eigenvalues with multiplicity degree d i > 1 can have any other orthogonal basis in the subspace spanned by the corresponding eigenvectors as valid eigenvectors. To be specific, for multiple eigenvalues with multiplicity degree d i > 1 , the corresponding eigenvectors U λ i ∈ R n × d i form an orthonormal basis of a subspace. Then any orthonormal matrix Q ∈ R d i × d i can be applied to U λ i to generate a new set of valid eigenvectors for λ i . Because of these ambiguities, we can get distinct GNN outputs for the same graph, resulting in unstable and suboptimal performance [17, 25, 18, 29]. In Appendix A, we elaborate the challenges posed by these ambiguities.

3 Laplacian Canonization for Sign and Basis Invariance

Rather than incorporating additional modules to learn new sign and basis invariants [25, 29], we explore a straightforward, learning-free approach named Laplacian Canonization (LC). The general idea of LC is to determine a unique direction for each eigenvector u among all its sign and basis equivalents. In this way, the ambiguities can be directly addressed in the pre-processing stage. For example, for two sign ambiguous vectors u and -u , we aim to find an algorithm that determines a unique direction among them to obtain sign invariance. Despite some naïve canonization rules (discussed in Appendix G.1), there still lacks a systematical understanding of the following key questions of LC:

What kind of canonization algorithm should we look for? (Section 3.1)

What kind of eigenvectors are canonizable or non-canonizable? (Section 3.2)

Is there an efficient canonization algorithm for all canonizable features? (Section 3.3)

In this section, we answer the three problems by establishing the first formal theory of Laplacian canonization and characterizing the canonizability for sign and basis invariance; based on these analyses, we also propose an efficient algorithm named MAP for LC that is guaranteed to canonize all sign canonizable features. To get a glimpse of our final results, Table 2 shows that MAP can resolve both sign and basis ambiguities for more than 90% of all eigenvectors on real-world data.

3.1 Definition of Canonization

To begin with, we first find out what properties a desirable canonization algorithm should satisfy. The ultimate goal of canonization is to eliminate ambiguities by selecting a canonical form among these ambiguous outputs. Generally speaking, we can characterize ambiguity as a multivalued function f : X → Y , i.e. , there could be multiple outputs y 1 , . . . , y n corresponding to the same input x . In our sign/basis ambiguity case, f refers to a mapping from a graph x ∈ X to the eigenvectors of a certain eigenvalue via the ambiguous eigendecomposition. The ambiguous eigenvectors are not independent; they are related by a sign/basis transformation. In general, we can assume that all possible outputs y 1 , . . . , y n of any x belong to the same equivalence class induced by some group action g ∈ G , where G acts on Y . That is, if f ( x ) = y 1 = y 2 , then there exists g ∈ G such that y 1 = gy 2 . Moreover, eigenvectors of graphs obey a fundamental symmetry: permutation equivariance. In general, we assume that f is equivariant to a group H acting on X and Y . That is, for any h ∈ H , if y 1 , . . . , y n are all possible outputs of x , then hy 1 , . . . , hy n are all possible outputs of hx .

Specifically, for the goal of canonizing ambiguous eigenvectors, i.e. , Laplacian canonization, we are interested in algorithms invariant to sign/basis transformations in the corresponding orthogonal group G ( O (1) for sign invariance and O ( d ) for basis invariance). Meanwhile, to preserve the symmetry of graph data, this algorithm should also maintain the permutation equivariant property of eigenvectors (Section 2) w.r.t. the permutation group H . Third, it also should still be discriminative enough to produce different canonical forms for different graphs (like the original spectral embedding). Combining these desiderata, we have a formal definition of canonization.

Definition 1. A mapping A : Y → Y is called a ( f, G, H )-canonization when it satisfies:

A is G -invariant : ∀ y ∈ Y , g ∈ G , A ( y ) = A ( gy ) ;

A is H -equivariant : ∀ x ∈ X , h ∈ H , A ( f ( hx ) ) = h A ( f ( x ) ) ;

̸

̸

A is universal : ∀ x ∈ X , h ∈ H , x = hx ⇒A ( f ( x ) ) = A ( f ( hx ) ) .

Accordingly, for any x ∈ X , if there exists a canonization A for x , we say x is canonizable .

3.2 Theoretical Properties of Canonization

Following Definition 1, we are further interested in the question that whether any eigenvector u is canonizable, i.e., , there exists a canonization that can determine its unique direction. Unfortunately, the answer is NO. For example, the vector u = (1 , -1) cannot be canonized by any canonization algorithm, since a permutation of u gives ( -1 , 1) that equals to -u 3 . But as long as there is only a small number of uncanonizable eigenvectors like u , a canonization algorithm can still resolve the ambiguity issue to a large extent.

Therefore, we are interested in the fundamental question of which eigenvectors are canonizable. The following theorem establishes a general necessary and sufficient condition of the canonizability for general groups G,H , which may be of independent interest.

̸

Theorem 2. An input x ∈ X is canonizable on the embedding function f iff there does not exist h ∈ H and g ∈ G such that x = hx and f ( hx ) = gf ( x ) .

This theorem states that for inputs from the same equivalence class in X (induced by H ), as long as they are not mapped to the same equivalence class in Y (induced by G ), these inputs are canonizable. In particular, by applying Theorem 2 to the specific group G induced by sign/basis invariance, we can derive some simple rules to determine whether there exists a canonizable rule for given eigenvector(s).

Corollary 1 (Sign canonizability) . A vector u ∈ R n is canonizable under sign ambiguity iff there does not exist a permutation matrix P ∈ R n × n such that u = -Pu .

̸

Corollary 2 (Basis canonizability) . The base eigenvectors U ∈ R n × d of the eigenspace V are canonizable under basis ambiguity iff there does not exist a permutation matrix P ∈ R n × n such that U = PU and span( PU ) = span( U ) = V .

Remark. From a probabilistic view, almost all eigenvectors are canonizable. Let U ∈ R n × d be basis vectors sampled from a continuous distribution in R n , then Pr { U is canonizable } = 1 .

In the next subsection, we will further design an efficient algorithm to canonize all sign and basis canonizable eigenvectors in the pre-processing stage, so the network does not have to bear the ambiguities of these eigenvectors.

3.3 MAP: A Practical Algorithm for Laplacian Canonization

Built upon theoretical properties in Section 3.2, we aim to design a general, powerful, and efficient canonization to resolve both sign and basis ambiguities for as many eigenvectors as possible. Here, we choose to adopt axis projection as the basic operator in our canonization algorithm named Maximal Axis Projection (MAP). The key observation is that the standard basis vectors ( i.e. , the axis) of the Euclidean space e i ∈ R n are permutation equivariant, and in the meantime, the eigenspace spanned by the eigenvectors V = span( U ) are also permutation equivariant. This means that when projecting the axis to the eigenspace, the obtained angles are also permutation equivariant, based on which we could apply permutation invariant functions (such as max ) to obtain permutation invariant statistics that can be used for canonization. Meanwhile, due to the generality of projection, it can be applied to both sign ambiguity (for a single eigenvector) and basis ambiguity (for the eigenspace with multiple eigenvectors). We provide illustrative examples to help understand this algorithm in Appendix F.

Preparation step: Axis projection. Consider unit eigenvector(s) U ∈ R n × d corresponding to an eigenvalue λ with geometric multiplicity d ≥ 1 . These eigenvectors span a d -dimensional eigenspace V = span( U ) with the projection matrix P = UU ⊤ . To start with, we calculate the projected angle between V and each standard basis ( i.e. , the axis) of the Euclidean space e i (a one-hot vector whose i -th element is 1 ), i.e. , α i = | Pe i | , i = 1 , . . . , n . Assume that there are k distinct values in { α i , i = 1 , . . . , n } , according to which we can divide all basis vectors { e i } into k disjoint groups B i (arranged in descending order of the distinct angles). Each B i represents an equivalent class of axes that u i has the same projection on. Then, we define a summary vector x i for the axes in each group B i as their total sum x i = ∑ e j ∈B i e j + c , where c is a tunable constant. Next, we introduce how to adopt these summary vectors to canonize the eigenvectors for sign and basis invariance, respectively.

3.3.1 Sign Canonization with MAP

Step 1. Find non-orthogonal axis. For sign canonization, we calculate the angles between the eigenvalue u and each summary vector x i one by one, and terminate the procedure as long as we find a summary vector x h with non-zero angle α h = u ⊤ x h = 0 , and return <none> otherwise.

̸

̸

̸

̸

Assumption 1. There exists a summary vector x h that is non-orthogonal to u , i.e., u ⊤ x h = 0 ,

Step 2. Sign canonization. As long as Assumption 1 holds, we can utilize x h to determine the canonical direction u ∗ by requiring its projected angle to be positive, i.e. ,

We call this algorithm MAP-sign, and summarize it in Algorithm 1 in Appendix C.1. The following theorem proves that it yields a valid canonization for sign invariance under Assumption 1.

Theorem 3. Under Assumption 1, our MAP-sign algorithm gives a sign-invariant, permutationequivariant and universal canonization of u .

One would wonder how restrictive the non-orthogonality condition (Assumption 1) is for sign canonization. In the following theorem, we establish a strong result showing that sign canonizability is equivalent to non-orthogonality. In other words, any sign canonizable eigenvectors can be canonized by MAP-sign . Due to this equivalence, MAP-sign can also serve as a complete algorithm to determine sign canonizability: a vector u is sign canonizable iff x h is not <none> in equation 1.

̸

Theorem 4. A vector u ∈ R n is sign canonizable iff there exists a summary vector x h s.t. u ⊤ x h = 0 .

Remark. Theorem 3 is proved in Appendix K.5 and experimentally verified in Appendix C.2. We also show that MAP is not the only complete canonization algorithm for sign sinvariance by proposing another polynomial-based algorithm in Appendix K.9. We observe that most eigenvectors in real-world datasets are sign canonizable. For instance, on the ogbg-molhiv dataset, the ratio of non-canonizable eigenvectors is only 2.8%. A thorough discussion on the feasibilty of Assumption 1 is in Appendix G.1. For the left non-canonizable eigenvectors, we could apply previous methods (like RandSign, SAN, and SignNet) to further eliminate their ambiguity, which could save more compute since there are only a few non-canonizable eigenvectors. In practice, we also obtain good performance by simply ignoring these non-canonizable features.

3.3.2 Basis Canonization with MAP

Wefurther extend MAP-sign to solve the more challenging basis ambiguity with multiple eigenvectors, named MAP-basis. Now, MAP relies on two conditions to produce canonical eigenvectors. The first one requires there are enough distinctive summary vectors to determine d canonical eigenvectors.

Assumption 2. The number of distinctive angles k (i.e., the number of summary vectors { x i } ) is larger or equal to the multiplicity d , i.e., k ≥ d .

Under this condition, we can determine each u i iteratively with the corresponding summary vector x i . At the i -th step where u 1 , . . . , u i -1 have already been determined, we choose u i to be the vector that is closest to the summary vector x i in the orthogonal complement space of ⟨ u 1 , . . . , u i -1 ⟩ in V :

With a compact feasibility region, the maximum is attainable, and we can further show that the solution u i is unique ( c.f. , the proof of Theorem 5). Equation 3 can be directly solved using the projection matrix of ⟨ u 1 , . . . , u i -1 ⟩ ⊥ (see details in Algorithm 3 in Appendix C.1). Repeating this process gives us a canonical basis of V . We also require non-orthogonality condition at each step to obtain a valid eigenvector.

Assumption 3. For any 1 ≤ i ≤ d , x i is not perpendicular to ⟨ u 1 , . . . , u i -1 ⟩ ⊥ .

We summarize MAP-basis in Algorithm 2 in Appendix C.1. In Theorem 5, we prove under Assumption 2 and Assumption 3, MAP-basis gives a basis-invariant, permutation-equivariant and universal canonization of U . Though, in this scenario, MAP-basis cannot canonize all basis canonizable features, and whether such an algorithm exists remains an open problem for future research.

Theorem 5. Under Assumption 2 and Assumption 3, our MAP-basis algorithm gives a basis-invariant, permutation-equivariant and universal canonization of U .

Remark. Assumption 2 and Assumption 3 exclude some symmetries of the eigenspace, which we discuss in details in Appendix G.2. For random orthonormal matrices, the possibility that either assumption is violated is equal to 0 , which we verify with random simulation in Appendix C.3. For real-world datasets, these assumptions are not restrictive either. For instance, on the large ogbgmolpcba dataset, the eigenvalues violating Assumption 2 or Assumption 3 make up 0.87% of all eigenvalues. More statistics and discussions are provided in Appendix G.2.

3.3.3 Summary

We provide the complete pseudo-code of MAP to address both sign and basis invariance in Appendix C.1, along with a detailed time complexity analysis. Overall, the extra complexity introduced by MAP is O ( n 2 log n ) , which is better than eigendecomposition itself with O ( n 3 ) . It only needs to be computed once per dataset and can be easily incorporated with various GNN architectures.

Combining Theorem 1, Theorem 3 and Theorem 5 gives us the universality of first-order GNNs [33] such that it respects all graph symmetries under certain assumptions (Assumptions 1, 2 & 3). We show that the probability of violating these assumptions asymptotically converges to zero on random graphs (see Appendix H). We also count the ratio of violation in real-world datasets in Table 2, and observe that the ratio of uncanonizable eigenvectors is less than 10% on all datasets. Thus, MAP greatly eases the harm caused by sign and basis ambiguities.

So far only GNNs using higher-order tensors have achieved universality while respecting all graph symmetries [33, 24], but they are typically computationally prohibitive in practice. It is still an open problem whether it is also possible for first-order GNNs. The proposed Laplacian Canonization presents a new approach in this direction trying to alleviate the harm of sign and basis ambiguities in a minimalist approach. By establishing universality-invariance results for first-order GNNs under certain assumptions, LC could hopefully could bring some insights to the GNN community.

4 Experiments

We evaluate the proposed MAP positional encoding on sparse MP-GNNs and Transformer GNNs using PyTorch [40] and DGL [53]. For sparse MP-GNNs we consider GatedGCN [11] and PNA [15], and for Transformer GNNs we consider SAN [25] and GraphiT [34]. We conduct experiments on three standard molecular benchmarks-ZINC [22], OGBG-MOLTOX21 and OGBG-MOLPCBA [20]. ZINC and MOLTOX21 are of medium scale with 12K and 7.8K graphs respectively, whereas MOLPCBA is of large scale with 437.9K graphs. Details about these datasets are described in Appendix L. We follow the same protocol as in Lim et al. [29], where we replace their SignNet with a normal GNN and the Laplacian eigenvectors with our proposed MAP. We fairly compare several models on a fixed number of 500K model parameters on ZINC and relax the model sizes to larger parameters for evaluation on the two OGB datasets, as being practised on their leaderboards [20]. We also compare our results with GNNs using LapPE and random sign (RS) [17], GNNs using SignNet [29], and GNNs with no PE. For a fair comparison, all models use a limited number k of eigenvectors for positional encodings. The results of all our experiments are presented in Table 3, 4 & 5. Further implementation details are included in Appendix M.

4.1 Performance on Benchmark Datasets

As shown in Table 3, 4 & 5, using MAP improves the performance of all GNNs on all datasets, demonstrating that removing ambiguities of the eigenvectors is beneficial for graph-level tasks. First, by comparing models with LapPE and RS with models with no PE, we observe that the use of LapPE significantly improves the performance on ZINC, showing the benefits of incorporating expressive

PEs with GNNs, especially with MP-GNNs whose expressive power is limited by the 1-WL test. However, on MOLTOX21 and MOLPCBA, using LapPE has no significant effects. This is because unlike ZINC, OGB-MOL* datasets contain additional structural features that are informative, e.g. , if an atom is in ring, among others [18]. Thus the performance gain by providing more positional information is less obvious. Second, MAP outperms LapPE with RS by a large margin especially on ZINC. Although RS also alleviates sign ambiguity by randomly flipping signs during training, MAP removes such ambiguity before training, enabling the network to focus on the real meaningful features and achieves a better performance. Third, we also observe that MAP and SignNet achieve comparable performance. This is because both methods aim at the same goal-eliminating ambiguity. However, SignNet does so in the training stage while MAP does so in the pre-processing stage, thus the latter is more computationally efficient. Lastly, we would also like to highlight that as a kind of positional encoding, MAP can be easily incorporated with any GNN architecture by passing the pre_transform function to the dataset class with a single line of code.

4.2 Empirical Understandings

Computation time. We demonstrate the efficiency of MAP by measuring the pre-processing time and training time on the large OGBG-MOLPCBA dataset, and compare them with SignNet. For a fair comparison, we use identical model size, hyperparameters and random seed and conduct experiments on the same NVIDIA 3090 GPU. The results are shown in Table 6. We observe that model with MAP train 41% faster than its SignNet counterpart, saving 44 hours of training time. Since SignNet takes the form ρ ( ϕ ( u ) + ϕ ( -u ) ) while we use models taking the form ρ ( ϕ ( u ) ) , models with MAP would always train faster than those with SignNet under the same hyperparameters. We also observe that the pre-processing time is negligible compared with training time (< 3%), since pre-processing only needs to be done once. This makes MAP overall more efficient while achieving the same goal of tackling ambiguities.

Spectral embedding dimension. Next, we study the effects of k . We train GatedGCN with MAP on ZINC with different number of eigenvectors used in the PE and report the results in Table 7. The hyperparameters are the same across these experiments. It can be observed that the performance drops when k is too small, meaning that SE provides crucial structural information to the model. Using larger k has limited influence on the performance, meaning that the model relies more on low-frequency information of spectral embedding.

Ablation study. Finally, we conduct ablation study of MAP by removing each component of MAP and evaluate the performance. The results are shown in Table 8. Removing sign invariance hurts the performance the most, because most eigenvectors are single and thus sign ambiguity has the most influence on the model's performance. Removing basis invariance also has negative effects since a small portion of eigenvalues are multiple. Removing eigenvalues has moderate negative effects, showing that the incorporation of eigenvalue information is beneficial for the network.

5 Conclusion

In this paper, we explored a new approach called Laplacian Canonization for addressing sign and basis ambiguities of Laplacian eigenvectors while also preserving permutation invariance and universal expressive power. We developed a novel theoretical framework to characterize canonization and the canonizability of eigenvectors. Then we proposed a practical canonization algorithm called Maximal Axis Projection (MAP). Theoretically, it is sign/basis-invariant, permutation-equivariant and universal. Empirically, it canonizes most eigenvectors on synthetic and real-world data while showing promising performance on various datasets and GNN architectures.

References

Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The Surprising Power of Graph Neural Networks with Random Node Initialization. In IJCAI , 2021.

Rohith Agaram, Shaurya Dewan, Rahul Sajnani, Adrien Poulenard, Madhava Krishna, and Srinath Sridhar. Canonical Fields: Self-Supervised Learning of Pose-Canonicalized Neural Fields. arXiv preprint arXiv:2212.02493 , 2022.

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD , 2019.

Greg W Anderson, Alice Guionnet, and Ofer Zeitouni. An Introduction to Random Matrices . Cambridge university press, 2010.

Muhammet Balcilar, Renton Guillaume, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the Expressive Power of Graph Neural Networks in a Spectral Perspective. In ICLR , 2021.

Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional Graph Networks. In ICML , 2021.

Mikhail Belkin and Partha Niyogi. Laplacian Eigenmaps for Dimensionality Reduction and Data Representation. Neural Computation , 15(6):1373-1396, 2003.

Florent Benaych-Georges and Raj Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Journal of Multivariate Analysis , 111: 120-135, 2012.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M Bronstein, and Haggai Maron. Equivariant Subgraph Aggregation Networks. In ICLR , 2022.

Avrim Blum and Ronald Rivest. Training a 3-Node Neural Network is NP-Complete. In NeurIPS , 1988.

Xavier Bresson and Thomas Laurent. Residual Gated Graph ConvNets. arXiv preprint arXiv:1711.07553 , 2017.

Rasmus Bro, Evrim Acar, and Tamara G Kolda. Resolving the Sign Ambiguity in the Singular Value Decomposition. Journal of Chemometrics: A Journal of the Chemometrics Society , 22(2): 135-140, 2008.

Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-Induced Graph Implicit Nonlinear Diffusion. In ICML , 2022.

Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with GNNs. In NeurIPS , 2019.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal Neighbourhood Aggregation for Graph Nets. In NeurIPS , 2020.

Vijay Prakash Dwivedi and Xavier Bresson. A Generalization of Transformer Networks to Graphs. In AAAI Workshop on Deep Learning on Graphs: Methods and Applications , 2021.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking Graph Neural Networks. arXiv preprint arXiv:2003.00982 , 2020.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph Neural Networks with Learnable Structural and Positional Representations. In ICLR , 2022.

NT Hoang and Takanori Maehara. Frequency Analysis for Graph Convolution Network. In ICLR , 2020.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. arXiv preprint arXiv:2005.00687 , 2020.

Yinan Huang, William Lu, Joshua Robinson, Yu Yang, Muhan Zhang, Stefanie Jegelka, and Pan Li. On the Stability of Expressive Positional Encodings for Graph Neural Networks. arXiv preprint arXiv:2310.02579 , 2023.

John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. ZINC: A Free Tool to Discover Chemistry for Biology. Journal of chemical information and modeling , 52(7):1757-1768, 2012.

Sékou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equivariance with Learned Canonicalization Functions. In NeurIPS Workshop on Symmetry and Geometry in Neural Representations , 2022.

Nicolas Keriven and Gabriel Peyré. Universal Invariant and Equivariant Graph Neural Networks. In NeurIPS , 2019.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In NeurIPS , 2021.

Rongjie Lai and Hongkai Zhao. Multiscale Nonrigid Point Cloud Registration Using RotationInvariant Sliced-Wasserstein Distance via Laplace-Beltrami Eigenmap. SIAM Journal on Imaging Sciences , 10(2):449-483, 2017.

Mingjie Li, Xiaojun Guo, Yifei Wang, Yisen Wang, and Zhouchen Lin. G2CN: Graph Gaussian Convolution Networks with Concentrated Graph Filters. In ICML , 2022.

Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance Encoding-Design Provably More Powerful GNNs for Structural Representation Learning. In NeurIPS , 2020.

Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and Basis Invariant Networks for Spectral Graph Representation Learning. In ICLR , 2023.

Andreas Loukas. What Graph Neural Networks Cannot Learn: Depth vs Width. In ICLR , 2020.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably Powerful Graph Networks. In NeurIPS , 2019.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and Equivariant Graph Networks. In ICLR , 2019.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the Universality of Invariant Networks. In ICML , 2019.

Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal. GraphiT: Encoding Graph Structure in Transformers. arXiv preprint arXiv:2106.05667 , 2021.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks. In AAAI , 2019.

Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational Pooling for Graph Representations. In ICML , 2019.

Sean O'Rourke, Van Vu, and Ke Wang. Eigenvectors of random matrices: A survey. Journal of Combinatorial Theory, Series A , 144:361-442, 2016.

Sean O'Rourke, Van Vu, and Ke Wang. Random perturbation of low rank matrices: Improving classical bounds. Linear Algebra and its Applications , 540:26-59, 2018.

Richard S Palais and Chuu-Lian Terng. A General Theory of Canonical Forms. In Transactions of the American Mathematical Society , 1987.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS , 2019.

Omri Puny, Heli Ben-Hamu, and Yaron Lipman. Global Attention Improves Graph Networks Generalization. arXiv preprint arXiv:2006.07846 , 2020.

Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a General, Powerful, Scalable Graph Transformer. In NeurIPS , 2022.

Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas J Guibas, and Srinath Sridhar. ConDor: Self-Supervised Canonicalization of 3D Pose for Partial Shapes. In CVPR , 2022.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random Features Strengthen Graph Neural Networks. In SDM , 2021.

Balasubramaniam Srinivasan and Bruno Ribeiro. On the Equivalence between Positional Node Embeddings and Structural Graph Representations. In ICLR , 2020.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The Journal of Machine Learning Research , 15:1929-1958, 2014.

Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey Hinton, and Kwang Moo Yi. Canonical Capsules: Self-Supervised Capsules in Canonical Pose. In NeurIPS , 2021.

Edric Tam and David Dunson. Multiscale Graph Comparison via the Embedded Laplacian Discrepancy. arXiv preprint arXiv:2201.12064 , 2022.

Terence Tao and Van Vu. Random matrices: Universal properties of eigenvectors. Random Matrices: Theory and Applications , 1(01):1150001, 2012.

Terence Tao and Van Vu. Random matrices have simple spectrum. Combinatorica , 37:539-553, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS , 2017.

Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks. In ICLR , 2022.

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the Diffusion Process in Linear Graph Convolutional Networks. In NeurIPS , 2021.

Max Welling and Thomas N Kipf. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR , 2017.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying Graph Convolutional Networks. In ICML , 2019.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In ICLR , 2019.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do Transformers Really Perform Badly for Graph Representation? In NeurIPS , 2019.

Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware Graph Neural Networks. In ICML , 2019.

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware Graph Neural Networks. In AAAI , 2021.

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are Transformers universal approximators of sequence-to-sequence functions? In ICLR , 2020.

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph Transformer Networks. In NeurIPS , 2019.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In NeurIPS , 2017.

Appendix

Table of Contents

A Why sign and basis invariance matters

It is perfectly understandable that one may not see the significance and benefits of sign and basis invariance at once. For instance, below is a quote from one of the reviews of Lim et al. [29] when they first submitted their paper to NeurIPS 2022:

It is not clear 'why' to preserve the two symmetries this paper proposes to preserve including sign invariance and basis invariance. In graph & molecule setting, two important symmetries are permutation ( i.e. if we permute the nodes/atoms, the output stays the same) and rotation ( i.e. if we rotate the 3D coordinates associating with each atom, the prediction for a physical property stays the same). I don't see a strong evidence that suggests preserving sign & basis invariances on top of permutation & rotation invariances brings any benefits.

Indeed, it is not easy to see why issues related with something matters if they don't even use it. One may overlook the importance of sign and basis invariance for two reasons: (1) people not familiar with Laplacian PE may not even realize sign/basis ambiguity is a thing; (2) people that does have some knowledge about Laplacian PE may not realize sign and basis ambiguities are markedly hindering its performance. Nonetheless, Laplacian PE does have advantages over other commonly used PEs that it is worth the efforts addressing their drawbacks. For these reasons we feel it is of great significance that we explain the motivation behind preserving sign and basis invariances in more details.

Why use spectral embedding: trade-off between universality and ambiguities. Sign and basis ambiguities arise only in spectral embeddings, so the first important question is why to use them. There are many kinds of positional encodings in the literature, such as Random Walk PE (RWPE), Position-aware Encoding, Relational Pooling (RP), random features, etc . Without addressing sign and basis ambiguities, some works even reported superior performance of RWPE over spectral embedding [18, 42]. However, these other PEs all have drawbacks either in their expressive power or permutation equivariance. RP and random features are universal, but they do not guarantee permutation equivariance, which is an important inductive bias in graph-structured data. RWPE and positional-aware encoding are permutation equivariant, but their expressive power is limited. Thus one may wonder whether there exists a kind of positional encoding method that achieves both. Spectral Embedding, as it turns out, is both permutation equivariant and universal (when coupled with eigenvalue information, see Sec 2). The good thing about SE is that through eigendecomposition, it transforms a 'harder' permutation equivariance formulated by

into a 'easier' permutation equivariance formulated by

where P is an arbitrary permutation matrix, ˆ A is the normalized adjacency matrix of the graph and U are its eigenvectors. Universality and equivariant results for equation 4 are only known for high-order tensor networks [32], but similar results for equation 5 have long been known to be achievable even by first-order networks such as DeepSets [63]. Thus SE enables efficient and universal graph neural networks that respects permutation equivariance. However, SE also brings new problems, sign and basis ambiguities. It is not fair to say models with SE are universal without achieving sign and basis invariance, since we expect graph neural networks to respect all symmetries among isomorphic graphs.

Sign and basis invariances are well-established problems in the literature. Sign and basis ambiguities have been recognized by numerous works in the literature to be challenging and important issues when incorporating SE as positional encoding. Below we list some existing discussions in these papers.

Quote from Dwivedi et al. [17]:

We propose an alternative to reduce the sampling space, and therefore the amount of ambiguities to be resolved by the network. Laplacian eigenvectors are hybrid positional and structural encodings, as they are invariant by node reparametrization. However, they are also limited by natural symmetries such as the arbitrary sign of eigenvectors (after being normalized to have unit length). The

number of possible sign flips is 2 k , where k is the number of eigenvectors. In practice we choose k ≪ n , and therefore 2 k is much smaller than n ! (the number of possible ordering of the nodes). During the training, the eigenvectors will be uniformly sampled at random between the 2 k possibilities. If we do not seek to learn the invariance w.r.t. all possible sign flips of eigenvectors, then we can remove the sign ambiguity of eigenvectors by taking the absolute value. This choice seriously degrades the expressivity power of the positional features.

Quote from Kreuzer et al. [25]:

As noted earlier, there is a sign ambiguity with the eigenvectors. With the sign of ϕ being independent of its normalization, we are left with a total of 2 k possible combination of signs when choosing k eigenvectors of a graph. Previous work has proposed to do data augmentation by randomly flipping the sign of the eigenvectors [6, 16, 17], and although it can work when k is small, it becomes intractable for large k .

Quote from Beaini et al. [6]:

For instance, a pair of eigenvalues can have a multiplicity of 2 meaning that they can be generated by different pairs of orthogonal eigenvectors. For an eigenvalue of multiplicity 1, there are always two unit norm eigenvectors of opposite sign, which poses a problem during the directional aggregation. We can make a choice of sign and later take the absolute value. An alternative is to take a sample of orthonormal basis of the eigenspace and use each choice to augment the training.

Quote from Mialon et al. [34]:

Note that eigenvectors of the Laplacian computed on different graphs could not be compared to each other in principle, and are also only defined up to a ± 1 factor. While this raises a conceptual issue for using them in an absolute positional encoding scheme, it is shown in Dwivedi and Bresson [16]-and confirmed in our experiments-that the issue is mitigated by the Fourier interpretation, and that the coordinates used in LapPE are effective in practice for discriminating between nodes in the same way as the position encoding proposed in Vaswani et al. [51] for sequences. Yet, because the eigenvectors are defined up to a ± 1 factor, the sign of the encodings needs to be randomly flipped during the training of the network.

Quote from Dwivedi and Bresson [16]:

In particular, Dwivedi et al. [17] make the use of available graph structure to precompute Laplacian eigenvectors [7] and use them as node positional information. Since Laplacian PEs are generalization of the PE used in the original transformers [51] to graphs and these better help encode distance-aware information ( i.e. , nearby nodes have similar positional features and farther nodes have dissimilar positional features), we use Laplacian eigenvectors as PE in Graph Transformer. Although these eigenvectors have multiplicity occuring due to the arbitrary sign of eigenvectors, we randomly flip the sign of the eigenvectors during training, following Dwivedi et al. [17].

Quote from Dwivedi et al. [18]:

Another PE candidate for graphs can be Laplacian Eigenvectors [17, 16] as they form a meaningful local coordinate system, while preserving the global graph structure. However, there exists sign ambiguity in such PE as eigenvectors are defined up to ± 1 , leading to 2 k number of possible sign values when selecting k eigenvectors which a network needs to learn. Similarly, the eigenvectors may be unstable due to eigenvalue multiplicities.

Quote from Lim et al. [29]:

However, there are nontrivial symmetries that should be accounted for when processing eigenvectors. If v is a unit-norm eigenvector, then so is -v , with the same eigenvalue. More generally, if an eigenvalue has higher multiplicity, then there are infinitely many unit-norm eigenvectors that can be chosen. Indeed, a full set of orthonormal eigenvectors is only defined up to a change of basis in each eigenspace. In the case of sign invariance, for any k eigenvectors there are 2 k possible choices of sign. Accordingly, prior works randomly flip eigenvector signs during training in order to approximately learn sign invariance [25, 17]. However, learning all 2 k invariances is challenging and limits the effectiveness of Laplacian eigenvectors for encoding positional information. Sign invariance is a special case of basis invariance when all eigenvalues are distinct, but higher dimensional basis invariance is even more difficult to deal with, and we show that these higher dimensional eigenspaces are abundant in real datasets.

Quote from Wang et al. [52]:

Srinivasan and Ribeiro [45] states that PE using the eigenvectors of the randomly permuted graph Laplacian matrix keeps permutation equivariant. Dwivedi et al. [17], Kreuzer et al. [25] argue that such eigenvectors are unique up to their signs and thus propose PE that randomly perturbs the signs of those eigenvectors. Unfortunately, these methods may have risks. They cannot provide permutation equivariant GNNs when the matrix has multiple eigenvalues, which thus are dangerous when applying to many practical networks. For example, large social networks, when not connected, have multiple 0 eigenvalues; small molecule networks often have non-trivial automorphism that may give multiple eigenvalues. Even if the eigenvalues are distinct, these methods are unstable. We prove that the sensitivity of node representations to the graph perturbation depends on the inverse of the smallest gap between two consecutive eigenvalues, which could be actually large when two eigenvalues are close.

Sign and basis invariances make the learning tasks easier. Permutation invariance is an important inductive bias in graphs in the sense that applying permutations on graphs results in isomorphic graphs and they should have the same properties and produce the same outputs. Empirically constraining the network to be invariant to such permutations benefits the performance, since otherwise the network has to learn these n ! ambiguities by itself. Similarly, sign and basis ambiguities result in 2 k and infinite possible choices of positional features respectively that a network has to learn, which could degrade its performance greatly. By preserving sign and basis invariance, a network will not need to learn the equivalence between all these possible choices and the learning task is significantly easier.

There are less ambiguity in signs than in permutations. For a given graph, there are n ! possible ordering of the nodes that all represent the same graph, resulting in n ! number of ambiguities. Considering that the vast majority of eigenvectors on real-world data have distinct eigenvalue (Table 9), when selecting k eigenvectors as positional encoding to present structural information, the number of ambiguities is 2 k , which is much smaller than n ! . This transformation from permutation ambiguity to sign ambiguity greatly reduces the amount of random noise in the input data and makes the model much more stable.

When sign invariance meets permutation equivariance. Sign ambiguity is not an issue when we do not take permutation symmetry into account. For each eigenvector pair ± u , we can choose the one whose first non-zero entry is positive. This simple canonization solves the sign ambiguity problem with ease. The problem of sign ambiguity is only non-trivial when combined with permutation equivariance, that is, we require the canonization process to be both sign-invariant and permutationequivariant. Since the eigenvectors can now be permuted arbitrarily, there is no 'first' non-zero entry and the simple canonization above fails.

Moreover, such canonization only makes sense when we require it to be universal . For instance, mapping all eigenvectors to j = (1 , 1 , . . . , 1) is also a sign-invariant and permutation-equivariant canonization, but this canonization provides us with no information. Thus one may wonder whether a network can be both sign-invariant, permutation-equivariant and universal at the same time. Unfortunately, as far as we know, no existing work addresses all of them. The universality of SignNet only considers sign invariance, as stated in their theorem:

Theorem 6 (Universal representation for SignNet) . A continuous function f : ( S n -1 ) k → R s is sign invariant, i.e. f ( s 1 v 1 , . . . , s k v k ) = f ( v 1 , . . . , v k ) for any s i ∈ {-1 , 1 } , if and only if there exists a continuous ϕ : R n → R 2 n -2 and a continuous ρ : R (2 n -2) k → R s such that

In fact, being sign-invariant, permutation-equivariant and universal at the same time is not even well-defined . As mentioned in the main text, for an uncanonizable eigenvector u with u = -Pu for some permutation matrix P , let f be an arbitrary continuous sign-invariant, permutation-equivariant and universal function and denote f ( u ) = y . The sign invariance property requires that f ( -u ) = y , while the permutation equivariance property requires that f ( Pu ) = Py . Since -u = Pu , we have y = Py . However u = Pu , thus universality is not satisfied, leading to a contradiction. This can be illustrated in Figure 1, where the three properties cannot be met at the same time.

sign invariance

permutation equivariance

universality

SignNet

MLP

MAP

Remark. The universality in Definition 1 is defined on eigenvectors U ∈ R n × n . The permutation equivariance property of eigenvectors is defined as f ( PU ) = P f ( U ) , and here universality implies that if two eigenvectors are not equal up to permutation, then f should be able to tell them apart. As mentioned above, such set-universal networks (like DeepSets or MLP) cannot be permutation equivariant and sign invariant at the same time. In the context of graphs, we may also care about universality defined on the adjacency matrix A ∈ R n × n , in which case the permutation equivariance property is defined as f ( PAP ⊤ ) = P f ( A ) . Such graph-universal networks are always able to tell two non-isomorphic adjacency matrices apart. We can view f ( A ) as a function of U (we omit the eigenvalues for simplicity), and we denote g ( U ) = f ( A ) , then to achieve the graph universality of f , g does not have to be set-universal, thus may not suffer from the dilemma above.

Much effort has been devoted to the structure of invariant and equivariant networks in the literature, but few has studied the case when invariance and equivariance combine (in this case you may have to sacrifice some invariance, or some equivariance, or some expressive power). Our work takes a step in this direction, providing some insights to this dilemma and characterize the necessary and sufficient conditions where these three properties conflict with each other.

B Related work

Theoretical Expressivity It has been shown that neighborhood-aggregation based GNNs are not more powerful than the WL test [57, 35]. Since then, many attempts have been made to endow graph neural networks with expressive power beyond the 1-WL test, such as injecting random attributes [36, 44, 59, 45, 17], injecting positional encodings [28, 60], and designing higher-order GNNs [35, 32, 33, 31, 14].

Positional Encodings Recently, there are many works that aim to improve the expressive power of GNNs by positional encoding (PE). Murphy et al. [36] assigned each node an identifier depending on its index. Li et al. [28] proposed to use distances between nodes characterized by powers of the random walk matrix. You et al. [59] proposed to use distances of nodes to a sampled anchor set of nodes. Using random node features can also improve the expressiveness of GNNs, even making them universal [44, 1], but it has several defects: (1) loss of permutation invariance, (2) slower convergence, (3) poor generalization on unseen graphs [59, 30]. More detailed discussion can be found in Appendix E.

Laplacian PE Another PE candidate for graphs is Laplacian Eigenvectors [16, 17] which form a meaningful local coordinate system while preserving global structure. However, there exists

̸

ambiguities in sign and basis that the network needs to learn, which harms the performance of GNNs. To address these ambiguities, many approaches have been proposed such as randomly flip the signs [17], use eigenfunctions over the edges [25] or use invariant network architectures [29].

Sign Invariance Several prior works have proposed ways of addressing sign ambiguity. A popular heuristic is Random Sign (RS), which randomly flips the signs of eigenvectors during training to encourage the insensitivity to different signs [17]. However, the network still has to learn these signs, leading to slower convergence and harder curve fitting task. Bro et al. [12] developed a datadependent method to choose signs for each singular vector of a singular value decomposition. Still, in the worst case the signs chosen will be arbitrary, and they do not handle rotational ambiguities in higher dimensional eigenspaces. Kreuzer et al. [25] proposed to use the relative Laplace embeddings of two nodes. However, their approach suffers from a major computational bottleneck with O ( n 4 ) complexity. Lim et al. [29] proposed SignNet that passes both u and -u to the same network, adds the outputs together and then passes them to another network. As both outputs of u and -u need to be computed, their approach adds to the training cost. Our approach happens at pre-processing time with O ( n 3 ) complexity.

Basis Invariance The only works we know addressing basis ambiguity are Lim et al. [29] and Huang et al. [21]. Lim et al. [29] proposed BasisNet that is basis invariant. However, to achieve universality, the components of BasisNet need to use higher order tensors in R n k where k can be as large as n ( n -1) 2 [33], rendering BasisNet impractical. Huang et al. [21] generalized BasisNet and proposed SPE that is both basis-invariant and stable to perturbation to the Laplacian matrix, but it still suffers from exponential complexity as BasisNet. In this paper, we showed that under certain assumptions, the issue of basis ambiguity can be resolved more efficiently in the pre-processing stage.

There is also a literature beyond GNN that also considers spectral embeddings of graphs and going around the sign/basis ambiguity problems. Lai and Zhao [26] proposed to address sign/basis ambiguities using optimal transport theory that involves solving a non-convex optimization problem, thus it could be less efficient than our approach. Tam and Dunson [48] proposed to symmetrize the embedding using a heuristic measure called ELD that is quite similar to the form of SignNet, while our MAP algorithm offers an axis projection approach and establish its theoretical guarantees.

Canonical Forms The theory of canonical forms has been widely studied in mathematics [39] and applied to many fields of machine learning. Sajnani et al. [43] proposed a self-supervised method named ConDor that learns to canonicalize the 3D orientation and position for full and partial 3D point clouds. Agaram et al. [2] presented Canonical Field Network (CaFi-Net), a self-supervised method to canonicalize the 3D pose of instances from an object category represented as neural fields, specifically neural radiance fields (NeRFs). Sun et al. [47] proposed an unsupervised capsule architecture for 3D point clouds. Kaba et al. [23] proposed to decouple the equivariance and prediction components of neural networks by predicting a transformation that can be used to align the input data to some canonical pose. After this alignment the remaining layers do not have to be equivariant anymore. However, as far as we know, no existing work addresses the issue of ambiguities , which arises when some function ( e.g. , eigendecomposition) is required to be both equivariant on the input space and invariant on the output space. No existing work addresses the canonizability of input data, which only arises when the invariance and equivariance property conflicts. Instead, we develop a novel theoretical framework that addresses these issues.

C Implementation and verification of MAP

C.1 Complete pseudo-code of MAP

A simplified pseudo-code for sign canonization with MAP is shown in Algorithm 1. A simplified pseudo-code for basis canonization with MAP is shown in Algorithm 2. ( j = (1 , 1 , . . . , 1) ∈ R n )

The complete pseudo-code of MAP is shown in Algorithm 3. Despite the sophisticated workflow, programmers do not have to know the principles of our algorithm to use it. The entire module can be passed as a pre_transform function to the dataset class with a single line of code.

There are three steps in Algorithm 3. The first is to eliminate sign ambiguity with O ( n 2 log n ) time complexity. The second is to eliminate basis ambiguity with O ( n 2 dm ) time complexity, where d is the multiplicity of the eigenvalue and m is the number of multiple eigenvalues of ˆ A . The third is

to incorporate eigenvalue information with O ( n 2 ) time complexity. The overall time complexity is O ( n 3 ) with the bottleneck being the eigendecomposition.

The second part of our algorithm (eliminating basis ambiguity) has time complexity O ( n 2 dm ) . We point out that in real-world datasets m is often quite small. As shown in Table 9, multiple eigenvalues only make up around 5 % of all eigenvalues, thus in practice the time complexity of the second part is O ( n 2 d ) , better than eigendecomposition. (Note it is important to take floating-point errors into account when counting these multiple eigenvalues)

Some parts of Algorithm 3 have time complexity O ( nd 4 ) . We also point out that d is usually small in real datasets. We show the number of eigenvalues and their multiplicities in logarithmic scale in Figure 2.

The details of the datasets are listed in Table 10.

C.2 Verifying the correctness of Theorem 3

We verify the correctness of Theorem 3 through random simulation. The program is shown in Algorithm 4. Let U be a random orthonormal matrix, P be a random permutation matrix and S be a random sign matrix (diagonal matrix of 1 and -1 ). We pass U , PU , US , PUS to the UNIQUESIGN function (Algorithm 1) and compare the outputs. If our algorithm is correct, U and US should have invariant outputs, while U , PU and PUS should have equivariant outputs.

We conduct 1000 trials. The results are p _ correct = q _ correct = pq _ correct = total = 1000 , showing that Algorithm 1 is both permutation-equivariant and unique (unambiguous).

Algorithm 3 Maximal Axis Projection

2

.

5

5

.

0

7

.

5

10

-

1

10

1

10

3

#eigenvalues

molesol

2

.

5

5

.

0

10

-

1

10

1

10

3

molfreesolv

0

25

10

-

1

10

2

10

5

molhiv

0

10

10

-

1

10

2

10

5

#eigenvalues

mollipo

0

20

10

-

1

10

2

10

5

moltox21

0

20

10

-

1

10

2

10

5

moltoxcast

0

25

multiplicity

10

-

1

10

2

10

5

#eigenvalues

Algorithm 4

print out the values of p _ correct , q _ correct , pq _ correct and total

C.3 Verifying the correctness of Theorem 5

We verify the correctness of Theorem 5 through random simulation. The program is shown in Algorithm 5. Let U be a random orthonormal matrix in R n × d , P be a random permutation matrix and Q be a random orthonormal matrix in R d × d . We pass U , PU , UQ , PUQ to the UNIQUEBASIS function (Algorithm 2) and compare the outputs. If our algorithm is correct, U and UQ should have invariant outputs, while U , PU and PUQ should have equivariant outputs.

We conduct 1000 trials. The results are p _ correct = q _ correct = pq _ correct = total = 1000 , showing that Algorithm 2 is both permutation-equivariant and unique. The function UNIQUEBASIS raises an assertion error when either Assumption 2 or Assumption 3 is violated, so Algorithm 5 also shows that random orthonormal matrices violate these assumptions with probability 0 .

C.4 Verifying the correctness of Theorem 1

We conduct experiment on the EXP dataset proposed in Abboud et al. [1], which is designed to explicitly evaluate the expressive power of GNNs. The dataset consists of a set of 1-WL indistinguishable non-isomorphic graph pairs, and each graph instance is a graph encoding of a propositional formula. The classification task is to determine whether the formula is satisfiable (SAT). Since the graph pairs in the EXP dataset are not distinguishable by 1-WL test, if a model shows above 50% accuracy on this dataset, it should have expressive power beyond the 1-WL algorithm.

The models we used on EXP dataset are as follows: an 8-layer GCN, GIN [57], PPGN [31], 1-23-GCN-L [35], 3-GCN [1], DeepSets-RNI (DeepSets with Random Node Initialization (RNI) [1]), GCN-RNI (GCN with Random Node Initialization (RNI)), Linear-RSE (linear network with RSE), and DeepSets-RSE. GCN and GIN belongs to the family of MP-GNNs whose expressive power is bounded by 1-WL, and RNI is a method to improve the expressive power of MP-GNNs. We verify the expressive power gain of RSE-based models by comparing them with GCN, and evaluate their efficiency by comparing them with other expressive models.

We evaluate all models on the EXP dataset using 10-fold cross-validation, and train each model for 500 epochs per fold. Mean test accuracy across all folds is measured and reported. The results are reported in Table 11. In addition, we also measure the learning curves of models to show their convergence rate, as shown in Figure 3.

In Table 11, we observe that vanilla GCN and DeepSets-RNI only achieves 50% accuracy, because they do not have expressive power beyond the 1-WL test. DeepSets-RSE achieves the best performance among all models with a near 100% accuracy, which demonstrates the expressive power of

0

100

200

300

400

500

Epoch

50

60

70

80

90

100

Test accuracy (%)

GCN­RNI

DeepSets­RSE

Linear­RSE

RSE-based models is beyond the 1-WL test. Other models, namely Linear-RSE and GCN-RNI, also achieve comparable performance. It's worth mentioning that even a simple linear model (Linear-RSE) could achieve performance above 99%. This indicates that the universal expressiveness of models with RSE is mostly from RSE itself, rather than the network structure.

From Figure 3, we find that the convergence rate of RSE-based models are much faster than their RNI-based counterpart. This is because RSE-based models are deterministic, while RNI-based models are random and require more training epochs to converge. The structures of DeepSets-RSE and Linear-RSE are simpler, which means they also train much faster than GCN-RNI.

D Synthetic experiment on basis invariance

As mentioned in Appendix B, SignNet [29] only deals with sign ambiguity, while BasisNet [29] has a prohibitive computational overhead. On the other hand, our proposed MAP addresses basis ambiguity efficiently, albeit with the existence of uncanonizable eigenspaces. In this section, we conduct a synthetic experiment to verify the ability of MAP on addressing basis ambiguity. We use graph isomorphic testing, a traditional graph task. Our focus is on 10 non-isomorphic random weighted graphs G 1 , . . . , G 10 , all exhibiting basis ambiguity issues (with the first three eigenvectors belonging to the same eigenspace). We sample 20 instances for each graph, introducing different permutations and basis choices for the initial eigenspace. The dataset is then split into a 9:1 ratio for training and testing, respectively. The task is a 10-way classification, where the aim is to determine the isomorphism of a given graph to one of the 10 original graphs. The model is given the first 3 eigenvectors as input ( i.e. k = 3 ). The results are averaged over 4 different runs.

As evident from the results in Table 12, approaches that address sign ambiguity (like RandSign and SignNet) cannot obtain nontrivial performance on this task. Conversely, MAP shows commendable performance. The 84% accuracy, although impressive, indicates potential avenues for further enhancement. We believe this synthetic task could also serve as a valuable benchmark for future studies addressing basis invariance through canonization. Code for this experiment is available at https://github.com/GeorgeMLP/basis-invariance-synthetic-experiment .

E Other positional encoding methods

In this paper, we proposed MAP, which is a kind of positional encodings. In the field of graph representation learning, many other positional encoding methods have also been proposed. Murphy et al. [36] proposed Relational Pooling (RP) that assigns each node with an identifier that depends on the index ordering. They showed that RP-GNN is strictly more expressive than the original WL-GNN. However, to ensure permutation equivalence, we have to account for all possible n ! node orderings, which is computationally intractable. You et al. [59] proposed learnable position-aware embeddings by computing the distance of a target node to an anchor-set of nodes, and showed that P-GNNs have more expressive power than existing GNNs. However, the expressive power of their model is dependent on the random selection of anchor sets. Sato et al. [44], Abboud et al. [1] proposed to use full or partial random node features and proved that their model has universal expressive power, but it has several defects: (1) the loss of permutation invariance, (2) slower convergence, and (3) poor generalization on unseen graphs [59, 30]. Li et al. [28] proposed Distance Encoding (DE) that captures the distance between the node set whose representation is to be learned and each node in the graph. They proved that DE can distinguish node sets embedded in almost all regular graphs where traditional GNNs always fail. However, their approach fails on distance regular graphs, and computation of power matrices can be a limiting factor for their model's scalability. Dwivedi et al. [18] proposed Learnable Structural and Positional Encodings (LSPE) that decouples structural and positional representations by inserting MPGNNs-LSPE layers and showed promising performance on three molecular benchmarks.

In particular, we will show that with Random Node Initialization (RNI), (1) A linear network is universal on a fixed graph, and (2) An MLP with just one additional message passing layer can be universal on arbitrary graphs.

In our work, we denote RNI as concatenating a random matrix to the input node features. The random matrix can be sampled from Gaussian distribution, uniform distribution, etc. Without loss of generality, we will assume that each entry of the random matrix is sampled independently from the standard Gaussian distribution N (0 , 1) .

Definition 2. A GNN with RNI is defined by concatenating a random matrix R to the input node features, i.e., X ′ = [ X , R ] , where X are the original node features, X ′ are the modified node features and each entry of R is sampled independently from the standard Gaussian distribution N (0 , 1) . The value of R is sampled at every forward pass of GNN.

To study the effects of RNI on the expressiveness of GNNs, we consider two types of tasks: tasks on a fixed graph (e.g., node classification) and tasks on arbitrary graphs (e.g. graph classification). On a fixed graph, we aim to learn a function f : R n × d → R n × d ′ that transforms the feature of each node v i to a presentation vector Z i, : ∈ R 1 × d ′ . We claim that a linear GNN with RNI in the form

is universal, where W ∈ R d × d ′ are the network parameters, and Z ∈ R n × d ′ is the desired output. In other words, we have the universality theorem of linear GNNs with RNI on a fixed graph:

Theorem 7. On a fixed graph G , a linear GNN with RNI defined by Equation 6 is equivariant and can produce any prediction Z ∈ R n × d ′ with probability 1 .

We prove Theorem 7 in Appendix K.8.

On arbitrary graphs, the target function is not only dependent on the node features, but on the graph structure ˆ A ∈ R n × n as well. Let Ω ⊂ R n × d × R n × n be a compact set of graphs with [ X , ˆ A ] ∈ Ω , where X are the node features and ˆ A is the normalized adjacency matrix. We wish to learn a function f : Ω → R that transforms each graph to its label. Since f is also dependent on ˆ A , an MLP-based network with X ′ as input is not expressive enough, and we need additional graph convolutional layers to obtain information about the graph structure. However, Puny et al. [41] proved that with just one additional message passing layer, an MLP with RNI can approximate any continuous invariant graph function f .

Theorem 8 (Puny et al. [41]) . Given a compact set of graphs Ω ⊂ R n × d × R n × n , a GNN with one message passing layer, an MLP network with RNI can approximate an arbitrary continuous invariant graph function f : Ω → R to an arbitrary precision.

Theorem 8 is a direct inference of the proof of Proposition 1 in Puny et al. [41], where the authors constructed a RGNN that first transfers the graph structural information to the node features through a message passing layer, and then approximates f with a DeepSets network, which is an MLP-based network.

F Toy Examples

F.1 Toy examples for Algorithm 1

x

y

z

x

y

z

+

u

-

u

+

u

-

u

x

y

z

+

u

-

u

x

h

:=

e

z

x

y

z

+

u

(+

u

)

T

x

h

>

0

x

y

z

+

u

-

u

x

y

z

+

u

x

h

:=

e

x

+

e

y

x

h

(+

u

)

T

x

h

>

0

x

h

Step 1:

⟨

u

,

e

z

⟩

<

⟨

u

,

e

y

⟩

<

⟨

u

,

e

x

⟩

, so we let

x

h

:=

e

z

Step 2:

(+

u

)

T

x

h

>

0

>

(

-

u

)

T

x

h

, so we choose

+

u

as PE

Step 1:

⟨

u

,

e

x

⟩

=

⟨

u

,

e

y

⟩

<

⟨

u

,

e

z

⟩

, so we let

x

h

:=

e

x

+

e

y

Step 2:

(+

u

)

T

x

h

>

0

>

(

-

u

)

T

x

h

, so we choose

+

u

as PE

We give toy examples to help illustrate our MAP-sign algorithm. As shown on the top row of Figure 4, we have n = 3 and two possible sign choices for the eigenvector u . Our algorithm first compares the angles (or equivalently, the absolute value of inner product) between u and the standard basis: e x , e y , e z , and pick the smallest one (the one with the largest absolute inner product), in this case e z . Thus we let x h = e z and choose the sign that maximize u ⊤ x h . In the first example we have (+ u ) ⊤ x h > 0 > ( -u ) ⊤ x h , thus + u is chosen instead of -u . This choice is unique and permutation-equivariant .

It is possible though, that the angle between u and more than 1 basis vectors are equal. As shown on the bottom row of Figure 4, the angle between both e x , e y and u are maximum. In this case we let x h = e x + e y be their sum and maximize u ⊤ x h , thus + u is chosen. A special case is when u and x h are perpendicular. If this happens, we go on to pick the basis vector with the second largest angle and continue this process.

x

1

:=

e

x

x

y

z

x

2

:=

e

y

x

1

x

2

find

u

1

∈

V

to maximize

u

T

1

x

1

x

1

x

2

u

1

V

V

⟨

u

1

⟩

⊥

find

u

2

u

1

⟩

⊥

to maximize

u

T

2

x

2

x

1

x

2

u

1

u

2

Step 1:

⟨

V,

e

x

⟩

<

⟨

V,

e

y

⟩

<

⟨

V,

e

z

⟩

, so we let

x

1

:=

e

x

,

x

2

:=

e

y

Step 2: find u 1 ∈ V to maximize u T 1 x 1

Step 3: find u 2 ∈ ⟨ u 1 ⟩ ⊥ to maximize u T 2 x 2

F.2 Toy examples for Algorithm 2

We give a toy example to help illustrate our MAP-basis algorithm. As shown in Figure 5, we have n = 3 , d = 2 with a two-dimensional eigenspace V . Our algorithm first compares the angles (or equivalently, the length of orthogonal projection) between V and the standard basis: e x , e y , e z , and pick the two smallest one (the two with the largest and second largest lengths of orthogonal projection), in this case e x and e y . Thus we let x 1 = e x , x 2 = e y . Then we choose u 1 ∈ V that maximizes u ⊤ 1 x 1 , which is just the orthogonal projection of x 1 onto V . Finally we choose u 2 ∈ ⟨ u 1 ⟩ ⊥ that maximizes u ⊤ 2 x 2 , which is the orthogonal projection of x 2 onto ⟨ u 1 ⟩ ⊥ . This gives us a basis u 1 , u 2 of V . This choice is unique and permutation-equivariant .

It is possible though, that the angle between V and more than 1 basis vectors are equal. For example, if ⟨ V, e x ⟩ = ⟨ V, e y ⟩ are both the smallest angles between V and the standard basis, then we let x 1 = e x + e y . The same goes for the second smallest angle, the third smallest angle, and so on.

G Discussions on the assumptions

G.1 Discussions on Assumption 1

The purpose of Section 3.3.1 is to uniquely determine the signs of the eigenvectors while also preserving their permutation-equivariance. One could easily come up with simple solutions such as choosing the signs such that the sum of the entries of eigenvectors are positive, or signs such that the element with the greatest absolute value is positive. In fact, Lim et al. [29] has proposed a more general way of choosing signs:

We also consider ablations in which we . . . choose a canonical sign for each eigenvector by maximizing the norm of positive entries.

This 'canonical sign' approach did not work well because (1) a large percentage of eigenvectors in real-world datasets have the same norm for positive and negative entries, thus this approach does not actually solve sign ambiguity of these eigenvectors; (2) this approach cannot be generalized to multiple eigenvalues. As a reference, we list the number of eigenvectors that 'canonical sign' fails in Table 13.

Some examples of such eigenvectors in real-world datasets are as follows.

[

[

[

Assumption 1, on the other hand, is less restrictive. It requires that the eigenvectors are not perpendicular to at least one of the vectors x i . For random unit vectors or random weighted graphs, Assumption 1 has 0 possibility of being violated. The number of eigenvectors violating Assumption 1 in real-world datasets are listed in Table 14. It can be observed that the ratio of violation tends to become smaller as the graph size becomes larger.

G.2 Discussions on Assumption 2 and Assumption 3

Assumption 2 requires that the projections of e 1 , e 2 , . . . , e n on V have at least d distinct lengths, while Assumption 3 requires that each x i is not perpendicular to ⟨ u 1 , . . . , u i -1 ⟩ ⊥ , which is a subspace of V . We illustrate two examples of these assumptions being violated when n = 3 and d = 2 in Figure 6. In the left figure, the projections of e 1 , e 2 , e 3 on V all have the same lengths, thus k = 1 < d , violating Assumption 2. In the right figure, x 2 is perpendicular to the orthogonal complementary space of span( u 1 ) in V , violating Assumption 3. We observe that the eigenspace V needs to obey certain kinds of symmetries in order to violate either assumptions. For random orthonormal matrices and random weighted graphs, these assumptions have 0 possibility of being violated; and in real-world datasets, the ratio of violation tends to become smaller as the graph size becomes larger.

The number of multiple eigenvalues violating these assumptions in real-world datasets are listed in Table 15. Indeed there is still around 20 % of violation on large datasets, but since multiple

V

e

1

e

2

e

3

V

x

1

x

2

u

1

eigenvalues only make up a small portion of all eigenvalues (Table 9), the ratio of violation is relatively small and has negligible influence on the model performance.

H Discussions on random graphs

In this section we summarize some existing results on random graphs and more generally random matrices. These discussions help us to have a better understanding of how eigenvalues and eigenvectors of random graphs distribute, and the probability that they are uncanonizable by our MAP algorithm. Due to the theoretical complexity, we do make some simplifications in our discussions.

We first give some basic definitions about random matrices and random graphs.

Definition 3 (Wigner matrix) . Let ξ, ζ be real random variables with mean zero. We say W is a Wigner matrix of size n with atom variables ξ, ζ if W = ( w ij ) n i,j =1 is a random real symmetric n × n matrix that satisfies the following conditions.

(independence) { w ij : 1 ≤ i ≤ j ≤ n } is a collection of independent random variables.

(off-diagonal entries) { w ij : 1 ≤ i < j ≤ n } is a collection of independent and identically distributed (iid) copies of ξ .

(diagonal entries) { w ii : 1 ≤ i ≤ n } is a collection of iid copies of ζ .

If ξ and ζ have the same distribution, we say W is a Wigner matrix with atom variable ξ . We always assume that ξ is non-degenerate , namely that there is no value c such that P ( ξ = c ) = 1 .

Definition 4 (symmetric Bernoulli matrix) . Let 0 < p < 1 , and take ξ to be the random variable

Then ξ has zero mean. Let B n ( p ) denote the n × n Wigner matrix with atom variable ξ . We refer to B n ( p ) as a symmetric Bernoulli matrix (with parameter p ).

Definition 5 (sub-exponential) . A random variable ξ is called sub-exponential with exponent α > 0 if there exists a constant β > 0 such that

Definition 6 (Erd˝ os-Rényi random graph) . Let G ( n, p ) denote the Erd˝ os-Rényi random graph on n vertices with edge density p . That is, G ( n, p ) is a simple graph on n vertices such that each edge { i, j } is in G ( n, p ) with probability p , independent of other edges.

Let A n ( p ) be the zero-one adjacency matrix of G ( n, p ) . A n ( p ) is not a Wigner matrix since its entries do not have mean zero. Let ˜ G ( n, p ) denote the Erd˝ os-Rényi random graph with loops on n vertices with edge density p . Let ˜ A n ( p ) denote the zero-one adjacency matrix of ˜ G ( n, p ) . Technically, ˜ A n ( p ) is not a Wigner random matrix because its entries do not have mean zero. However, we can view ˜ A n ( p ) as a low rank deterministic perturbation of a Wigner matrix. That is, we can write ˜ A n ( p ) as

where J n is the all-ones matrix.

Simplification in our discussions. The adjacency matrix of Erd˝ os-Rényi random graphs can be viewed as a rank-one perturbation of a zero-mean symmetric random matrix, which has no effect on the distribution of eigenvalues in the limit n →∞ [8]. In the finite case, the difference between eigenvalues and eigenvectors of the perturbed and unperturbed random matrices can be bounded as well, as stated in the following theorems.

Theorem 9 (O'Rourke et al. [38]) . Let E be an n × n Bernoulli random matrix, and let A be an n × n matrix with rank r . Let σ 1 ≥ σ 2 ≥ ·· · ≥ σ min { m,n } ≥ 0 be singular values of A , v 1 , v 2 , . . . , v min { m,n } be corresponding singular vectors of A , σ ′ 1 ≥ ·· · ≥ σ ′ min { m,n } ≥ 0 be singular values of A + E , v ′ 1 , . . . , v min { m,n } be corresponding singular vectors of A + E . For every ε > 0 there exists constants C 0 , δ 0 > 0 (depending only on ε ) such that if δ > δ 0 and σ 1 ≥ max { n, √ nδ } , then, with probability at least 1 -ε ,

Theorem 10 (O'Rourke et al. [38]) . Let E be an n × n Bernoulli random matrix, and let A be an n × n matrix with rank r satisfying σ 1 ≥ n . For every ε > 0 , there exists a constant C 0 > 0 (depending only on ε ) such that, with probability at least 1 -ε ,

In particular, when the rank r is significantly smaller than n , the bounds in the above theorems are significantly better. Thus the rank-one perturbation has little or no effect on the spectral distribution of the adjacency matrix. In the following discussions we will ignore the rank-one perturbation and assume that the adjacency matrix of random graphs have zero mean.

We will study two cases: (1) the adjacency matrix of the random graph is continuously distributed (random weighted graph); (2) the adjacency matrix of the random graph is discretely distributed (random unweighted graph).

It is easy to see that for a Wigner matrix with atom variables ξ, ζ , if the distribution of ξ is continuous, then with probability 1 it has simple spectrum ( i.e. , all eigenvalues have multiplicity one), thus no basis ambiguity exists. For its single eigenvectors, we also expect these eigenvectors to be continuously distributed on the unit sphere, thus the probability that they are uncanonizable is equal to 1 . In fact, we can even give a explicit formula for the distribution of some simple random matrices, as follows.

Definition 7 (GOE) . The Gaussian orthogonal ensemble (GOE) is defined as a Wigner random matrix with atom variables ξ, ζ , where ξ is a standard normal random variable and ζ is a normal random variable with mean zero and variance 2 .

Theorem 11 (Anderson et al. [4], Section 2.5.1) . Let M be a n × n real symmetric matrix drawn from the GOE, with eigenvalues λ 1 ≤ · · · ≤ λ n and corresponding eigenvectors v 1 , . . . , v n . Then the eigenvectors v 1 , . . . , v n are uniformly distributed on

and the eigenvalues λ 1 , . . . , λ n have joint density

where Z n is a normalization constant.

Thus the eigenvectors of such Gaussian random matrices are uniformly distributed on the unit sphere, meaning they are canonizable with probability 1 .

The discrete case is trickier, because when n is finite, the probability that the eigenvectors are uncanonizable is no longer 0 . We wish that this probability can be upper bounded and asymptotically converges to zero. The first fact is that for a large class of random matrices, perturbed or not, they have simple spectrum with probability 1 -o (1) .

Theorem 12 (Tao and Vu [50]) . Consider real symmetric random matrices M n = ( ξ ij ) 1 ≤ i,j ≤ n , where the entries ξ ij for i ≤ j are jointly independent with ξ ji = ξ ij , the upper-triangular entries ξ ij , i < j have non-trivial distribution ξ for some fixed µ > 0 such that P ( ξ = x ) ≤ 1 -µ for all x ∈ R . The diagonal entries ξ ii , 1 ≤ i ≤ n can have an arbitrary real distribution (and can be correlated with each other), but are required to be independent of the upper diagonal entries ξ ij , 1 ≤ i < j ≤ n . Then for every fixed A > 0 and n sufficiently large (depending on A,µ ), the spectrum of M n is simple with probability at least 1 -n -A .

Thus with probability 1 -o (1) no basis ambiguity exists.

For sign ambiguity, we expect that the probability that the single eigenvectors are canonizable asymptotically converges to 1 as n → ∞ . Denote j = (1 , 1 , . . . , 1) ∈ R n the all-one vector, ˆ j = ( 1 √ n , 1 √ n , . . . , 1 √ n ) ∈ R n the normalized all-one vector. One obvious fact is that for an eigenvector u ∈ R n , if it is non-canonizable, then u · j = 0 . Thus it suffices to show that the probability of the inner product between u and j being zero converges to 0 as n →∞ . This can be derived from the following theorem.

Theorem 13 (Tao and Vu [49]) . Let ξ, ζ be random variables such that

ξ and ζ are sub-exponential random variables,

E ( ξ ) = E ( ζ ) = E ( ξ 3 ) = 0 ,

E ( ξ 2 ) = 1 , E ( ξ 4 ) = 3 , E ( ζ 2 ) = 2 ,

and assume ξ is a symmetric random variable. For each n ≥ 1 , let W n be an n × n Wigner matrix with atom variables ξ, ζ . Let { a n } be a sequence of unit vectors with a n ∈ S n -1 such that lim n →∞ ∥ a n ∥ ℓ ∞ = 0 , and let { i n } be a sequence of indices with i n ∈ [ n ] . Then in distribution as n →∞ .

Thus in the limit as n →∞ , by taking a n = ˆ j , we have the inner product between the eigenvector and j following a normal distribution. The probability that they have zero inner product is 0 , indicating these eigenvectors are almost always canonizable.

Related readings. We refer the readers to Anderson et al. [4] for a introduction of random matrices, Benaych-Georges and Nadakuditi [8] for a survey about the singular values and vectors of low rank perturbations of large random matrices, O'Rourke et al. [37] for a comprehensive survey about eigenvectors of random matrices.

I Further weakening the assumptions

I.1 Further weakening Assumption 1

In Section 3.2, we mentioned that some eigenvectors are intrinsically uncanonizable , meaning that it is impossible to canonize them based on themselves. However, in graph-level tasks, the input is a whole graph with n eigenvectors, not a single eigenvector. When the input graph is permuted by a permutation σ , all the eigenvectors are also permuted in the same way. This raises the question of whether we can canonize one eigenvector based on other eigenvectors in the same graph.

Thus we propose to further weaken Assumption 1 in the following steps. For a given input graph, we first divide its eigenvectors into two sets: S 1 , containing all uncanonizable eigenvectors; and S 2 , containing all canonizable eigenvectors. Suppose | S 1 | = d 1 , | S 2 | = d 2 , where d 1 + d 2 = n . Define the matrix of canonizable eigenvectors

We fist canonize all the eigenvectors in S 2 , using our MAP-sign algorithm. This gives us a matrix of canonized eigenvectors U ∗ can ∈ R n × d 2 . Using a hash function, we can compress the matrix U ∗ can into a vector u can that preserves all the information of S 2 :

Then we can define the summary vectors x i of u can in the same way as in Section 3.3, and use them to canonize the eigenvectors in S 1 . Denote the projection matrix P = u can u ⊤ can and the projected angles α i = | Pe i | , 1 ≤ i ≤ n . Assume that there are k distinct values in { α i , i = 1 , . . . , n } , according to which we can divide all basis vectors { e i } into k disjoint groups B i (arranged in descending order of the distinct angles). Each B i represents an equivalent class of axis that u can has the same projection on. Then we define a summary vector x i of u can for the axes in each group B i as their total sum x i = ∑ e j ∈ B i e j + c , where c is a tunable constant.

For each u ∈ S 1 , we can try to canonize it using the MAP-sign algorithm as in Section 3.3.1, with the summary vectors of u replaced by the summary vectors of u can . We find the first non-orthongonal summary vector with u , denoted by x h , and choose the sign that maximizes u ⊤ x h . In this way, we are able to canonize some eigenvectors in S 1 that are originally uncanonizable, thus further weakening Assumption 1. The complete workflow is shown in Algorithm 6.

I.2 Further weakening Assumption 2 and Assumption 3

When reading Algorithm 2, some may find it not as strong as Algorithm 1. In Algorithm 1, we search over the summary vectors x i one by one, until we find a summary vector x h that is not

orthogonal to u . However, in Algorithm 2, we just assumed that x 1 is not orthogonal to V , x 2 is not orthogonal to ⟨ u 1 ⟩ ⊥ , x 3 is not orthogonal to ⟨ u 1 , u 2 ⟩ ⊥ , etc . We did not search for non-orthogonal summary vectors; we just assumed that they are. This observation is more obvious when we look at the special case of Algorithm 2 when d = 1 . In this case, Assumption 2 is always satisfied, and Assumption 3 requires that x 1 is not orthogonal to u , which is stricter than Assumption 1 in our MAP-sign algorithm. This raises the question of whether we can strengthen Algorithm 2 such that it is as powerful as Algorithm 1 when taking d = 1 .

This can be achieved by searching for non-orthogonal summary vectors at each step just as in Algorithm 1. Suppose we have obtained the summary vectors x i , i = 1 , . . . , k . First we search { x i } for the first summary vector x ∗ 1 that is not orthogonal to V , and choose u 1 ∈ V that maximizes u ⊤ 1 x ∗ 1 . Next we search { x i } for the first summary vector x ∗ 2 that is not orthogonal to ⟨ u 1 ⟩ ⊥ , and choose u 2 ∈ ⟨ u 1 ⟩ ⊥ that maximizes u ⊤ 2 x ∗ 2 . Then we search { x i } for the first summary vector x ∗ 3 that is not orthogonal to ⟨ u 1 , u 2 ⟩ ⊥ , and choose u 3 ∈ ⟨ u 1 , u 2 ⟩ ⊥ that maximizes u ⊤ 3 x ∗ 3 , and so on. The complete workflow is shown in Algorithm 7.

In order for Algorithm 7 to succeed, we require that such non-orthogonal summary vector x ∗ i exists at each step, which is less restrictive than Assumption 2 and Assumption 3. Thus we obtain a stronger algorithm for eliminating basis ambiguity.

J An alternative approach to dealing with sign ambiguity

In our design for algorithms that eliminate sign ambiguity, we find one type of eigenvectors especially difficult to deal with. That is, uncanonizable eigenvectors u ∈ R n such that there exists a permutation matrix P ∈ R n × n satisfying u = -Pu . This means that u and -u only differ by a permutation. Since they are uncanonizable by Corollary 1, none of the solutions mentioned in Appendix G.1 can handle such eigenvectors. However, depending on the model architecture, such eigenvectors may not cause ambiguities at all. For example, consider the DeepSets-like architecture ρ ( ∑ ϕ ( u i )) , where ϕ is permutation-invariant. Then since u and -u only differ by a permutation, they produce the same output when fed to a permutation-invariant network, thus no ambiguities exist. This shows we can delay the handling of uncanonizable eigenvectors to the training stage, though it may result in loss of expressive power.

On the other hand, for eigenvectors that are canonizable, we already know that Algorithm 1 canonizes them. Here we give an equivalent algorithm for removing sign ambiguity, shown in Algorithm 8.

Theorem 14. Algorithm 8 uniquely decides the signs of canonizable eigenvectors and is permutationequivariant.

Algorithm 8 is well-motivated and better to understand than Algorithm 1 in some sense. Consider a naïve canonization of u where we choose the sign such that u has positive sum. This canonization algorithm is quite simple, but it cannot canonize all canonizable eigenvectors, such as the ones with

̸

zero sum. What Algorithm 8 does is to go on to look at the sum of the 3rd power of u and, if it is non-zero, choose the sign such that it is positive. If unfortunately it is zero, we go on to look at the sum of the 5th power and so on. It can be proved that there must exists a positive odd integer h ≤ n such that the sum of the h -th power of u is non-zero. Thus Algorithm 8 terminates within n +1 2 steps, successfully canonizing all canonizable eigenvectors.

Algorithm 8 offers an alternative approach to dealing with sign ambiguity in addition to Algorithm 1, though it cannot generalize to the basis ambiguity case. The time complexity of Algorithm 8 is O ( n 2 log n ) .

K Proofs

K.1 Proof of Theorem 1

We first prove that UΛ 1 2 is a real-valued matrix.

Lemma 1. Suppose ˆ A is the normalized adjacency matrix of a graph G , and ˆ A = UΛU ⊤ is its spectral decomposition. Then, UΛ 1 2 ∈ R n × n .

Proof. Let Λ = diag( λ ) . It suffices to show that λ i ≥ 0 for i = 1 , 2 , . . . , n .

Let G = ( V , E ) , where V is the vertex set and E is the edge set. For node i , we denote the degree of node i by d i . For any x ∈ R n , we have

thus the Rayleigh quotient of ˆ A is bounded by x ⊤ ˆ Ax x ⊤ x ≥ 0 . The Rayleigh quotient gives the lower bound of eigenvalues of ˆ A , therefore we have λ i ≥ 0 , and this completes the proof.

Then we give the proof of Theorem 1.

Proof. We can rewrite f such that it is a continuous set invariant function on the set consisting of the rows of its input:

Using the permutation invariance property of f , we can verify that F is set invariant by observing:

Thus a universal network on sets can approximate F to an arbitrary precision.

K.2 Proof of Theorem 2

Proof. On the one hand, if f ( hx ) = gf ( x ) holds for some h ∈ H and g ∈ G , then we have A ( f ( hx ) ) = h A ( f ( x ) ) by the equivariance property of A and A ( gf ( x ) ) = A ( f ( x ) ) by the invariance property of A . However, since A is a mapping and f ( hx ) = gf ( x ) , there must be A ( f ( hx ) ) = A ( gf ( x ) ) and thus A ( f ( x ) ) = h A ( f ( x ) ) . Now we have x = hx and A ( f ( x ) ) = h A ( f ( x ) ) , contradicting the universality property of A .

̸

̸

On the other hand, if there does not exist h ∈ H and g ∈ G such that x = hx and f ( hx ) = gf ( x ) , we can construct a canonization of x as follows. First arbitrarily choose y 0 such that f ( x ) = y 0 , and let A ( f ( x ) ) = y 0 . For any h ∈ H such that x = hx , by the equivariance of f we know that f ( hx ) = hy 0 , thus we let A ( f ( hx ) ) = hy 0 . Since f ( hx ) = gf ( x ) for any g , we know that y 0 = hy 0 , thus the universality property of A holds. A is also invariant, since A ( f ( x ) ) is uniquely determined; and equivariant, since A ( f ( hx ) ) = hy 0 = h A ( f ( x ) ) for any h . We can repeat this process for all equivalence classes in X and obtain an invariant, equivariant and universal canonization for all inputs.

̸

K.3 Proof of Corollary 1

Proof. Under sign ambiguity, we have H = S n and G = { +1 , -1 } . By Theorem 2, u is canonizable if and only if there does not exist a permutation σ such that u = σ ( u ) and u = ± σ ( u ) . This is equivalent to say that there does not exist a permutation matrix P ∈ R n × n such that u = -Pu .

K.4 Proof of Corollary 2

̸

Proof. Under basis ambiguity, we have H = S n and G = O ( d ) . By Theorem 2, U is canonizable if and only if there does not exist a permutation σ and an orthonormal matrix Q ∈ O ( d ) such that U = σ ( U ) and U = σ ( U ) Q . This is equivalent to say that there does not exist a permutation matrix P ∈ R n × n such that U = PU and U and PU span the same d -dimensional subspace V ⊆ R n . Note here we used a lemma from Lim et al. [29]: for any orthonormal bases V and W of the same subspace, there exists an orthogonal Q ∈ O ( d ) such that V Q = W .

̸

K.5 Proof of Theorem 3

Proof. Without loss of generality, we can always assume that the angles { α i } are sorted (if they are not, we can simply rearrange e i to make them sorted and proceed in the same way):

where k is the number of distinct lengths of Pe i , ∑ k i =1 n i = n . Here we divide e i into k groups according to the angles between them and the eigenspace, with each group sharing the same | Pe i | . Define j = (1 , 1 , . . . , 1) ∈ R n , then x i can be expressed as

̸

The sign of u 0 is selected based on the sign of u ⊤ x h ( = 0) , which is sign-equivariant. No matter what the sign of u is, we will always choose the one that maximizes u ⊤ x h , thus u 0 is sign-invariant.

Suppose the entries of the input eigenvector u is permutated by σ ∈ S n , where S n is the permutation group of order n . Then we have u ′ i = u σ ( i ) (1 ≤ i ≤ n ) and

thus ( x ′ i ) i = ( x i ) σ ( i ) , i.e. , the vectors x i (1 ≤ i ≤ k ) are permutation-equivariant. Let x ′ h be defined as in Section 3.3.1 (the number h is permutation-invariant because each u ′⊤ x ′ i is permutationinvariant). The sign of u ′ 0 is determined by the sign of the dot product of u ′ and x ′ h , both of which are permutation-equivariant. This indicates that the sign of u ′⊤ x ′ h (and thus the sign of u ′ 0 ) is

̸

̸

̸

permutation-invariant ( i.e. , unique). Since u ′ 0 = u ′ or u ′ 0 = -u ′ , u ′ 0 is permutation-equivariant as well.

̸

If there exists a permutation σ (acting on entries of u ) such that u = σ ( u ) but they have the same canonical form, then either u = + σ ( u ) or u = -σ ( u ) . The former is impossible since we already assumed they are not equal, and the latter violates Assumption 1, leading to a contradiction. Thus the canonization of u is universal.

K.6 Proof of Theorem 4

Proof. Without loss of generality, we can always assume that the angles { α i } are sorted (if they are not, we can simply rearrange e i to make them sorted and proceed in the same way):

where k is the number of distinct lengths of Pe i , ∑ k i =1 n i = n . Here we divide e i into k groups according to the angles between them and the eigenspace, with each group sharing the same | Pe i | . Define j = (1 , 1 , . . . , 1) ∈ R n , then x i can be expressed as

Notice that | Pe i | = | uu ⊤ e i | = u i | u | , thus

Suppose u violates Assumption 1. Then for any 1 ≤ j ≤ k , we have

Let ˜ u j := ( u n 1 + ··· + n j -1 +1 , . . . , u n 1 + ··· + n j ) ⊤ . The above equations show that (1) the absolute value of the entries of ˜ u j are all equal; (2) the sum of the entries of ˜ u j is 0 . Thus for any entry of ˜ u j , either it is 0 or its positive and negative value appears in pairs. In conclusion, +˜ u j and -˜ u j are equal up to a permutation for all j , thus + u and -u are equal up to a permutation. By Theorem 1, it is not canonizable.

On the other hand, suppose u satisfies Assumption 1. Then there exists 1 ≤ j ≤ k such that

̸

Let ˜ u j := ( u n 1 + ··· + n j -1 +1 , . . . , u n 1 + ··· + n j ) ⊤ . ˜ u j contains all entries of u with absolute value | u n 1 + ··· + n j | . Since the sum of ˜ u j is non-zero, the numbers of positive and negative entries in ˜ u j are different. No matter how we permute + u and -u , their corresponding entries in ˜ u j will not align. Thus + u and -u are not equal up to any permutation. By Theorem 1, it is canonizable.

K.7 Proof of Theorem 5

We first point out that the orthogonal projector P = UU ⊤ is invariant to the choice of basis.

Lemma 2. Let U = [ u 1 , . . . , u d ] ∈ R n × d and V = [ v 1 , . . . , v d ] ∈ R n × d be two sets of orthonormal vectors that span the same d -dimensional subspace V ⊆ R n . Then UU ⊤ = V V ⊤ .

Proof. We have U ⊤ U = V ⊤ V = I by the definition of U and V . Let U = V Q where Q is an invertible matrix, then

Lemma 2 shows that Pe i (and thus x i ) is invariant to the choice of basis in V . If we permute e 1 , . . . , e n by σ ∈ S n , then the sorted sequence of | Pe i | is also permuted by σ . Thus the elements of each x i is permuted by σ as well. This shows the choice of x i is permutation-equivariant .

Then we prove Theorem 5.

Proof. Without loss of generality, we can always assume that the angles { α i } are sorted (if they are not, we can simply rearrange e i to make them sorted and proceed in the same way):

where k is the number of distinct lengths of Pe i , ∑ k i =1 n i = n . Here we divide e i into k groups according to the angles between them and the eigenspace, with each group sharing the same | Pe i | . Define j = (1 , 1 , . . . , 1) ∈ R n , then x i can be expressed as

We have already shown the existence of the maximum value of f ( u ) . To show that basis-invariance of U 0 , it suffices to show the uniqueness of the maximum point of f ( u ) . Thus no matter what basis of U is, Algorithm 2 always yields the same output.

Notice that f ( -u ) = -f ( u ) , thus either the maximum value of f ( u ) is positive, or f ( u ) = u ⊤ x i ≡ 0 . However, f ( u ) ≡ 0 implies that x i is perpendicular to ⟨ u 1 , . . . , u i -1 ⟩ ⊥ , violating Assumption 3. Thus we conclude the maximum value of f ( u ) is positive.

̸

Suppose there exists u ′ = u ′′ such that f ( u ′ ) = f ( u ′′ ) takes maximum value. Consider the vector α u ′ + α u ′′ where α = √ 1 2(1+ u ′⊤ u ′′ ) > 1 2 . Obviously, α u ′ + α u ′′ ∈ ⟨ u 1 , . . . , u i -1 ⟩ ⊥ , | α u ′ + α u ′′ | = 1 , and f ( α u ′ + α u ′′ ) = 2 α u ′⊤ x i > u ′⊤ x i . This leads to a contradiction. Therefore, the choice of u i is unique.

The permutation-equivariance of Algorithm 2 can be proved by observing that each step of Algorithm 2 is permutation-equivariant. Since each x i is permutation-equivariant, its eigenprojection on the subspace V (and thus U 0 ) is also permutation-equivariant.

̸

If there exists a permutation σ (acting on rows of U ) such that U = σ ( U ) but they have the same canonical form, then U and σ ( U ) spans the same subspace. On the one hand, U = σ ( U ) means that at least one of the eigenvectors in U is not σ -invariant. On the other hand, since all u 1 , . . . , u d are permutation-equivariant but unchanged after σ , they are all σ -invariant. This leads to a contradiction, since it is impossible to have a nonσ -invariant eigenvector in a σ -invariant eigenspace.

K.8 Proof of Theorem 7

We first prove the following lemmas.

Lemma 3. Let R ∈ R n × n be a random matrix, and each entry of R is sampled independently from the standard Gaussian distribution N (0 , 1) . Then with probability 1 , R has full rank.

Proof. We denote the first column of R by R : , 1 . It is linearly independent because R : , 1 = 0 with probability 1 . Then we view R : , 1 as fixed, and consider the second column R : , 2 . The probability that R : , 2 falls into the span of R : , 1 is 0 , thus with probability 1 , R : , 1 and R : , 2 are linearly independent.

Generally, let us consider the k -th column R : ,k . The first k -1 columns of R forms a subspace in R n whose Lebesgue measure is 0 . Thus R : ,k falls into this subspace with probability 0 . By inference, we have all the columns of R are linearly independent with probability 1 , i.e. , P (rank( R ) = n ) = 1 .

Lemma 4. Let A ∈ R s × n , B ∈ R s × m be two matrices. Then the equation AX = B has a solution iff. rank( A ) = rank([ A , B ]) .

Proof. First we prove the necessity. Suppose AX = B has a solution. Then,

̸

where M : ,i denotes the i -th column of matrix M . This means each column of B can be expressed as a linear combination of the columns of A , and therefore each column of [ A , B ] can be expressed as a linear combination of the columns of A .

On the other hand, it is obvious that each column of A can be expressed as a linear combination of the columns of [ A , B ] . Thus we have rank( A ) = rank([ A , B ]) .

Then we prove the sufficiency. Since rank( A ) = rank([ A , B ]) , and each column of A can be expressed as a linear combination of columns of [ A , B ] , we have the columns of A and the columns of [ A , B ] are equivalent. Therefore, each column of B can be expressed as a linear combination of columns of A , i.e. , Ax = B : ,i has a solution for i = 1 , . . . , m . Thus the equation AX = B has a solution.

Then we give the proof of Theorem 7.

Proof. For any prediction Z ∈ R n × d ′ , we wish to prove that with probability 1 , there exists parameters of a linear GNN with RNI W ∈ R d × d ′ such that

By Lemma 4, the necessary and sufficient condition that Equation 7 has a solution W is rank([ X , R ]) = rank([ X , R , Z ]) .

By Lemma 3, with probability 1 , rank( R ) = n , therefore rank([ X , R ]) = rank([ X , R , Z ]) = n .

Thus, in conclusion, with probability 1 , there exists parameters of a linear GNN with RNI W ∈ R d × d ′ such that the GNN produces Z .

We can also prove linear GNNs' equivariance by observing that for any permutation matrix P ∈ R n × n ,

where we used P R = R because each entry of P R is also sampled from the standard Gaussian matrix N (0 , 1) .

K.9 Proof of Theorem 14

The following lemmas are used in our proof.

Lemma 5 (Newton's Identities) . Let x 1 , x 2 , . . . , x n be variables, denote for k ≥ 1 by P k the k -th power sum:

and for k ≥ 0 denote by e k the elementary symmetric polynomial. Then we have

for all n ≥ 1 and n ≥ k ≥ 1 .

Lemma 6 (Vieta's Formulas) . Let f ( x ) = a n x n + a n -1 x n -1 + · · · + a 1 x + a 0 be a polynomial of degree n , e k be the elementary symmetric polynomial. Then we have

We first show that h exists.

Lemma 7. Let u ∈ R n . Assume that there does not exists a permutation matrix P ∈ R n × n such that u = -Pu . Then there exists a positive odd integer h ≤ n such that ∑ n i =1 u h i = 0 .

̸

Proof. Suppose the opposite holds, i.e. , P i = 0 for all odd 1 ≤ i ≤ n , where P i is the i -th power sum of entries of u . Let u 1 , u 2 , . . . , u n be the roots of the polynomial f ( x ) = a n x n + a n -1 x n -1 + · · · + a 1 x + a 0 .

If n is even, by Lemma 5,

which gives us e 1 = e 3 = · · · = e n -1 = 0 . Then, by Lemma 6, we have a n -1 = a n -3 = · · · = a 1 = 0 . This indicates that f ( x ) = g ( x 2 ) for some polynomial g ( x ) . Similarly, if n is odd, by Lemma 5,

which gives us e 1 = e 3 = · · · = e n = 0 . Then, by Lemma 6, we have a n -1 = a n -3 = · · · = a 0 = 0 . This indicates that f ( x ) = xg ( x 2 ) for some polynomial g ( x ) . Either way, all n roots of f ( x ) are symmetric with respect to the y -axis. Then there must exist a permutation matrix such that u = -Pu , leading to a contradiction. Thus Lemma 7 holds.

Then we prove Theorem 14.

Proof. By Lemma 7, we have shown the existence of h . Since flipping the sign of u also flips the sign of ∑ n i =1 u h i (because h is odd), Algorithm 8 uniquely decides the sign of u . Since the algorithm outputs either u or -u , it is also permutation-equivariant.

K.10 Proof of Theorem 15

We prove that under mild conditions, the loss of expressive power induced by truncating RSE can be upper bounded, as shown in the following theorem.

Theorem 15. Let Ω ⊂ R n × d × R n × n be a compact set of graphs, [ X , ˆ A ] ∈ Ω . Let NN be a universal neural network on sets. Given an invariant graph function f defined over Ω that can be ε -approximated by an L p -Lipschitz continuous function and arbitrary ε > 0 , for any integer 0 < k ≤ n , there exist parameters of NN such that for all graphs [ X , ˆ A ] ∈ Ω ,

Here the L p -Lipschitz continuity of f is defined using the Frobenius norm on the input domain, 0 ≤ λ 1 ≤ · · · ≤ λ n ≤ 2 are the eigenvalues of ˆ A , 0 ∈ R n × ( n -k ) .

We can see from Theorem 15 that the upper bound of the loss of expressive power decreases when k increases, and when k = n , the network becomes universal. We give its proof as follows.

By Lemma 1, we know that λ i ≥ 0 for i = 1 , 2 , . . . , n . Next we prove that λ i ≤ 2 .

Lemma 8. Suppose ˆ A is the normalized adjacency matrix of a graph G , and λ 1 < · · · < λ n are its eigenvalues. Then λ i ≤ 2 , for i = 1 , 2 , . . . , n .

Proof. In the proof of Lemma 1, we proved x ⊤ ( I + ˜ A ) x ≥ 0 . Similarly, we have

Thus,

This shows that the Rayleigh quotient is bounded by x ⊤ ˆ Ax x ⊤ x ≤ 2 , therefore λ i ≤ 2 .

Then we give the proof of Theorem 15.

Proof. Let 0 ≤ λ 1 ≤ · · · ≤ λ n ≤ 2 be the eigenvalues of ˆ A and u 1 , . . . , u n be the corresponding eigenvectors. Then

We also define

By Theorem 1 and the assumptions in our theorem, we know that there exists a permutation-invariant network on sets such that

Since f can be approximated by an L p -Lipschitz continuous function, we have

Combining the two inequalities above gives us

L Dataset details

ZINC (MIT License) consists of 12K molecular graphs from the ZINC database of commercially available chemical compounds. These molecular graphs are between 9 and 37 nodes large. Each node represents a heavy atom (28 possible atom types) and each edge represents a bond (3 possible types). The task is to regress constrained solubility (logP) of the molecule. The dataset comes with a predefined 10K/1K/1K train/validation/test split.

OGBG-MOLTOX21 and OGBG-MOLPCBA (MIT License) are molecular property prediction datasets adopted by OGB from MoleculeNet. These datasets use a common node (atom) and edge (bond) featurization that represent chemophysical properties. OGBG-MOLTOX21 is a multi-mask binary graph classification dataset where a qualitative (active/inactive) binary label is predicted against 12 different toxicity measurements for each molecular graph. OGBG-MOLPCBA is also a multi-task binary graph classification dataset from OGB where an active/inactive binary label is predicted for 128 bioassays.

Details of the three datasets are summarized in Table 16.

M Hyperparameter settings

M.1 Real-world tasks

We evaluate the proposed MAP on three real-world datasets: ZINC, OGBG-MOLTOX21 and OGBGMOLPCBA, on a server with 6 NVIDIA 3080 Ti GPUs and 2 NVIDIA 1080 Ti GPUs. We consider

4 GNN architectures: GatedGCN, PNA, SAN and GraphiT, with 4 different positional encodings: no PE, LapPE with random Sign, SignNet and MAP. We follow the same settings as Dwivedi et al. [18] for models with no PE or LapPE, and same settings as Lim et al. [29] for models with SignNet or MAP. All baseline scores reported in Table 3, 4 & 5 are taken from the original papers. As shown in Figure 7, for models with no PE or LapPE, the input features are directly fed into the base model; for models with SignNet, the eigenvectors are first processed by SignNet and then concatenated with the original node features as input to the base model; for models with MAP, the PEs are first processed by a normal GNN and then concatenated with the original node features as input to the base model. These settings align with the original papers.

X

or

[

X

,

U

]

base model

Y

U

ρ

(

ϕ

(

u

) +

ϕ

(

-

u

)

)

X

Y

UΛ

1

2

ρ

(

ϕ

(

u

)

)

Y

no PE/LapPE

SignNet

MAP

base model

base model

X

The main hyperparameters in our experiments are listed as follows.

k : the number of eigenvectors used in the PE.

L 1 : the number of layers of the base model.

h 1 : the hidden dimension of the base model.

h 2 : the output dimension of the base model.

λ : the initial learning rate.

t : the patience of the learning rate schedular.

r : the factor of the learning rate schedular.

λ min : the minimum learning rate of the learning rate schedular.

L 2 : the number of layers of SignNet or the normal GNN (when using MAP as PE).

h 3 : the hidden dimension of SignNet or the normal GNN (when using MAP as PE).

The values of these hyperparameters in our experiments are listed in Table 17.

M.2 Synthetic tasks

To verify the expressive power of RSE, we conduct experiments on the synthetic EXP dataset. The dataset consists of a set of 1-WL indistinguishable non-isomorphic graph pairs. If a network reaches above 50 % accuracy on this dataset, it must have expressive power beyond the 1-WL test. DeepSetsRSE is a two-layer DeepSets model with RSE as PE, whereas Linear-RSE is a one-layer linear model with RSE as PE. We use Optuna [3] to optimize the hyperparameters of our models. The values of hyperparameters of our models are as follows:

DeepSets-RSE : the learning rate λ = 0 . 002385602941230316 , the hidden dimension of the first linear layer w 1 = 60 , the hidden dimension of the second linear layer w 2 = 76 , the dropout rate [46] p = 0 . 13592575703525184 , the weight decay of Adam optimizer wd = 0 . 0005 .

Linear-RSE : the learning rate λ = 0 . 0006867736568978745 , the hidden dimension w = 109 , the weight decay of Adam optimizer wd = 0 . 0001 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of expressiveness in graph neural networks. Positional encodings using spectral approaches have suffered from sign and basis invariance. The authors propose Laplacian canonization which finds unique representations, and they analyze what properties the canonization should preserve and in practice what is the ratio of eigenvectors that fulfill these conditions. They propose a new simple canonization algorithm for sign and  basis invariance—Minimal Axis Projection (MAP)---that uses axis projection functions to identify canonical directions. They also present conditions under which MAP can guarantee sign and basis canonization. Experimental results evaluate MAP on graph classification benchmarks and show improved results with lowest computational runtime.\n\nSTRENGTHS:\nOriginality: the approach appears to be novel.\n\nSignificance: Designing proper architectures that preserve symmetries and are more expressive is an important and open topic  currently in GNN literature. \n\nQuality: the approach is both simple and fast. Experiments show pairing MAP with multiple architectures to obtain gains in performance.  The ablation study demonstrates the contributions of the different parts of the approach. Full implementation details are provided in the supplement. \n\nClarity:  the paper is well-written.\n\nWEAKNESSES:\nExperimental results show incremental / modest improvement compared to BasisNet. The main contribution is in reduced runtime. Furthermore, BasisNet model with k=all in their paper (Table 1) achieve better performance than MAP in the current paper (Table 3). Authors should explain decision to not include these results from the BasisNet paper\n\nQUESTIONS:\n1. RSE: the eigenvectors should be unit-normalized (or normalized to same length) \n2. If using the low frequency eigenvectors, the corresponding eigenvalues are the smallest, and thus the values are very small (especially if also unit normalized and n is large). How is this useful in practice? Are these concatenated scaled in some way at input to the network? \n3. Are the uncanonizable eigenvectors in the datasets concentrated in low or high frequency, or spread uniformly? \n4. How would this method work for a ring graph (cycle)?\n5. Why are result using SignNet not included for tables 4 and 5?\n6. Figures 4 and 5 in the appendix are very useful toy illustration that should be moved to the main paper\n7. What if all eigenvectors are used for MAP, as in the BasisNet paper (k=all)? \n\nMinor comments:\n* line 176: “question that which…” \n* line 190 - can reference Table 2\n* 3.3.3 - heading should be “summary” not “summarization” \n* line 273: eigendecomposition complexity is cubic for dense Laplacians, whereas most real world graphs are sparse, so that is a worst case complexity. In practice it is lower.  \nLine 336: why does increasing k (8–>16) make performance worse? Shouldn’t the network just utilize the lower freq vectors if they are more important?\n\nLIMITATIONS:\nlimitations aren't discussed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new approach called Laplacian Canonization (LC) for ensuring the sign and basis invariance of spectral embeddings. This is done by determining the canonical direction of eigenvectors in the pre-processing stage. They propose to perform the Laplacian Canonization via Maximal Axis Projection (MAP) algorithm that is guaranteed to canonize all sign canonizable features. Experiments are performed on  molecular benchmarks.\n\nSTRENGTHS:\n- The paper is well written, contains clear rigorous definitions and is easy to follow. \n- Experiments show consistent improvements.\n\nWEAKNESSES:\nWith my limited knowledge of the topic, I could not point out any obvious weaknesses of the method. I do have two questions regarding the guarantees of the method: \n- The authors claim that the method can canonize more than 90% of all eigenvectors, however, this has only been tested on molecular graphs. Are there any guarantees for other graphs?\n- You observe that your assumptions are violated for some small percentage of eigenvectors on real-world datasets. I was wondering if these eigenvectors are evenly spread out over the eigenvalue spectrum or could it be the case that these are all for the lowest eigenvalues? This could be problematic since these are the eigenvectors one would like to use.\n\nQUESTIONS:\nBesides the questions, listed in the weaknesses section, I have a few more minor questions:\n- In the axis projection step, why do you need c?\n- I suppose that in practice alpha_i will not be exactly the same. Do you then consider all values or do you apply some binning algorithm to get B_i? I also wonder what is k in practice?\n- in Tables 4 and 5, why is it that MAP increases the number of parameters?\n\nLIMITATIONS:\nI have no limitations to point out and the code is provided in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe recently emerging graph transformers uses the spectral embedding. \nThe spectral embedding has two empirically known problems; I) sign invariance ii) basis invariance.  \nThe existing remedy for these problem comes at the cost.\nThis paper addresses this problem by the method this paper proposes called Laplacian Canonization (LC).\nThis paper provide the theoretical analyses of LC.\nAlso, the experimental results show that the LC outperforms the existing methods.\n\nSTRENGTHS:\n- The simplicity of the proposed algorithm. \n\n- Theoretical guarantee of the canonoization for the MAP algorithm.\n\nWEAKNESSES:\n- We do not know **why** the sign and basis invariance hinder the performance of PE.\nThus, I kind of feel little anxious that the readers cannot judge if the work on top of the series of these studies is in the right direction or not to address the fundamental problems. See also the limitation section.\n\n- Seeing the experiment, there is a little improvement from SignNet even though the LC deals with not only sign but also the basis invariance. \nWhy some method is not conducted with sign net? The basis invariance does not matter? Or The LC show the weaker accuracy performance regarding sign ambiguity than SignNet?\n\nQUESTIONS:\nRegarding Table 2.\nFrom the intuition, the basis invariance is quite rare, while the sign invariance always happens, since the basis invariance needs the \"shared eigenvalues.\"\nBut looking at the table 2, sign non-canonized is less than basis invariance. \nThus, how do you justify the definition of the canonization? Or basis invariance matters more?\nAlso, if the basis invariance is more frequent than sign invariance, how do you defend the little improvement from SignNet?\n\n\n\nBasis invariance vs. sign invariance. \nAs I raise the weakness section, the LC improves the accuracy of the SignNet only marginally. But seeing the Table 2, the there are more basis invariance. \nIs this an indirect evidence that the LC does not deal with sign not quite well? Or basis invaraince actually does not matter?\n\nIt will be interesting if the authors conduct the experiments using the \"separate\" MAP regarding basis and sign.\n\nThus, I believe that at least one of the following is happening\n\ni) some non-intuitional thing is happening in Table 2 regarding the number of noncanonization of sign v basis. \n\nii) Even if the number of non canonization of the basis, basis does not matter\n\niii) The proposed method deal with the sign case less well than SignNet.\n\nAs I wrote in this section, I am not clear the point how much the sign and basis matters, and how MAP addresses each invariance, and therefore I'm giving 4.\n\n---\nPOST REBUTTAL: Given the synthetic data experiment regarding basis, I slightly increase my score from 4 to 5. \n\n\nAlso, the discussion above on Table 2 contains my misunderstanding. Please see the discussion below for the details.\n\nLIMITATIONS:\nThe whole community does not know why the sign and basis invariance hamper the performance of PE as I raised in the weakness section, as far as I understand. This study is built on these studies. But although I read the Appendix A where the authors summarize the point of basis and invariance ambiguity, I'm less confident on this point.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Laplacian canonization, a way to select canonical Laplacian embeddings that resolve the sign and basis ambiguities often present in graph embeddings. The proposed method is a preprocessing step that is relatively fast. The authors perform experiments to evaluate the performance of Laplacian canonization.\n\nSTRENGTHS:\n- Originality: Laplacian canonization is an original idea. To the best of my knowledge, this is a novel contribution. \n\n- Quality: I see this paper as mainly making a methodological contribution. As such, the quality of the experiments are sufficiently extensive to be convincing, and the arguments/theoretical derivations are sound, to the best of my knowledge. \n\n- Clarity: The presentation of the paper and the motivations are clear.  \n\n- Significance: It is certainly very relevant and important to study graph embeddings for GNNs these days. The approach is novel and works well. It is significant enough to warrant publication at a venue like NeurIPS.\n\nWEAKNESSES:\n- One potential weakness is the theoretical portion of the paper, whose results are rather marginal and unsurprising. However, I see this as a methods paper, and the empirical good performance of the proposed method more than makes up for it. \n\n- One potential limitation of this approach is that not all eigenvectors can be canonized (even though in the datasets 90% of them can be). The authors seem to regard the 90% canonizable rate as a good feature of their approach, rather than a shortcoming. This is a fair perspective. But to provide a more balanced discussion, I would like to see the authors discuss more on this potential limitation, especially since 1. other methods for resolving ambiguities do not suffer from this issue and 2. it is unclear to me whether it is possible that the non-canonizable eigenvectors share any common patterns/structures in real datasets that might bias the results. \n\n- There is a literature (beyond GNN) that also considers spectral embeddings of graphs and going around the basis/sign ambiguity problems. For example, in point cloud registration, Lai and Zhao's \"Multiscale Nonrigid Point Cloud Registration Using Rotation-Invariant Sliced-Wasserstein Distance via Laplace-Beltrami Eigenmap. SIAM J. Imaging Sci. 10(2): 449-483 (2017)\", and in graph comparison Tam and Dunson's  \"Multiscale graph comparison via the embedded laplacian distance. arXiv preprint arXiv:2201.12064 (2022).\" I suggest incorporating these references and others to round out the prior work section in the appendix. \n\n\n- Typos: \n\nline 347: practical\n\nQUESTIONS:\n- See above section\n\nLIMITATIONS:\n- See above section on weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the Laplacian canonization approach to address the sign and basis ambiguities of eigenvectors. Previous sign- and basis-invariant methods suffer from high complexity and the proposed canonization method is light-weighted and can be used for any graph neural networks. Since the Laplacian canonization algorithm only runs in the pre-processing stage, it significantly reduces the forward and backward overhead of the neural networks. Experimental results on various graph classification datasets validate the effectiveness and efficiency of the proposed method.\n\nSTRENGTHS:\n1. The proposed canonization method is effective and efficient. Existing sign- and basis-invariant models suffer from high complexity. And this paper addresses this issue by proposing a new pre-processing algorithm, which not only reduces the training computation costs but also makes the model suitable for any graph neural network.\n\n2. This paper gives a theoretical framework for Laplacian canonization and detailly discusses the conditions for canonizing the sign and basis invariance of eigenvectors, which I think is important for this community. Based on the theoretical results, this paper proposes an efficient canonization algorithm that can heuristically determine the signs of eigenvectors.\n\n3. This paper is well-written and easy to follow.\n\nWEAKNESSES:\nIn the experiment, this paper tests the performance of different positional encoding methods with the same base model. To ensure fairness, the authors should ensure that different methods have similar model parameters. For example, in the PCBA dataset, the model parameters of GatedGCN-MAP are 2.5 times that of the GatedGCN-LapPE but the performance improvement is negligible. Besides, it is confusing that the parameters of PNA-None and GraphiT-None are fewer than PNA-MAP and GraphiT-MAP. Why removing the positional encoding will increase the number of parameters?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThis paper discusses the limitation of the proposed algorithm that not all eigenvectors can be canonized and shows that this situation does not have a potential impact on the model.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}