SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Why use spectral embedding: trade-off between universality and ambiguities. Sign and basis ambiguities arise only in spectral embeddings, so the first important question is why to use them. There are many kinds of positional encodings in the literature, such as Random Walk PE (RWPE), Position-aware Encoding, Relational Pooling (RP), random features, etc. Without addressing sign and basis ambiguities, some works even reported superior performance of RWPE over spectral embedding [18, 42]. However, these other PEs all have drawbacks either in their expressive power or permutation equivariance. RP and random features are universal, but they do not guarantee permutation equivariance, which is an important inductive bias in graph-structured data. RWPE and positional-aware encoding are permutation equivariant, but their expressive power is limited. Thus one may wonder whether there exists a kind of positional encoding method that achieves both. Spectral Embedding, as it turns out, is both permutation equivariant and universal (when coupled with eigenvalue information, see Sec 2). The good thing about SE is that through eigendecomposition, it transforms a "harder" permutation equivariance formulated by
$$
f \left(\boldsymbol {P} ^ {\top} \hat {\boldsymbol {A}} \boldsymbol {P}\right) = \boldsymbol {P} ^ {\top} f (\hat {\boldsymbol {A}}) \boldsymbol {P}, \tag {4}
$$
into a "easier" permutation equivariance formulated by
$$
f (\boldsymbol {P} \boldsymbol {U}) = \boldsymbol {P} f (\boldsymbol {U}), \tag {5}
$$
where  $P$  is an arbitrary permutation matrix,  $\hat{A}$  is the normalized adjacency matrix of the graph and  $U$  are its eigenvectors. Universality and equivariant results for equation 4 are only known for high-order tensor networks [32], but similar results for equation 5 have long been known to be achievable even by first-order networks such as DeepSets [63]. Thus SE enables efficient and universal graph neural networks that respects permutation equivariance. However, SE also brings new problems, sign and basis ambiguities. It is not fair to say models with SE are universal without achieving sign and basis invariance, since we expect graph neural networks to respect all symmetries among isomorphic graphs.
Sign and basis invariances are well-established problems in the literature. Sign and basis ambiguities have been recognized by numerous works in the literature to be challenging and important issues when incorporating SE as positional encoding. Below we list some existing discussions in these papers.
Quote from Dwivedi et al. [17]:
We propose an alternative to reduce the sampling space, and therefore the amount of ambiguities to be resolved by the network. Laplacian eigenvectors are hybrid positional and structural encodings, as they are invariant by node reparametrization. However, they are also limited by natural symmetries such as the arbitrary sign of eigenvectors (after being normalized to have unit length). The
number of possible sign flips is  $2^k$ , where  $k$  is the number of eigenvectors. In practice we choose  $k \ll n$ , and therefore  $2^k$  is much smaller than  $n!$  (the number of possible ordering of the nodes). During the training, the eigenvectors will be uniformly sampled at random between the  $2^k$  possibilities. If we do not seek to learn the invariance w.r.t. all possible sign flips of eigenvectors, then we can remove the sign ambiguity of eigenvectors by taking the absolute value. This choice seriously degrades the expressivity power of the positional features.
Quote from Kreuzer et al. [25]:
As noted earlier, there is a sign ambiguity with the eigenvectors. With the sign of  $\phi$  being independent of its normalization, we are left with a total of  $2^{k}$  possible combination of signs when choosing  $k$  eigenvectors of a graph. Previous work has proposed to do data augmentation by randomly flipping the sign of the eigenvectors [6, 16, 17], and although it can work when  $k$  is small, it becomes intractable for large  $k$ .
Quote from Beaini et al. [6]:
For instance, a pair of eigenvalues can have a multiplicity of 2 meaning that they can be generated by different pairs of orthogonal eigenvectors. For an eigenvalue of multiplicity 1, there are always two unit norm eigenvectors of opposite sign, which poses a problem during the directional aggregation. We can make a choice of sign and later take the absolute value. An alternative is to take a sample of orthonormal basis of the eigenspace and use each choice to augment the training.
Quote from Mialon et al. [34]:
Note that eigenvectors of the Laplacian computed on different graphs could not be compared to each other in principle, and are also only defined up to a  $\pm 1$  factor. While this raises a conceptual issue for using them in an absolute positional encoding scheme, it is shown in Dwivedi and Bresson [16]—and confirmed in our experiments—that the issue is mitigated by the Fourier interpretation, and that the coordinates used in LapPE are effective in practice for discriminating between nodes in the same way as the position encoding proposed in Vaswani et al. [51] for sequences. Yet, because the eigenvectors are defined up to a  $\pm 1$  factor, the sign of the encodings needs to be randomly flipped during the training of the network.
Quote from Dwivedi and Bresson [16]:
In particular, Dwivedi et al. [17] make the use of available graph structure to precompute Laplacian eigenvectors [7] and use them as node positional information. Since Laplacian PEs are generalization of the PE used in the original transformers [51] to graphs and these better help encode distance-aware information (i.e., nearby nodes have similar positional features and farther nodes have dissimilar positional features), we use Laplacian eigenvectors as PE in Graph Transformer. Although these eigenvectors have multiplicity occurring due to the arbitrary sign of eigenvectors, we randomly flip the sign of the eigenvectors during training, following Dwivedi et al. [17].
Quote from Dwivedi et al. [18]:
Another PE candidate for graphs can be Laplacian Eigenvectors [17, 16] as they form a meaningful local coordinate system, while preserving the global graph structure. However, there exists sign ambiguity in such PE as eigenvectors are defined up to  $\pm 1$ , leading to  $2^{k}$  number of possible sign values when selecting  $k$  eigenvectors which a network needs to learn. Similarly, the eigenvectors may be unstable due to eigenvalue multiplicities.
Quote from Lim et al. [29]:
However, there are nontrivial symmetries that should be accounted for when processing eigenvectors. If  $v$  is a unit-norm eigenvector, then so is  $-v$ , with the same eigenvalue. More generally, if an eigenvalue has higher multiplicity, then there are infinitely many unit-norm eigenvectors that can be chosen. Indeed, a full set of orthonormal eigenvectors is only defined up to a change of basis in each eigenspace. In the case of sign invariance, for any  $k$  eigenvectors there are  $2^k$  possible choices of sign. Accordingly, prior works randomly flip eigenvector signs during training in order to approximately learn sign invariance [25, 17]. However, learning all  $2^k$  invariances is challenging and limits the effectiveness of Laplacian eigenvectors for encoding positional information. Sign invariance is a special case of basis invariance when all eigenvalues are distinct, but higher dimensional basis invariance is even more difficult to deal with, and we show that these higher dimensional eigenspaces are abundant in real datasets.
Quote from Wang et al. [52]:
Srinivasan and Ribeiro [45] states that PE using the eigenvectors of the randomly permuted graph Laplacian matrix keeps permutation equivariant. Dwivedi et al. [17], Kreuzer et al. [25] argue that such eigenvectors are unique up to their signs and thus propose PE that randomly perturbs the signs of those eigenvectors. Unfortunately, these methods may have risks. They cannot provide permutation equivariant GNNs when the matrix has multiple eigenvalues, which thus are dangerous when applying to many practical networks. For example, large social networks, when not connected, have multiple 0 eigenvalues; small molecule networks often have non-trivial automorphism that may give multiple eigenvalues. Even if the eigenvalues are distinct, these methods are unstable. We prove that the sensitivity of node representations to the graph perturbation depends on the inverse of the smallest gap between two consecutive eigenvalues, which could be actually large when two eigenvalues are close.
Sign and basis invariances make the learning tasks easier. Permutation invariance is an important inductive bias in graphs in the sense that applying permutations on graphs results in isomorphic graphs and they should have the same properties and produce the same outputs. Empirically constraining the network to be invariant to such permutations benefits the performance, since otherwise the network has to learn these  $n!$  ambiguities by itself. Similarly, sign and basis ambiguities result in  $2^k$  and infinite possible choices of positional features respectively that a network has to learn, which could degrade its performance greatly. By preserving sign and basis invariance, a network will not need to learn the equivalence between all these possible choices and the learning task is significantly easier.
There are less ambiguity in signs than in permutations. For a given graph, there are  $n!$  possible ordering of the nodes that all represent the same graph, resulting in  $n!$  number of ambiguities. Considering that the vast majority of eigenvectors on real-world data have distinct eigenvalue (Table 9), when selecting  $k$  eigenvectors as positional encoding to present structural information, the number of ambiguities is  $2^k$ , which is much smaller than  $n!$ . This transformation from permutation ambiguity to sign ambiguity greatly reduces the amount of random noise in the input data and makes the model much more stable.
When sign invariance meets permutation equivariance. Sign ambiguity is not an issue when we do not take permutation symmetry into account. For each eigenvector pair  $\pm u$ , we can choose the one whose first non-zero entry is positive. This simple canonization solves the sign ambiguity problem with ease. The problem of sign ambiguity is only non-trivial when combined with permutation equivariance, that is, we require the canonization process to be both sign-invariant and permutation-equivariant. Since the eigenvectors can now be permuted arbitrarily, there is no "first" non-zero entry and the simple canonization above fails.
Moreover, such canonization only makes sense when we require it to be universal. For instance, mapping all eigenvectors to  $j = (1, 1, \dots, 1)$  is also a sign-invariant and permutation-equivariant canonization, but this canonization provides us with no information. Thus one may wonder whether a network can be both sign-invariant, permutation-equivariant and universal at the same time. Unfortunately, as far as we know, no existing work addresses all of them. The universality of SignNet only considers sign invariance, as stated in their theorem:
Theorem 6 (Universal representation for SignNet). A continuous function  $f\colon (\mathbb{S}^{n - 1})^k \to \mathbb{R}^s$  is sign invariant, i.e.  $f(s_{1}v_{1},\ldots ,s_{k}v_{k}) = f(v_{1},\ldots ,v_{k})$  for any  $s_i \in \{-1,1\}$ , if and only if there exists a continuous  $\phi \colon \mathbb{R}^n \to \mathbb{R}^{2n - 2}$  and a continuous  $\rho \colon \mathbb{R}^{(2n - 2)k} \to \mathbb{R}^s$  such that
$$
f (v _ {1}, \dots , v _ {k}) = \rho \bigl ([ \phi (v _ {i}) + \phi (- v _ {i}) ] _ {i = 1} ^ {k} \bigr).
$$
In fact, being sign-invariant, permutation-equivariant and universal at the same time is not even well-defined. As mentioned in the main text, for an uncanonicalizable eigenvector  $\mathbf{u}$  with  $\mathbf{u} = -\mathbf{P}\mathbf{u}$  for some permutation matrix  $\mathbf{P}$ , let  $f$  be an arbitrary continuous sign-invariant, permutation-equivariant and universal function and denote  $f(\mathbf{u}) = \mathbf{y}$ . The sign invariance property requires that  $f(-\mathbf{u}) = \mathbf{y}$  while the permutation equivariance property requires that  $f(P\mathbf{u}) = P\mathbf{y}$ . Since  $-\mathbf{u} = P\mathbf{u}$ , we have  $\mathbf{y} = P\mathbf{y}$ . However  $\mathbf{u} \neq P\mathbf{u}$ , thus universality is not satisfied, leading to a contradiction. This can be illustrated in Figure 1, where the three properties cannot be met at the same time.
Figure 1: The dilemma where permutation-equivariance, sign-invariance and universal expressive power cannot be achieved at the same time.
Remark. The universality in Definition 1 is defined on eigenvectors  $\mathbf{U} \in \mathbb{R}^{n \times n}$ . The permutation equivariance property of eigenvectors is defined as  $f(\mathbf{PU}) = \mathbf{Pf}(\mathbf{U})$ , and here universality implies that if two eigenvectors are not equal up to permutation, then  $f$  should be able to tell them apart. As mentioned above, such set-universal networks (like DeepSets or MLP) cannot be permutation equivariant and sign invariant at the same time. In the context of graphs, we may also care about universality defined on the adjacency matrix  $\mathbf{A} \in \mathbb{R}^{n \times n}$ , in which case the permutation equivariance property is defined as  $f(\mathbf{PAP}^{\top}) = \mathbf{Pf}(\mathbf{A})$ . Such graph-universal networks are always able to tell two non-isomorphic adjacency matrices apart. We can view  $f(\mathbf{A})$  as a function of  $\mathbf{U}$  (we omit the eigenvalues for simplicity), and we denote  $g(\mathbf{U}) = f(\mathbf{A})$ , then to achieve the graph universality of  $f$ ,  $g$  does not have to be set-universal, thus may not suffer from the dilemma above.
Much effort has been devoted to the structure of invariant and equivariant networks in the literature, but few has studied the case when invariance and equivariance combine (in this case you may have to sacrifice some invariance, or some equivariance, or some expressive power). Our work takes a step in this direction, providing some insights to this dilemma and characterize the necessary and sufficient conditions where these three properties conflict with each other.
Theoretical Expressivity It has been shown that neighborhood-aggregation based GNNs are not more powerful than the WL test [57, 35]. Since then, many attempts have been made to endow graph neural networks with expressive power beyond the 1-WL test, such as injecting random attributes [36, 44, 59, 45, 17], injecting positional encodings [28, 60], and designing higher-order GNNs [35, 32, 33, 31, 14].
Positional Encodings Recently, there are many works that aim to improve the expressive power of GNNs by positional encoding (PE). Murphy et al. [36] assigned each node an identifier depending on its index. Li et al. [28] proposed to use distances between nodes characterized by powers of the random walk matrix. You et al. [59] proposed to use distances of nodes to a sampled anchor set of nodes. Using random node features can also improve the expressiveness of GNNs, even making them universal [44, 1], but it has several defects: (1) loss of permutation invariance, (2) slower convergence, (3) poor generalization on unseen graphs [59, 30]. More detailed discussion can be found in Appendix E.
Laplacian PE Another PE candidate for graphs is Laplacian Eigenvectors [16, 17] which form a meaningful local coordinate system while preserving global structure. However, there exists
ambiguities in sign and basis that the network needs to learn, which harms the performance of GNNs. To address these ambiguities, many approaches have been proposed such as randomly flip the signs [17], use eigenfunctions over the edges [25] or use invariant network architectures [29].
Sign Invariance Several prior works have proposed ways of addressing sign ambiguity. A popular heuristic is Random Sign (RS), which randomly flips the signs of eigenvectors during training to encourage the insensitivity to different signs [17]. However, the network still has to learn these signs, leading to slower convergence and harder curve fitting task. Bro et al. [12] developed a data-dependent method to choose signs for each singular vector of a singular value decomposition. Still, in the worst case the signs chosen will be arbitrary, and they do not handle rotational ambiguities in higher dimensional eigenspaces. Kreuzer et al. [25] proposed to use the relative Laplace embeddings of two nodes. However, their approach suffers from a major computational bottleneck with  $\mathcal{O}(n^4)$  complexity. Lim et al. [29] proposed SignNet that passes both  $\pmb{u}$  and  $-\pmb{u}$  to the same network, adds the outputs together and then passes them to another network. As both outputs of  $\pmb{u}$  and  $-\pmb{u}$  need to be computed, their approach adds to the training cost. Our approach happens at pre-processing time with  $\mathcal{O}(n^3)$  complexity.
Basis Invariance The only works we know addressing basis ambiguity are Lim et al. [29] and Huang et al. [21]. Lim et al. [29] proposed BasisNet that is basis invariant. However, to achieve universality, the components of BasisNet need to use higher order tensors in  $\mathbb{R}^{n^k}$  where  $k$  can be as large as  $\frac{n(n-1)}{2}$  [33], rendering BasisNet impractical. Huang et al. [21] generalized BasisNet and proposed SPE that is both basis-invariant and stable to perturbation to the Laplacian matrix, but it still suffers from exponential complexity as BasisNet. In this paper, we showed that under certain assumptions, the issue of basis ambiguity can be resolved more efficiently in the pre-processing stage.
There is also a literature beyond GNN that also considers spectral embeddings of graphs and going around the sign/basis ambiguity problems. Lai and Zhao [26] proposed to address sign/basis ambiguities using optimal transport theory that involves solving a non-convex optimization problem, thus it could be less efficient than our approach. Tam and Dunson [48] proposed to symmetrize the embedding using a heuristic measure called ELD that is quite similar to the form of SignNet, while our MAP algorithm offers an axis projection approach and establish its theoretical guarantees.
Canonical Forms The theory of canonical forms has been widely studied in mathematics [39] and applied to many fields of machine learning. Sajnani et al. [43] proposed a self-supervised method named ConDor that learns to canonicalize the 3D orientation and position for full and partial 3D point clouds. Agaram et al. [2] presented Canonical Field Network (CaFi-Net), a self-supervised method to canonicalize the 3D pose of instances from an object category represented as neural fields, specifically neural radiance fields (NeRFs). Sun et al. [47] proposed an unsupervised capsule architecture for 3D point clouds. Kaba et al. [23] proposed to decouple the equivariance and prediction components of neural networks by predicting a transformation that can be used to align the input data to some canonical pose. After this alignment the remaining layers do not have to be equivariant anymore. However, as far as we know, no existing work addresses the issue of ambiguities, which arises when some function (e.g., eigendecomposition) is required to be both equivariant on the input space and invariant on the output space. No existing work addresses the canonizability of input data, which only arises when the invariance and equivariance property conflicts. Instead, we develop a novel theoretical framework that addresses these issues.
C Implementation and verification of MAP
C.1 Complete pseudo-code of MAP
A simplified pseudo-code for sign canonization with MAP is shown in Algorithm 1. A simplified pseudo-code for basis canonization with MAP is shown in Algorithm 2.  $(j = (1,1,\dots ,1)\in \mathbb{R}^n)$
The complete pseudo-code of MAP is shown in Algorithm 3. Despite the sophisticated workflow, programmers do not have to know the principles of our algorithm to use it. The entire module can be passed as a pre_transform function to the dataset class with a single line of code.
There are three steps in Algorithm 3. The first is to eliminate sign ambiguity with  $\mathcal{O}(n^2\log n)$  time complexity. The second is to eliminate basis ambiguity with  $\mathcal{O}(n^{2}dm)$  time complexity, where  $d$  is the multiplicity of the eigenvalue and  $m$  is the number of multiple eigenvalues of  $\hat{\pmb{A}}$ . The third is
Algorithm 1 Maximal Axis Projection for eliminating sign ambiguity
Require: Input graph  $\mathcal{G} = (\mathbb{V},\mathbb{E},\boldsymbol {X})$    
Ensure: Spectral embedding of  $\mathcal{G}$    
Calculate the eigendecomposition  $\hat{\pmb{A}} = \pmb {U}\pmb {\Lambda}\pmb{U}^{\top}$ $\triangleright \mathcal{O}(n^{3})$  complexity   
for each eigenvector  $\pmb{u}$  do  $\triangleright \mathcal{O}(n^{2}\log n)$  complexity   
 $\pmb{x}_i\gets \sum_{e_j\in \mathcal{B}_i}\pmb {e}_j + c\pmb {j},i = 1,\dots ,k$ $\pmb{x}_h\gets$  non-orthogonal summary vector with smallest  $h$  (equation 1)   
if  $\pmb{x}_h$  is not <none> then   
 $\pmb {u}\gets \pmb{u}^{*}$  (choose the direction with  $\pmb {u}^{\top}\pmb{x}_h > 0$  as in equation 2)   
else   
 $\pmb {u}\gets \pmb{u}$  (no canonization)   
end if   
end for   
return U
Algorithm 2 Maximal Axis Projection for eliminating basis ambiguity
Require: Eigenvalue  $\lambda$  with multiplicity  $d > 1$    
Ensure: Spectral embedding corresponding to  $\lambda$    
Calculate the eigenvectors  $\pmb {U}\in \mathbb{R}^{n\times d}$  of  $\lambda$  through eigendecomposition  $\triangleright \mathcal{O}(n^3)$  complexity for  $i = 1,2,\ldots ,d$  do  $\triangleright \mathcal{O}(n^2 d)$  complexity  $\pmb{x}_i\gets \sum_{e_j\in \mathcal{B}_i}\pmb {e}_j + c\pmb{j}$  Choose  $\pmb {u}_i\in \langle \pmb {u}_1,\dots ,\pmb {u}_{i - 1}\rangle^{\perp},|\pmb {u}_i| = 1$  s.t.  $f(\pmb {u}_i) = \pmb {u}_i^\top \pmb {x}_i$  is maximized (equation 3)   
end for   
return  $U_0\coloneqq [u_1,\dots ,u_d]$
to incorporate eigenvalue information with  $\mathcal{O}(n^2)$  time complexity. The overall time complexity is  $\mathcal{O}(n^3)$  with the bottleneck being the eigendecomposition.
The second part of our algorithm (eliminating basis ambiguity) has time complexity  $\mathcal{O}(n^2 dm)$ . We point out that in real-world datasets  $m$  is often quite small. As shown in Table 9, multiple eigenvalues only make up around  $5\%$  of all eigenvalues, thus in practice the time complexity of the second part is  $\mathcal{O}(n^2 d)$ , better than eigendecomposition. (Note it is important to take floating-point errors into account when counting these multiple eigenvalues)
Table 9: The number of multiple eigenvalues in real-world datasets.
Dataset	ogbg-molesol	ogbg-molfreesolv	ogbg-molhiv	ogbg-mollico	ogbg-moltox21	ogbg-moltoxcast	ogbg-molpcba
#multiple eigenvalues	738	286	52367	5391	8772	10556	491247
#all eigenvalues	13420	4654	952055	104669	129730	141042	10627757
Ratio	5.50%	6.15%	5.50%	5.15%	6.76%	7.48%	4.62%
Some parts of Algorithm 3 have time complexity  $\mathcal{O}(nd^4)$ . We also point out that  $d$  is usually small in real datasets. We show the number of eigenvalues and their multiplicities in logarithmic scale in Figure 2.
The details of the datasets are listed in Table 10.
C.2 Verifying the correctness of Theorem 3
We verify the correctness of Theorem 3 through random simulation. The program is shown in Algorithm 4. Let  $U$  be a random orthonormal matrix,  $P$  be a random permutation matrix and  $S$  be a random sign matrix (diagonal matrix of 1 and -1). We pass  $U$ ,  $PU$ ,  $US$ ,  $PUS$  to the UNIQUE SIGN function (Algorithm 1) and compare the outputs. If our algorithm is correct,  $U$  and  $US$  should have invariant outputs, while  $U$ ,  $PU$  and  $PUS$  should have equivariant outputs.
We conduct 1000 trials. The results are  $p\_correct = q\_correct = pq\_correct = total = 1000$ , showing that Algorithm 1 is both permutation-equivariant and unique (unambiguous).