SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiangyan Ma1* Yifei Wang2* Yisen Wang3,4†
$^{1}$  School of Electronics Engineering and Computer Science, Peking University
$^{3}$  National Key Lab of General Artificial Intelligence,
Spectral embedding is a powerful graph embedding technique that has received a lot of attention recently due to its effectiveness on Graph Transformers. However, from a theoretical perspective, the universal expressive power of spectral embedding comes at the price of losing two important invariance properties of graphs, sign and basis invariance, which also limits its effectiveness on graph data. To remedy this issue, many previous methods developed costly approaches to learn new invariants and suffer from high computation complexity. In this work, we explore a minimal approach that resolves the ambiguity issues by directly finding canonical directions for the eigenvectors, named Laplacian Canonization (LC). As a pure pre-processing method, LC is light-weighted and can be applied to any existing GNNs. We provide a thorough investigation, from theory to algorithm, on this approach, and discover an efficient algorithm named Maximal Axis Projection (MAP) that works for both sign and basis invariance and successfully canonizes more than  $90\%$  of all eigenvectors. Experiments on real-world benchmark datasets like ZINC, MOLTOX21, and MOLPCBA show that MAP consistently outperforms existing methods while bringing minimal computation overhead. Code is available at https://github.com/PKU-ML/LaplacianCanonization.
Despite the popularity of Graph Neural Networks (GNNs) for graph representation learning [55, 56, 54, 27, 13], it is found that many existing GNNs have limited expressive power and cannot tell the difference between many non-isomorphic graphs [57, 35]. Existing approaches to improve expressive power, such as high-order GNNs [35] and subgraph aggregation [9], often incur significant computation costs over vanilla GNNs, limiting their use in practice. In comparison, a simple but effective strategy is to use discriminative node identifiers. For example, GNNs with random features can lead to universal expressive power [44, 1]. However, these unique node identifiers often lead to the loss of the permutation invariance property of GNNs, which is an important inductive bias of graph data that matters for sample complexity and generalization [24]. Therefore, during the pursuit of more expressive GNNs, we should also maintain the invariance properties of graph data. Balancing these two conflicting demands presents a significant challenge for the development of advanced GNNs.
Spectral embedding (SE) is a classical approach to encode node positions using eigenvectors  $\mathbf{U}$  of the Laplacian matrix  $\mathbf{L}$ , which has the advantage of being expressive and permutation equivariant.
Table 1: Comparison between prior works and our method.  $n$  is the number of nodes,  $m$  is the exponent of the feature dimension of BasisNet [29, 33].
Method	pre-processing time	universality	permutation invariance	addresses sign ambiguity	addresses basis ambiguity	feature dimension
LapPE [17]	O(n3)	x	✓	x	x	n
RandSign [17]	O(n3)	x	✓	✓	x	n
SAN [25]	O(n4)	✓	✓	✓	x	3n
SignNet [29]	O(n3)	✓	✓	✓	x	2n
BasisNet [29]	O(n3)	✓	✓	✓	✓	nm
MAP (ours)	O(n3)	✓	✓	✓	✓	n
GNNs using SE can attain universal expressive power (distinguishing any pair of non-isomorphic graphs) even under simple architectures (results in Section 2). However, spectral embedding faces two additional challenges in preserving graph invariance: sign ambiguity and basis ambiguity, due to the non-uniqueness of eigendecomposition. These ambiguities could lead to inconsistent predictions for the same graph under different eigendecompositions.
Many methods have been proposed to address sign and basis ambiguities of spectral embedding. A popular heuristic is RandSign [17] which randomly flips the signs of the eigenvectors during training. Although it is simple to use, this data augmentation approach does not offer any formal invariance guarantees and can lead to slower convergence due to all possible  $2^{n}$  sign flips. An alternative involves using sign-invariant eigenfunctions to attain sign invariance [25], which significantly increases the time complexity to  $\mathcal{O}(n^4)$ . Another solution is to design extra GNN modules for sign and basis invariant embeddings, e.g., SignNet and BasisNet [29], which can also add a substantial computational burden. Therefore, as summarized in Table 1, existing spectral embedding methods are all detrimental in a certain way that either hampers sign and basis invariance, or induces large computational overhead. More discussions about the related work can be found in Appendix B.
In this work, we explore a new approach called Laplacian Canonization (LC) that resolves the ambiguities by identifying a unique canonical direction for each eigenvector, amongst all its sign and basis equivalents. Although it is relatively easy to find a canonization rule that work for certain vectors, up to now, there still lacks a rigorous understanding of what kinds of vectors are canonizable and whether we could find a complete algorithm for all canonizable features. In this paper, we systematically answer this problem by developing a general theory for Laplacian canonization and characterizing the sufficient and necessary conditions for sign and basis canonizable features.
Based on these theoretical properties, we propose a practical canonization algorithm for sign and basis invariance, called Maximal Axis Projection (MAP), that adopts the permutation-invariant axis projection functions to determine the canonical directions. We theoretically characterize the conditions under which MAP can guarantee sign and basis canonization, and empirically verify that this condition holds for most synthetic and real-world graphs. It is worth noting that LC is a lightweight approach since it is only a pre-processing method and does not alter the dimension of the spectral embedding. Empirically, we show that employing the MAP-canonized spectral embedding yields significant improvements over the vanilla RandSign approach, and even matches SignNet on large-scale benchmark datasets like OGBG [20]. We summarize our contributions as follows:
- We explore Laplacian Canonization (LC), a new approach to restoring the sign and basis invariance of spectral embeddings via determining the canonical direction of eigenvectors in the pre-processing stage. We develop a general theoretical framework for LC and characterize the canonizability of sign and basis invariance.
- We propose an efficient algorithm for Laplacian Canonization, named Maximal Axis Projection (MAP), that works well for both sign and basis invariance. In particular, we show that MAP-sign is capable of canonizing all sign canonizable eigenvectors and thus is complete. The assessment of its feasibility shows that MAP can effectively canonize almost all eigenvectors on random graphs and more than  $90\%$  eigenvectors on real-world datasets.
- We evaluate the MAP-canonicalized spectral embeddings on graph classification benchmarks including ZINC, MOLTOX21 and MOLPCBA, and obtain consistent improvements over previous spectral embedding methods while inducing the smallest computational overhead.
2 Benefits and Challenges of Spectral Embedding for GNNs
Denote a graph as  $\mathcal{G} = (\mathbb{V},\mathbb{E},\boldsymbol {X})$  where  $\mathbb{V}$  is the vertex set of size  $n$ ,  $\mathbb{E}$  is the edge set, and  $\boldsymbol {X}\in \mathbb{R}^{n\times d}$  are the input node features. We denote  $\pmb{A}$  as the adjacency matrix, and let  $\hat{\pmb{A}} = D^{-\frac{1}{2}}\tilde{\pmb{A}} D^{-\frac{1}{2}} = D^{-\frac{1}{2}}(\pmb {I} + \pmb {A})D^{-\frac{1}{2}}$  be the normalized adjacency matrix, where  $\pmb{I}$  denotes the augmented self-loop,  $\pmb{D}$  is the diagonal degree matrix of  $\tilde{\pmb{A}}$  defined by  $D_{i,i} = \sum_{j = 1}^{n}\tilde{A}_{i,j}$ . A graph function  $f([X,\hat{A}])$  is permutation invariant if for all permutation matrix  $\pmb {P}\in \mathbb{R}^{n\times n}$ , we have  $f([PX,P\hat{AP}^{\top}]) = f([X,\hat{A}])$ . Similarly,  $f$  is permutation equivariant if  $f([PX,P\hat{AP}^{\top}]) = Pf([X,\hat{A}])$ .
Spectral Embedding (SE). Considering the limited expressive power of MP-GNNs, recent works explore more flexible GNNs like graph Transformers [58, 62, 16, 25, 42]. These models bypass the explicit structural inductive bias in MP-GNNs while encoding graph structures via positional embedding (PE). A popular graph PE is spectral embedding (SE), which uses the eigenvectors  $\mathbf{U}$  of the Laplacian matrix  $\mathbf{L} = \mathbf{I} - \hat{\mathbf{A}}$  with eigendecomposition  $\mathbf{L} = \mathbf{U}\Lambda \mathbf{U}^{\top}$ , where  $\Lambda = \mathrm{diag}(\lambda)$  is the diagonal matrix of ascending eigenvalues  $\lambda_{1} \leq \dots \leq \lambda_{n}$ , and the  $i$ -th column of  $\mathbf{U}$  is the eigenvector corresponding to  $\lambda_{i}$ . It is easy to see that spectral embedding is permutation equivariant: for any node permutation  $\mathbf{P}$  of the graph,  $\mathbf{P}\mathbf{U}$  is the spectral embedding of the new Laplacian since  $\mathbf{PLP}^{\top} = (\mathbf{P}\mathbf{U})\Lambda (\mathbf{P}\mathbf{U})^{\top}$ . Therefore, a permutation-invariant GNN (e.g., DeepSets [63], GIN [57], Graph Transformer [16]) using the SE-augmented input features  $\tilde{\mathbf{X}} = [\mathbf{X},\mathbf{U}]$  remains permutation invariant.
Reweighted Spectral Embedding (RSE). Previous works have shown that using spectral embedding improves the expressive power of MP-GNNs [6]. Nonetheless, we find that SE alone is insufficient to approximate an arbitrary graph function, as it does not contain all information about the graph, in particular, the eigenvalues. Consider two non-isomorphic graphs whose Laplacian matrices have identical eigenvectors but different eigenvalues. As their SE is the same, a network only using SE cannot tell them apart. To resolve this issue, we propose reweighted spectral embedding (RSE) that additionally reweights each eigenvector  $\pmb{u}_i$  with the square-root of its corresponding eigenvalue  $\lambda_i$ , i.e.,  $U_{\mathrm{RSE}} = U\Lambda^{\frac{1}{2}}$ . With the reweighting technique, RSE incorporates eigenvalue information without need extra dimensions.
Universality of RSE. In the following theorem, we prove that with RSE, any universal network on sets (e.g., DeepSets [63] and Transformer [61]) is universal on graphs while preserving permutation invariance. All proofs are deferred to Appendix K.
Theorem 1. Let  $\Omega \subset \mathbb{R}^{n\times d}\times \mathbb{R}^{n\times n}$  be a compact set of graphs,  $[\pmb {X},\hat{\pmb{A}} ]\in \Omega$  . Let NN be a universal neural network on sets. Given any continuous invariant graph function  $f$  defined over  $\Omega$  and arbitrary  $\varepsilon >0$  , there exist a set of NN parameters such that for all graphs  $[\pmb {X},\hat{\pmb{A}} ]\in \Omega$
$$
| \operatorname {N N} ([ X, U _ {\text {R S E}} ]) - f ([ X, \hat {A} ]) | <   \varepsilon .
$$
As far as we know, this theorem is the first to show that, with the help of graph embeddings like RSE, even an MLP network like DeepSets [63] (composed of a node-wise MLP, a global pooling layer, and a graph-level MLP) can achieve universal expressive power to distinguish any pair of non-isomorphic graphs. Notably, it does not violate the NP-hardness of graph isomorphism testing, since training NN itself is known to be NP-hard [10]. Actually, it is not always necessary to use all spectra of the graph. Existing studies find that high-frequency components are often unhelpful, or even harmful for representation learning [5, 19]. Thus, in practice, we only use the first  $k$  low-frequency components of RSE. An upper bound on the approximation error of truncated RSE can be found in Appendix K.10.
Ambiguities of eigenvectors. Although RSE enables universal GNNs, there exist two types of ambiguity in eigenvectors that hinder their applications. The first one, known as sign ambiguity, arises when an eigenvector  $\pmb{u}_{\lambda_i}$  corresponding to eigenvalue  $\lambda_{i}$  is equally valid with its sign flipped, i.e.,  $-\pmb{u}_{\lambda_i}$  is also an eigenvector corresponding to  $\lambda_{i}$ . The second one, termed basis ambiguity, occurs when eigenvalues with multiplicity degree  $d_{i} > 1$  can have any other orthogonal basis in the subspace spanned by the corresponding eigenvectors as valid eigenvectors. To be specific, for multiple eigenvalues with multiplicity degree  $d_{i} > 1$ , the corresponding eigenvectors  $U_{\lambda_i} \in \mathbb{R}^{n\times d_i}$  form an orthonormal basis of a subspace. Then any orthonormal matrix  $Q \in \mathbb{R}^{d_i\times d_i}$  can be applied to  $U_{\lambda_i}$  to generate a new set of valid eigenvectors for  $\lambda_{i}$ . Because of these ambiguities, we can get distinct GNN outputs for the same graph, resulting in unstable and suboptimal performance [17, 25, 18, 29]. In Appendix A, we elaborate the challenges posed by these ambiguities.
Table 2: The ratio of uncanonizable eigenvectors w.r.t. each invariance property with our MAP algorithm on three real-world datasets: ZINC, MOLTOX21, and MOLPCBA.
Invariance	ZINC	MOLTOX21	MOLPCBA
Sign	2.46 %	3.04 %	2.24 %
Basis	1.59 %	3.31 %	7.37 %
Total	4.05 %	6.35 %	9.61 %
3 Laplacian Canonization for Sign and Basis Invariance
Rather than incorporating additional modules to learn new sign and basis invariants [25, 29], we explore a straightforward, learning-free approach named Laplacian Canonization (LC). The general idea of LC is to determine a unique direction for each eigenvector  $\pmb{u}$  among all its sign and basis equivalents. In this way, the ambiguities can be directly addressed in the pre-processing stage. For example, for two sign ambiguous vectors  $\pmb{u}$  and  $-\pmb{u}$ , we aim to find an algorithm that determines a unique direction among them to obtain sign invariance. Despite some naïve canonization rules (discussed in Appendix G.1), there still lacks a systematical understanding of the following key questions of LC:
1. What kind of canonization algorithm should we look for? (Section 3.1)
2. What kind of eigenvectors are canonizable or non-canonical? (Section 3.2)
3. Is there an efficient canonization algorithm for all canonizable features? (Section 3.3)
In this section, we answer the three problems by establishing the first formal theory of Laplacian canonization and characterizing the canonizability for sign and basis invariance; based on these analyses, we also propose an efficient algorithm named MAP for LC that is guaranteed to canonize all sign canonizable features. To get a glimpse of our final results, Table 2 shows that MAP can resolve both sign and basis ambiguities for more than  $90\%$  of all eigenvectors on real-world data.
3.1 Definition of Canonization
To begin with, we first find out what properties a desirable canonization algorithm should satisfy. The ultimate goal of canonization is to eliminate ambiguities by selecting a canonical form among these ambiguous outputs. Generally speaking, we can characterize ambiguity as a multivalued function  $f\colon \mathcal{X}\to \mathcal{Y}$ , i.e., there could be multiple outputs  $y_{1},\ldots ,y_{n}$  corresponding to the same input  $x$ . In our sign/basis ambiguity case,  $f$  refers to a mapping from a graph  $x\in \mathcal{X}$  to the eigenvectors of a certain eigenvalue via the ambiguous eigendecomposition. The ambiguous eigenvectors are not independent; they are related by a sign/basis transformation. In general, we can assume that all possible outputs  $y_{1},\dots ,y_{n}$  of any  $x$  belong to the same equivalence class induced by some group action  $g\in G$ , where  $G$  acts on  $\mathcal{V}$ . That is, if  $f(x) = y_{1} = y_{2}$ , then there exists  $g\in G$  such that  $y_{1} = gy_{2}$ . Moreover, eigenvectors of graphs obey a fundamental symmetry: permutation equivariance. In general, we assume that  $f$  is equivariant to a group  $H$  acting on  $\mathcal{X}$  and  $\mathcal{V}$ . That is, for any  $h\in H$ , if  $y_{1},\ldots ,y_{n}$  are all possible outputs of  $x$ , then  $hy_1,\ldots ,hy_n$  are all possible outputs of  $hx$ .
Specifically, for the goal of canonizing ambiguous eigenvectors, i.e., Laplacian canonization, we are interested in algorithms invariant to sign/basis transformations in the corresponding orthogonal group  $G(O(1)$  for sign invariance and  $O(d)$  for basis invariance). Meanwhile, to preserve the symmetry of graph data, this algorithm should also maintain the permutation equivariant property of eigenvectors (Section 2) w.r.t. the permutation group  $H$ . Third, it also should still be discriminative enough to produce different canonical forms for different graphs (like the original spectral embedding). Combining these desiderata, we have a formal definition of canonization.
Definition 1. A mapping  $\mathcal{A}\colon \mathcal{Y}\to \mathcal{Y}$  is called a  $(f,G,H)$ -canonization when it satisfies:
-  $\mathcal{A}$  is  $G$ -invariant:  $\forall y \in \mathcal{V}, g \in G, \mathcal{A}(y) = \mathcal{A}(gy)$ ;
-  $\mathcal{A}$  is  $H$ -equivariant:  $\forall x \in \mathcal{X}, h \in H, \mathcal{A}\big(f(hx)\big) = h\mathcal{A}\big(f(x)\big)$ ;
-  $\mathcal{A}$  is universal:  $\forall x \in \mathcal{X}, h \in H, x \neq hx \Rightarrow \mathcal{A}\big(f(x)\big) \neq \mathcal{A}\big(f(hx)\big)$ .
3.2 Theoretical Properties of Canonization
Following Definition 1, we are further interested in the question that whether any eigenvector  $\pmb{u}$  is canonizable, i.e., there exists a canonization that can determine its unique direction. Unfortunately, the answer is NO. For example, the vector  $\pmb{u} = (1, -1)$  cannot be canonized by any canonization algorithm, since a permutation of  $\pmb{u}$  gives  $(-1, 1)$  that equals to  $-\pmb{u}^3$ . But as long as there is only a small number of uncanonical eigenvectors like  $\pmb{u}$ , a canonization algorithm can still resolve the ambiguity issue to a large extent.
Therefore, we are interested in the fundamental question of which eigenvectors are canonizable. The following theorem establishes a general necessary and sufficient condition of the canonizability for general groups  $G, H$ , which may be of independent interest.
Theorem 2. An input  $x \in \mathcal{X}$  is canonizable on the embedding function  $f$  iff there does not exist  $h \in H$  and  $g \in G$  such that  $x \neq hx$  and  $f(hx) = gf(x)$ .
This theorem states that for inputs from the same equivalence class in  $\mathcal{X}$  (induced by  $H$ ), as long as they are not mapped to the same equivalence class in  $\mathcal{Y}$  (induced by  $G$ ), these inputs are canonizable. In particular, by applying Theorem 2 to the specific group  $G$  induced by sign/basis invariance, we can derive some simple rules to determine whether there exists a canonizable rule for given eigenvector(s).
Corollary 1 (Sign canonizability). A vector  $\mathbf{u} \in \mathbb{R}^n$  is canonizable under sign ambiguity iff there does not exist a permutation matrix  $\mathbf{P} \in \mathbb{R}^{n \times n}$  such that  $\mathbf{u} = -\mathbf{P}\mathbf{u}$ .
Corollary 2 (Basis canonizability). The base eigenvectors  $\mathbf{U} \in \mathbb{R}^{n \times d}$  of the eigenspace  $V$  are canonizable under basis ambiguity iff there does not exist a permutation matrix  $\mathbf{P} \in \mathbb{R}^{n \times n}$  such that  $\mathbf{U} \neq \mathbf{PU}$  and  $\operatorname{span}(\mathbf{PU}) = \operatorname{span}(\mathbf{U}) = V$ .
Remark. From a probabilistic view, almost all eigenvectors are canonizable. Let  $\mathbf{U} \in \mathbb{R}^{n \times d}$  be basis vectors sampled from a continuous distribution in  $\mathbb{R}^n$ , then  $\operatorname*{Pr}\{\mathbf{U}$  is canonizable\} = 1.
In the next subsection, we will further design an efficient algorithm to canonize all sign and basis canonizable eigenvectors in the pre-processing stage, so the network does not have to bear the ambiguities of these eigenvectors.
3.3 MAP: A Practical Algorithm for Laplacian Canonization
Built upon theoretical properties in Section 3.2, we aim to design a general, powerful, and efficient canonization to resolve both sign and basis ambiguities for as many eigenvectors as possible. Here, we choose to adopt axis projection as the basic operator in our canonization algorithm named Maximal Axis Projection (MAP). The key observation is that the standard basis vectors (i.e., the axis) of the Euclidean space  $\pmb{e}_i \in \mathbb{R}^n$  are permutation equivariant, and in the meantime, the eigenspace spanned by the eigenvectors  $V = \operatorname{span}(U)$  are also permutation equivariant. This means that when projecting the axis to the eigenspace, the obtained angles are also permutation equivariant, based on which we could apply permutation invariant functions (such as max) to obtain permutation invariant statistics that can be used for canonization. Meanwhile, due to the generality of projection, it can be applied to both sign ambiguity (for a single eigenvector) and basis ambiguity (for the eigenspace with multiple eigenvectors). We provide illustrative examples to help understand this algorithm in Appendix F.
**Preparation step:** Axis projection. Consider unit eigenvector(s)  $U \in \mathbb{R}^{n \times d}$  corresponding to an eigenvalue  $\lambda$  with geometric multiplicity  $d \geq 1$ . These eigenvectors span a  $d$ -dimensional eigenspace  $V = \operatorname{span}(U)$  with the projection matrix  $P = U U^{\top}$ . To start with, we calculate the projected angle between  $V$  and each standard basis (i.e., the axis) of the Euclidean space  $e_i$  (a one-hot vector whose  $i$ -th element is 1), i.e.,  $\alpha_i = |Pe_i|, i = 1, \dots, n$ . Assume that there are  $k$  distinct values in  $\{\alpha_i, i = 1, \dots, n\}$ , according to which we can divide all basis vectors  $\{e_i\}$  into  $k$  disjoint groups  $\mathcal{B}_i$  (arranged in descending order of the distinct angles). Each  $\mathcal{B}_i$  represents an equivalent class of axes that  $u_i$  has the same projection on. Then, we define a summary vector  $x_i$  for the axes in each group  $\mathcal{B}_i$  as their total sum  $x_i = \sum_{e_j \in \mathcal{B}_i} e_j + c$ , where  $c$  is a tunable constant. Next, we introduce how to adopt these summary vectors to canonize the eigenvectors for sign and basis invariance, respectively.
Accordingly, for any  $x \in \mathcal{X}$ , if there exists a canonization  $\mathcal{A}$  for  $x$ , we say  $x$  is canonizable.
<sup>3</sup>In this case, sign invariance conflicts with permutation equivariance: making  $\mathbf{u}$  and  $-\mathbf{u}$  output the same vector violates permutation equivariance, and vice versa.
3.3.1 Sign Canonization with MAP
Step 1. Find non-orthogonal axis. For sign canonization, we calculate the angles between the eigenvalue  $\pmb{u}$  and each summary vector  $\pmb{x}_i$  one by one, and terminate the procedure as long as we find a summary vector  $\pmb{x}_h$  with non-zero angle  $\alpha_h = \pmb{u}^\top \pmb{x}_h \neq 0$ , and return <none> otherwise.
$$
\boldsymbol {x} _ {h} = \left\{ \begin{array}{l l} \min  _ {i} \Phi , & \text {i f} \Phi \neq \varnothing , \\ <   \text {n o n e} >, & \text {i f} \Phi = \varnothing , \end{array} \right. \text {w h e r e} \Phi = \left\{i \mid \boldsymbol {u} ^ {\top} \boldsymbol {x} _ {i} \neq 0 \right\}. \tag {1}
$$
Assumption 1. There exists a summary vector  $\pmb{x}_h$  that is non-orthogonal to  $\pmb{u}$ , i.e.,  $\pmb{u}^\top \pmb{x}_h \neq 0$
Step 2. Sign canonization. As long as Assumption 1 holds, we can utilize  $\boldsymbol{x}_h$  to determine the canonical direction  $\boldsymbol{u}^*$  by requiring its projected angle to be positive, i.e.,
$$
\boldsymbol {u} ^ {*} = \left\{ \begin{array}{l l} \boldsymbol {u}, & \text {i f} \boldsymbol {u} ^ {\top} \boldsymbol {x} _ {h} > 0, \\ - \boldsymbol {u}, & \text {i f} \boldsymbol {u} ^ {\top} \boldsymbol {x} _ {h} <   0. \end{array} \right. \tag {2}
$$
We call this algorithm MAP-sign, and summarize it in Algorithm 1 in Appendix C.1. The following theorem proves that it yields a valid canonization for sign invariance under Assumption 1.