SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3 Maximal Axis Projection
Require: Graph  $\mathcal{G} = (\mathbb{V},\mathbb{E})$  , its normalized adjacency matrix  $\hat{A}$    
Ensure: Maximal Axis Projection of  $\mathcal{G}$    
Calculate the eigendecomposition  $\hat{A} = U\Lambda U^{\top}$ $\triangleright O(n^{3})$  complexity   
for each single eigenvector  $\pmb {u}\in \mathbb{R}^n$  do  $\triangleright O(n^{2}\log n)$  complexity   
proj  $\leftarrow (|\boldsymbol {u}\boldsymbol {u}^{\top}\boldsymbol {e}_1|,|\boldsymbol {u}\boldsymbol {u}^{\top}\boldsymbol {e}_2|,\dots ,|\boldsymbol {u}\boldsymbol {u}^{\top}\boldsymbol {e}_n|\right)$ $\triangleright O(n)$  complexity   
len, ind  $\leftarrow$  SORT(proj)  $\triangleright O(n\log n)$  complexity   
len  $\leftarrow$  UNIQUE(len)   
 $k\gets |len|$    
for  $i = 1,2,\ldots ,k$  do  $\triangleright O(n)$  complexity   
 $\pmb{x_i}\gets \sum_j\pmb{e}_{ind[j]}$  such that projind[j] = len[i]   
 $\pmb{x_i}\gets \pmb{x_i} + c\pmb{j}$ $\pmb{u_0}\gets$  NORMALIZE(  $\boldsymbol {u}\boldsymbol {u}^{\top}\boldsymbol{x}_{i})$    
if  $\| u_0\| >\varepsilon$  then  $\triangleright$  floating-point errors are considered   
substitute  $\pmb{u}$  with  $\pmb{u_0}$    
break   
end if   
end for   
end for   
for each multiple eigenvalue and its eigenvectors  $\pmb{U}\in \mathbb{R}^{n\times d}$  do  $\triangleright O(n^{2}dm)$  complexity   
proj  $\leftarrow (|\boldsymbol {U}\boldsymbol {U}^{\top}\boldsymbol {e}_1|,|\boldsymbol {U}\boldsymbol {U}^{\top}\boldsymbol {e}_2|,\dots ,|\boldsymbol {U}\boldsymbol {U}^{\top}\boldsymbol {e}_n|)$ $\triangleright O(n^{2}d)$  complexity   
len, ind  $\leftarrow$  SORT(proj)   
len  $\leftarrow$  UNIQUE(len)   
if  $k <   d$  then   
break  $\triangleright$  Assumption 2 not satisfied   
end if   
for  $i = 1,2,\ldots ,d$  do  $\triangleright O(n)$  complexity   
 $\pmb{x_i}\gets \sum_j\pmb{e}_{ind[j]}$  such that projind[j] = len[i]   
 $\pmb{x_i}\gets \pmb{x_i} + c\pmb{j}$    
end for   
 $U_{\mathrm{span}}\gets$  empty matrix of shape  $n\times 0$ $U_{\mathrm{perp}}\gets U$ $\triangleright$  orthogonal complementary space   
for  $i = 1,2,\ldots ,d$  do  $\triangleright O(nd^4)$  complexity   
 $\pmb{u_i}\gets U_{\mathrm{perp}}\pmb{U}_{\mathrm{perp}}^\top \pmb{x_i}$ $\triangleright O(nd)$  complexity   
if  $\| u_i\| <  \varepsilon$  then  $\triangleright$  floating-point errors are considered   
break Assumption 3 not satisfied   
end if   
 $\pmb{u_i}\gets$  NORMALIZE(ui)   
substitute  $U_{:,i}$  with  $u_{i}$ $U_{\mathrm{span}}\gets [U_{\mathrm{span}},u_i]$ $U_{\mathrm{base}}\gets U_{\mathrm{span}}$    
for  $j = 1,2,\ldots ,d$  do  $\triangleright O(nd^3)$  complexity   
 $U_{\mathrm{temp}}\gets [U_{\mathrm{base}},U_{:,j}]$    
if rank(Utemp)=j+1 then  $\triangleright O(nd^2)$  complexity   
 $U_{\mathrm{base}}\gets U_{\mathrm{temp}}$    
end if   
if  $U_{\mathrm{base}}\in \mathbb{R}^{n\times d}$  then break end if   
end for   
 $U_{\mathrm{base}}\gets$  GRAMSCHMIDTORTHOGONALIZATION(Ubase)  $\triangleright O(nd^2)$  complexity   
 $U_{\mathrm{perp}}\gets (U_{\mathrm{base}}),i + 1:d$    
end for   
end for   
 $U\gets U A^{\frac{1}{2}}$ $\triangleright O(n^{2})$  complexity
multiplicity
Figure 2: #Eigenvalues w.r.t. their multiplicities in real-world datasets (in logarithmic scale).
Table 10: The details of the datasets. Reg: Regression; Bin: Binary classification.
Dataset	ogbg-molesol	ogbg-molfreesolv	ogbg-molhiv	ogbg-mollipo	ogbg-moltox21	ogbg-moltoxcast	ogbg-molpcba
#Graphs	1128	642	41127	4200	7831	8576	437929
Avg #Nodes	13.3	8.7	25.5	27.0	18.6	18.8	26.0
Avg #Edges	13.7	8.4	27.5	29.5	19.3	19.3	28.1
#Tasks	1	1	1	1	12	617	128
Task Type	Reg	Reg	Bin	Reg	Bin	Bin	Bin
Metric	RMSE	RMSE	ROC-AUC	RMSE	ROC-AUC	ROC-AUC	AP
Algorithm 4 Verify the correctness of Theorem 3
$p_{-}$  correct  $\leftarrow 0$ $q_{-}$  correct  $\leftarrow 0$ $pq_{-}$  correct  $\leftarrow 0$  ,total  $\leftarrow 0$    
for  $i = 1,2,\ldots$  ,trials do   
 $n\gets$  a random positive integer   
 $U\gets$  a random orthonormal matrix in  $\mathbb{R}^{n\times n}$ $U_0\gets$  UNIQUESIGN(U)   
 $P\gets$  a random permutation matrix   
 $V\gets PU$ $V_0\gets$  UNIQUESIGN(V)   
 $p\_ correct\gets p\_ correct + 1$  if  $|PU_0 - V_0| <   \varepsilon$ $\triangleright$  test permutation-equivariance   
 $S\gets$  a random sign matrix (diagonal matrix of 1 and -1)   
 $W\gets US$ $W_0\gets$  UNIQUESIGN(W)   
 $q_{-}$  correct  $\leftarrow q_{-}$  correct  $+1$  if  $|U_0 - W_0| <   \varepsilon$ $\triangleright$  test uniqueness   
 $\pmb {Y}\gets \pmb{PW}$ $\pmb {Y}_0\gets$  UNIQUESIGN(Y)   
 $pq\_ correct\gets pq\_ correct + 1$  if  $|PU_0 - Y_0| <   \varepsilon$ $\triangleright$  test both   
total  $\leftarrow$  total  $+1$    
end for   
print out the values of  $p_{-}$  correct,  $q_{-}$  correct,  $pq_{-}$  correct and total
C.3 Verifying the correctness of Theorem 5
We verify the correctness of Theorem 5 through random simulation. The program is shown in Algorithm 5. Let  $U$  be a random orthonormal matrix in  $\mathbb{R}^{n \times d}$ ,  $P$  be a random permutation matrix and  $Q$  be a random orthonormal matrix in  $\mathbb{R}^{d \times d}$ . We pass  $U, PU, UQ, PUQ$  to the UNIQUE BASIS function (Algorithm 2) and compare the outputs. If our algorithm is correct,  $U$  and  $UQ$  should have invariant outputs, while  $U, PU$  and  $PUQ$  should have equivariant outputs.
Algorithm 5 Verify the correctness of Theorem 5
p.correct  $\leftarrow 0$  q.correct  $\leftarrow 0$  ,pq.correct  $\leftarrow 0$  total  $\leftarrow 0$    
for  $i = 1,2,\ldots$  ,trials do   
 $n\gets$  a random positive integer (greater than 1)   
 $d\gets$  a random positive integer (less than  $n$ $U\gets$  a random orthonormal matrix in  $\mathbb{R}^{n\times d}$ $U_0\gets$  UNIQUEBASIS(U)   
 $P\gets$  a random permutation matrix   
 $V\gets PU$ $V_{0}\gets$  UNIQUEBASIS(V)   
 $p\_ correct\gets p\_ correct + 1$  if  $|PU_0 - V_0| <   \varepsilon$ $\triangleright$  test permutation-equivariance   
 $Q\gets$  a random orthonormal matrix in  $\mathbb{R}^{d\times d}$ $W\gets UQ$ $W_{0}\gets$  UNIQUEBASIS(W)   
 $q\_ correct\gets q\_ correct + 1$  if  $|U_0 - W_0| <   \varepsilon$ $\triangleright$  test uniqueness   
 $Y\gets PW$ $Y_{0}\gets$  UNIQUEBASIS(Y)   
 $pq\_ correct\gets pq\_ correct + 1$  if  $|PU_0 - Y_0| <   \varepsilon$ $\triangleright$  test both   
total  $\leftarrow$  total  $+1$    
end for   
print out the values of  $p\_ correct$  , q.correct, pq.correct and total
We conduct 1000 trials. The results are  $p\_correct = q\_correct = pq\_correct = total = 1000$ , showing that Algorithm 2 is both permutation-equivariant and unique. The function UNIQUEBASIS raises an assertion error when either Assumption 2 or Assumption 3 is violated, so Algorithm 5 also shows that random orthonormal matrices violate these assumptions with probability 0.
C.4 Verifying the correctness of Theorem 1
We conduct experiment on the EXP dataset proposed in Abboud et al. [1], which is designed to explicitly evaluate the expressive power of GNNs. The dataset consists of a set of 1-WL indistinguishable non-isomorphic graph pairs, and each graph instance is a graph encoding of a propositional formula. The classification task is to determine whether the formula is satisfiable (SAT). Since the graph pairs in the EXP dataset are not distinguishable by 1-WL test, if a model shows above  $50\%$  accuracy on this dataset, it should have expressive power beyond the 1-WL algorithm.
The models we used on EXP dataset are as follows: an 8-layer GCN, GIN [57], PPGN [31], 1-2-3-GCN-L [35], 3-GCN [1], DeepSets-RNI (DeepSets with Random Node Initialization (RNI) [1]), GCN-RNI (GCN with Random Node Initialization (RNI)), Linear-RSE (linear network with RSE), and DeepSets-RSE. GCN and GIN belongs to the family of MP-GNNs whose expressive power is bounded by 1-WL, and RNI is a method to improve the expressive power of MP-GNNs. We verify the expressive power gain of RSE-based models by comparing them with GCN, and evaluate their efficiency by comparing them with other expressive models.
We evaluate all models on the Exp dataset using 10-fold cross-validation, and train each model for 500 epochs per fold. Mean test accuracy across all folds is measured and reported. The results are reported in Table 11. In addition, we also measure the learning curves of models to show their convergence rate, as shown in Figure 3.
In Table 11, we observe that vanilla GCN and DeepSets-RNI only achieves  $50\%$  accuracy, because they do not have expressive power beyond the 1-WL test. DeepSets-RSE achieves the best performance among all models with a near  $100\%$  accuracy, which demonstrates the expressive power of
Table 11: Accuracy results on EXP.
Model	Test Accuracy (%)
GCN	50.0
GIN	50.0
PPGN	50.0
1-2-3-GCN-L	50.0
3-GCN	99.7 ± 0.0
DeepSets-RNI	50.0
GCN-RNI	97.6 ± 2.5
Linear-RSE	99.1 ± 1.8
DeepSets-RSE	99.8 ± 0.5
Figure 3: Learning curves on EXP.
RSE-based models is beyond the 1-WL test. Other models, namely Linear-RSE and GCN-RNI, also achieve comparable performance. It's worth mentioning that even a simple linear model (Linear-RSE) could achieve performance above  $99\%$ . This indicates that the universal expressiveness of models with RSE is mostly from RSE itself, rather than the network structure.
From Figure 3, we find that the convergence rate of RSE-based models are much faster than their RNI-based counterpart. This is because RSE-based models are deterministic, while RNI-based models are random and require more training epochs to converge. The structures of DeepSets-RSE and Linear-RSE are simpler, which means they also train much faster than GCN-RNI.
D Synthetic experiment on basis invariance
As mentioned in Appendix B, SignNet [29] only deals with sign ambiguity, while BasisNet [29] has a prohibitive computational overhead. On the other hand, our proposed MAP addresses basis ambiguity efficiently, albeit with the existence of uncanonicalizable eigenspaces. In this section, we conduct a synthetic experiment to verify the ability of MAP on addressing basis ambiguity. We use graph isomorphic testing, a traditional graph task. Our focus is on 10 non-isomorphic random weighted graphs  $\mathcal{G}_1,\ldots ,\mathcal{G}_{10}$ , all exhibiting basis ambiguity issues (with the first three eigenvectors belonging to the same eigenspace). We sample 20 instances for each graph, introducing different permutations and basis choices for the initial eigenspace. The dataset is then split into a 9:1 ratio for training and testing, respectively. The task is a 10-way classification, where the aim is to determine the isomorphism of a given graph to one of the 10 original graphs. The model is given the first 3 eigenvectors as input (i.e.  $k = 3$ ). The results are averaged over 4 different runs.
Table 12: Test accuracy of the synthetic graph isomorphic testing task with DeepSets [63] using different PEs.
Positional Encoding	Accuracy
LapPE	0.11 ± 0.08
LapPE + RS	0.10 ± 0.09
LapPE + SignNet	0.10 ± 0.03
LapPE + MAP	0.84 ± 0.21
As evident from the results in Table 12, approaches that address sign ambiguity (like RandSign and SignNet) cannot obtain nontrivial performance on this task. Conversely, MAP shows commendable performance. The  $84\%$  accuracy, although impressive, indicates potential avenues for further enhancement. We believe this synthetic task could also serve as a valuable benchmark for future studies addressing basis invariance through canonization. Code for this experiment is available at https://github.com/GeorgeMLP/basis-invariance-synthetic-experiment.
E Other positional encoding methods
In this paper, we proposed MAP, which is a kind of positional encodings. In the field of graph representation learning, many other positional encoding methods have also been proposed. Murphy et al. [36] proposed Relational Pooling (RP) that assigns each node with an identifier that depends on the index ordering. They showed that RP-GNN is strictly more expressive than the original WL-GNN. However, to ensure permutation equivalence, we have to account for all possible  $n!$  node orderings, which is computationally intractable. You et al. [59] proposed learnable position-aware embeddings by computing the distance of a target node to an anchor-set of nodes, and showed that P-GNNs have more expressive power than existing GNNs. However, the expressive power of their model is dependent on the random selection of anchor sets. Sato et al. [44], Abboud et al. [1] proposed to use full or partial random node features and proved that their model has universal expressive power, but it has several defects: (1) the loss of permutation invariance, (2) slower convergence, and (3) poor generalization on unseen graphs [59, 30]. Li et al. [28] proposed Distance Encoding (DE) that captures the distance between the node set whose representation is to be learned and each node in the graph. They proved that DE can distinguish node sets embedded in almost all regular graphs where traditional GNNs always fail. However, their approach fails on distance regular graphs, and computation of power matrices can be a limiting factor for their model's scalability. Dwivedi et al. [18] proposed Learnable Structural and Positional Encodings (LSPE) that decouples structural and positional representations by inserting MPGNNs-LSPE layers and showed promising performance on three molecular benchmarks.
In particular, we will show that with Random Node Initialization (RNI), (1) A linear network is universal on a fixed graph, and (2) An MLP with just one additional message passing layer can be universal on arbitrary graphs.
In our work, we denote RNI as concatenating a random matrix to the input node features. The random matrix can be sampled from Gaussian distribution, uniform distribution, etc. Without loss of generality, we will assume that each entry of the random matrix is sampled independently from the standard Gaussian distribution  $N(0,1)$ .
Definition 2. A GNN with RNI is defined by concatenating a random matrix  $\mathbf{R}$  to the input node features, i.e.,  $\mathbf{X}' = [\mathbf{X},\mathbf{R}]$ , where  $\mathbf{X}$  are the original node features,  $\mathbf{X}'$  are the modified node features and each entry of  $\mathbf{R}$  is sampled independently from the standard Gaussian distribution  $N(0,1)$ . The value of  $\mathbf{R}$  is sampled at every forward pass of GNN.
To study the effects of RNI on the expressiveness of GNNs, we consider two types of tasks: tasks on a fixed graph (e.g., node classification) and tasks on arbitrary graphs (e.g. graph classification). On a fixed graph, we aim to learn a function  $f\colon \mathbb{R}^{n\times d}\to \mathbb{R}^{n\times d'}$  that transforms the feature of each node  $v_{i}$  to a presentation vector  $Z_{i,:}\in \mathbb{R}^{1\times d'}$ . We claim that a linear GNN with RNI in the form
$$
[ \mathbf {X}, \mathbf {R} ] \mathbf {W} = \mathbf {Z} \tag {6}
$$
is universal, where  $\mathbf{W} \in \mathbb{R}^{d \times d'}$  are the network parameters, and  $\mathbf{Z} \in \mathbb{R}^{n \times d'}$  is the desired output. In other words, we have the universality theorem of linear GNNs with RNI on a fixed graph:
Theorem 7. On a fixed graph  $\mathcal{G}$ , a linear GNN with RNI defined by Equation 6 is equivariant and can produce any prediction  $Z \in \mathbb{R}^{n \times d'}$  with probability 1.
We prove Theorem 7 in Appendix K.8.
On arbitrary graphs, the target function is not only dependent on the node features, but on the graph structure  $\hat{A} \in \mathbb{R}^{n \times n}$  as well. Let  $\Omega \subset \mathbb{R}^{n \times d} \times \mathbb{R}^{n \times n}$  be a compact set of graphs with  $[X, \hat{A}] \in \Omega$ , where  $X$  are the node features and  $\hat{A}$  is the normalized adjacency matrix. We wish to learn a function  $f: \Omega \to \mathbb{R}$  that transforms each graph to its label. Since  $f$  is also dependent on  $\hat{A}$ , an MLP-based network with  $X'$  as input is not expressive enough, and we need additional graph convolutional layers to obtain information about the graph structure. However, Puny et al. [41] proved that with just one additional message passing layer, an MLP with RNI can approximate any continuous invariant graph function  $f$ .
Theorem 8 (Puny et al. [41]). Given a compact set of graphs  $\Omega \subset \mathbb{R}^{n\times d}\times \mathbb{R}^{n\times n}$ , a GNN with one message passing layer, an MLP network with RNI can approximate an arbitrary continuous invariant graph function  $f\colon \Omega \to \mathbb{R}$  to an arbitrary precision.
Theorem 8 is a direct inference of the proof of Proposition 1 in Puny et al. [41], where the authors constructed a RGNN that first transfers the graph structural information to the node features through a message passing layer, and then approximates  $f$  with a DeepSets network, which is an MLP-based network.
F Toy Examples
F.1 Toy examples for Algorithm 1
Figure 4: A toy example illustrating our algorithm for eliminating sign ambiguity. Top: The angle between the  $z$ -axis and  $\pmb{u}$  is the smallest, so we choose  $+u$  to maximize  $\pmb{u}^{\top}\pmb{e}_z$ . Bottom: The angle between both  $x$  and  $y$ -axes and  $\pmb{u}$  are the smallest, so we choose  $+u$  to maximize  $\pmb{u}^{\top}(e_x + e_y)$ .
We give toy examples to help illustrate our MAP-sign algorithm. As shown on the top row of Figure 4, we have  $n = 3$  and two possible sign choices for the eigenvector  $\mathbf{u}$ . Our algorithm first compares the angles (or equivalently, the absolute value of inner product) between  $\mathbf{u}$  and the standard basis:  $e_x, e_y, e_z$ , and pick the smallest one (the one with the largest absolute inner product), in this case  $e_z$ . Thus we let  $\mathbf{x}_h = \mathbf{e}_z$  and choose the sign that maximizes  $\mathbf{u}^\top \mathbf{x}_h$ . In the first example we have  $(+\mathbf{u})^\top \mathbf{x}_h > 0 > (-\mathbf{u})^\top \mathbf{x}_h$ , thus  $+\mathbf{u}$  is chosen instead of  $-\mathbf{u}$ . This choice is unique and permutation-equivariant.
It is possible though, that the angle between  $\mathbf{u}$  and more than 1 basis vectors are equal. As shown on the bottom row of Figure 4, the angle between both  $\mathbf{e}_x,\mathbf{e}_y$  and  $\mathbf{u}$  are maximum. In this case we let  $\pmb{x}_h = \pmb {e}_x + \pmb {e}_y$  be their sum and maximize  $\pmb{u}^{\top}\pmb{x}_h$ , thus  $+u$  is chosen. A special case is when  $\mathbf{u}$  and  $\pmb{x}_h$  are perpendicular. If this happens, we go on to pick the basis vector with the second largest angle and continue this process.
Figure 5: A toy example illustrating our algorithm for eliminating basis ambiguity. First, we sort the angles between  $V$  and the standard basis vectors, and set  $\boldsymbol{x}_1 \coloneqq \boldsymbol{e}_x$ ,  $\boldsymbol{x}_2 \coloneqq \boldsymbol{e}_y$ . Next, we choose  $\boldsymbol{u}_1 \in V$  that maximizes  $\boldsymbol{u}_1^{\mathrm{T}}\boldsymbol{x}_1$ . Finally, we choose  $\boldsymbol{u}_2 \in \langle \boldsymbol{u}_1\rangle^\perp$  that maximizes  $\boldsymbol{u}_2^{\mathrm{T}}\boldsymbol{x}_2$ . This gives us a unique basis  $\boldsymbol{u}_1, \boldsymbol{u}_2$  of  $V$ .
F.2 Toy examples for Algorithm 2
We give a toy example to help illustrate our MAP-basis algorithm. As shown in Figure 5, we have  $n = 3$ ,  $d = 2$  with a two-dimensional eigenspace  $V$ . Our algorithm first compares the angles (or equivalently, the length of orthogonal projection) between  $V$  and the standard basis:  $e_x, e_y, e_z$ , and pick the two smallest one (the two with the largest and second largest lengths of orthogonal projection), in this case  $e_x$  and  $e_y$ . Thus we let  $x_1 = e_x$ ,  $x_2 = e_y$ . Then we choose  $u_1 \in V$  that maximizes  $u_1^\top x_1$ , which is just the orthogonal projection of  $x_1$  onto  $V$ . Finally we choose  $u_2 \in \langle u_1 \rangle^\perp$  that maximizes  $u_2^\top x_2$ , which is the orthogonal projection of  $x_2$  onto  $\langle u_1 \rangle^\perp$ . This gives us a basis  $u_1, u_2$  of  $V$ . This choice is unique and permutation-equivariant.
It is possible though, that the angle between  $V$  and more than 1 basis vectors are equal. For example, if  $\langle V, e_x \rangle = \langle V, e_y \rangle$  are both the smallest angles between  $V$  and the standard basis, then we let  $x_1 = e_x + e_y$ . The same goes for the second smallest angle, the third smallest angle, and so on.
G Discussions on the assumptions
G.1 Discussions on Assumption 1
The purpose of Section 3.3.1 is to uniquely determine the signs of the eigenvectors while also preserving their permutation-equivariance. One could easily come up with simple solutions such as choosing the signs such that the sum of the entries of eigenvectors are positive, or signs such that the element with the greatest absolute value is positive. In fact, Lim et al. [29] has proposed a more general way of choosing signs:
We also consider ablations in which we ... choose a canonical sign for each eigenvector by maximizing the norm of positive entries.
This "canonical sign" approach did not work well because (1) a large percentage of eigenvectors in real-world datasets have the same norm for positive and negative entries, thus this approach does not actually solve sign ambiguity of these eigenvectors; (2) this approach cannot be generalized to multiple eigenvalues. As a reference, we list the number of eigenvectors that "canonical sign" fails in Table 13.
Table 13: The number of eigenvectors with the same norm for positive and negative entries, the total number of eigenvectors, and the ratio of eigenvectors that "canonical sign" approach fails in real-world datasets.
Dataset	ogbg-molesol	ogbg-molfreesolv	ogbg-molhiv	ogbg-mollipo	ogbg-moltox21	ogbg-moltoxcast	ogbg-molpcba
#Violation	4060	2135	209854	15115	31365	35174	1536415
#Eigenvectors	14991	5600	1049163	113568	145459	161088	11373137
Ratio	27.08 %	38.13 %	20.00 %	9.84 %	21.56 %	21.84 %	13.51 %
Some examples of such eigenvectors in real-world datasets are as follows.
[0.0000, 0.0000, -0.0000, 0.0000, 0.1364, -0.0965, -0.0965, -0.1575, 0.0965, -0.0000, 0.1575, -0.0965, -0.2784, 0.2227, -0.1364, -0.0000, 0.3341, -0.2363, -0.2363, -0.1364, 0.2227, -0.0000, -0.0487, 0.0345, 0.0345, -0.0877, 0.0620, 0.0620, 0.2784, -0.2227, 0.1364, 0.0000, -0.3341, 0.2363, 0.2363, 0.1364, -0.2227],
[0.0000, 0.0000, 0.0000, -0.0796, 0.0796, 0.0975, -0.0796, -0.0563, 0.0000, 0.2087, -0.3615, 0.2951, 0.0000, -0.1815, 0.1815, 0.4767, -0.4767, -0.0029, 0.0051, -0.0042, 0.0000, -0.0458, 0.0458, 0.0416, -0.0416, -0.1495, 0.2589, -0.2114, 0.0000, -0.0975, 0.0975, -0.1139, 0.1139],
[0.0000, 0.0000, 0.0000, -0.0025, 0.0025, 0.0031, -0.0025, -0.0018, 0.0000, -0.0025, 0.0043, -0.0035, -0.0000, -0.0987, 0.0987, 0.0952, -0.0952, -0.1271, 0.2202, -0.1798, 0.0000, -0.3234, 0.3234, 0.1436, -0.1436, 0.1313, -0.2275, 0.1858, 0.0000, -0.2524, 0.2524, 0.4381, -0.4381].
Assumption 1, on the other hand, is less restrictive. It requires that the eigenvectors are not perpendicular to at least one of the vectors  $\pmb{x}_i$ . For random unit vectors or random weighted graphs, Assumption 1 has 0 possibility of being violated. The number of eigenvectors violating Assumption 1 in real-world datasets are listed in Table 14. It can be observed that the ratio of violation tends to become smaller as the graph size becomes larger.
Table 14: The number of eigenvectors violating Assumption 1, the total number of eigenvectors, and the ratio of violation in real-world datasets. We ignore small graphs with no more than 5 nodes.
Dataset	ogbg-molesol	ogbg-molfreesolv	ogbg-molhiv	ogbg-mollipo	ogbg-moltox21	ogbg-moltoxcast	ogbg-molpcba
#Violation	727	388	29558	3328	5418	6032	343088
#Eigenvectors	14551	5048	1049101	113568	144421	159987	11372381
Ratio	5.00 %	7.69 %	2.82 %	2.93 %	3.75 %	3.77 %	3.02 %
G.2 Discussions on Assumption 2 and Assumption 3