SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for sign equivariant linear  $W^{(2)}$  and  $W^{(1)}$ , and a sign invariant polynomial  $p_{\mathrm{inv}}: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$ .
This reduction of sign equivariant polynomials to sign invariant polynomials combined with simple operations is convenient, as it enables us to leverage recent universal models for sign invariant functions [Lim et al., 2023]. The proof of this statement is in Appendix C, which proceeds by showing that sign equivariance leads to linear constraints on the coefficients of a polynomial, which requires the polynomial to take the form stated in the Theorem.
4A class of model functions  $\mathcal{F}_{\mathrm{m}}$  from  $\mathcal{X}\rightarrow \mathcal{Y}$  is universal with respect to a target class  $\mathcal{F}_{\mathrm{t}}$  if for all compact  $\mathcal{D}\subseteq \mathcal{X}$ ,  $f_{\mathrm{t}}\in \mathcal{F}_{\mathrm{t}}$ , and  $\epsilon >0$ , there is an  $f_{\mathrm{m}}\in \mathcal{F}_{\mathrm{m}}$  such that  $\| f_{\mathrm{m}}(x) - f_{\mathrm{t}}(x)\| < \epsilon$  for all  $x\in \mathcal{D}$ .
Table 2: Link prediction AUC and runtime per epoch for structural edge models.
Model	Erdős-Rényi	Barabási-Albert
Test AUC	Runtime (s)	Test AUC	Runtime (s)
GCN (constant input)	.497±.06	.058±.00	.705±.01	.048±.00
SignNet	.498±.00	.120±.00	.707±.00	.095±.00
\( V_{i,:}^{\top}V_{j,:} \)	.570±.01	.010±.01	.597±.01	.008±.00
MLP(\( V_{i,:} \odot V_{j,:} \))	.614±.02	.050±.00	.651±.03	.040±.00
Sign Equivariant	.751±.00	.063±.00	.773±.01	.054±.00
3.3 Sign Equivariance without Permutation Symmetries
Using Theorem 1, we can now develop sign equivariant architectures. We parameterize sign equivariant functions  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  as a composition of layers  $f_{l}$ , each of the form
$$
f _ {l} (V) = \left[ W _ {1} ^ {(l)} v _ {1}, \dots , W _ {k} ^ {(l)} v _ {k} \right] \odot \operatorname {S i g n N e t} _ {l} (V), \tag {6}
$$
in which the  $W_{i}^{(l)}: \mathbb{R}^{n} \to \mathbb{R}^{n'}$  are arbitrary linear maps, and  $\mathrm{SignNet}_l: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  is sign invariant [Lim et al., 2023]. In the case of  $n = n' = 1$ , there is a simple universal form: we can write a sign equivariant function  $f: \mathbb{R}^k \to \mathbb{R}^k$  as  $f(v) = v \odot \mathrm{MLP}(|v|)$ , where  $|v|$  is the elementwise absolute value. These two architectures are universal because they can approximate sign equivariant polynomials. Here, the sign invariant part captures interactions between eigenvectors that the equivariant linear maps cannot.
Proposition 3. Functions of the form  $v \mapsto v \odot \mathrm{MLP}(|v|)$  universally approximate continuous sign equivariant functions  $f: \mathbb{R}^k \to \mathbb{R}^k$ .
Compositions  $f_{2} \circ f_{1}$  of functions  $f_{l}$  as in equation 6 universally approximate continuous sign equivariant functions  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$ .
3.4 Sign Equivariance and Permutation Equivariance
For models on eigenvectors that stem from graphs or point clouds, in addition to sign equivariance, we may demand permutation equivariance, i.e.,  $f(PV) = Pf(V)$  for all permutation matrices  $P \in \mathbb{R}^{n \times n}$ . To add permutation equivariance to our neural network architecture from Section 3.3, we use it within the framework of DeepSets for Symmetric Elements (DSS) [Maron et al., 2020]. For a hidden dimension size of  $d_f$ , each layer  $f_l: \mathbb{R}^{n \times k \times d_f} \to \mathbb{R}^{n \times k \times d_f}$  of our DSS-based sign equivariant network takes the following form on row  $i$ :
$$
f _ {l} (V) _ {i,:} = f _ {l} ^ {(1)} \left(V _ {i,:}\right) + f _ {l} ^ {(2)} \left(\sum_ {j \neq i} V _ {j,:}\right), \tag {7}
$$
where  $f_{l}^{(1)}$  and  $f_{l}^{(2)}$  are sign equivariant functions as in Section 3.3. Sometimes we take  $d_{f} = 1$ , in which case we can use the simpler  $\mathbb{R}^k \to \mathbb{R}^k$  sign equivariant networks ( $v \odot \mathrm{MLP}(|v|)$ ) as  $f_{l}^{(1)}$  and  $f_{l}^{(2)}$ . If we have graph information, then we can do message-passing by changing the sum over  $j \neq i$  to a sum over a neighborhood of node  $i$ . DSS has universal approximation guarantees [Maron et al., 2020], but they only apply for groups that act as permutation matrices, whereas the sign group  $\{-1, 1\}^k$  does not. Hence, the universal approximation properties of our proposed DSS-based architecture are still an open question.
Our theoretical results in Section 2 predict benefits of sign equivariance in various tasks: link prediction in nearly symmetric graphs, orthogonally equivariant simulations in n-body problems, and node clustering with positional information. Next, we probe these suggested benefits empirically.
4.1 Link Prediction in Nearly Symmetric Graphs.
We begin with a synthetic link prediction task, which is carefully controlled to test the theoretically foreseen benefits of sign equivariance explained in Section 2.1. With the intuition of Figure 2 we first
either generate an Erdős-Rényi [Erdős et al., 1960] or Barabási-Albert [Barabási and Albert, 1999] random graph  $H$  of 1000 nodes. Then we form a larger graph  $G$  that contains two disjoint copies of  $H$ , along with 1000 uniformly-randomly added edges (both between and within copies of  $H$ ). Without the random edges, each node in one copy of  $H$  is automorphic to the corresponding node in the other copy, so we expect many nodes to be nearly automorphic with the randomly added edges.
In Table 2, we show the link prediction performance of several models that learn structural edge representations. The methods that use eigenvectors have a sign invariant final prediction for each edge. GCN [Kipf and Welling, 2017] where the node features are all ones and SignNet [Lim et al., 2023] both completely fail on the Erdős-Rényi task (these two models map automorphic nodes to the same embedding), while our sign equivariant model outperforms all methods. We also try two eigenvector baselines that maintain node positional information, but do not update eigenvector representations: taking the dot product  $V_{i,:}^{\top}V_{j,:}$  to be the logit of a link existing, or learning a simple decoder  $\mathrm{MLP}(V_{i,:}\odot V_{j,:})$ . Both perform substantially worse than our sign equivariant model, which shows that updating eigenvector representations is important here. Further, the sign equivariant model takes comparable runtime to GCN, and is significantly faster than SignNet. This is because we use networks of the form  $v\mapsto v\odot \mathrm{MLP}(|v|)$  in these experiments instead of the full SignNet-based model in equation 6. See Appendix E.2 for more details.
4.2 Orthogonal Equivariance in n-body Problems
Figure 4: Sign equivariant model versus frame averaging model for n-body experiments in varying dimensions. Lower  $y$ -axis is better for both plots. (Left) The runtime of frame averaging increases exponentially in dimension while the sign equivariant runtime is approximately constant. Frame averaging runs out of memory on  $d = 11$ . (Right) The error of the sign equivariant model is very similar to that of frame averaging.
In this section, we empirically test the ability of our sign equivariant models to parameterize orthogonally equivariant functions on point clouds, as outlined in Section 2.2. For this purpose, we consider simulating n-body problems, following the setup in Fuchs et al. [2020] and building on the code from Puny et al. [2022]. To test the favorable scaling of our method in the dimension  $d$  of the problem against the exponential  $2^{d}$  scaling of frame averaging, we generalize this problem to general dimensions  $d \geq 3$ . We maintain the choice of  $n = 5$  particles, and generate new point clouds using the same procedure as in Fuchs et al. [2020] (sampling random points and initial velocities in a general dimension  $d$ ). We measure model performance via mean squared error (MSE). We use a DSS-based model that we describe in more detail in Appendix E.3.
Figure 4 illustrates the runtime and MSE. The sign equivariant model scales well with dimension—the time-per-epoch is nearly constant as we increase the dimension. In contrast, frame averaging suffers from the expected exponential slowdown with dimension, and runs out of memory on a 32GB V100 GPU for  $d = 11$ . Considering the MSE, the equivariant model's performance closely follows that of frame averaging, i.e., we only have a small loss in accuracy with much better scalability. For  $d = 3$ , the sign equivariant model has an MSE of .00646, compared to the .00575 of frame averaging [Puny et al., 2022]. Additional  $d = 3$  comparisons to other baselines are included in Appendix E.3.
4.3 Node Clustering with Positional Information
As explained in Section 2.1, some applications on graph data call for positional node embeddings that can assign different representations to automorphic nodes. For instance, consider community detection or node clustering tasks on graphs, where a model makes a prediction for each node that
assigns it to a cluster. Structural encodings are insufficient for this task, as there may be automorphic or nearly-automorphic nodes that are far apart in the graph but look alike in a structural encoding. Hence, node structural encodings would guide the model to assign these nodes to the same cluster, even though they should belong to different clusters. As a concrete example, consider a graph of two clusters, and using Laplacian eigenvectors as positional encodings. The first nontrivial eigenvector will tend to assign a positive sign to one cluster and a negative sign to the other cluster. Thus, the sign information in the eigenvectors is crucial, so we expect sign equivariant models to perform well.
We test models on the CLUSTER dataset [Dwivedi et al., 2022a] for semi-supervised node clustering (viewed as node classification) in synthetic graphs. In these experiments, we build on the empirically well-performing GraphGPS model [Rampasek et al., 2022], and incorporate our sign equivariant models to update eigenvector representations within the version of GraphGPS that uses PEG [Wang et al., 2022] to process positional encodings. See Appendix E.4 for more experimental details.
As seen in Table 3, our sign equivariant models outperform all of the other GraphGPS-based eigenvector methods. Moreover, we achieve the second best performance across all methods, showing that sign equivariant models can indeed achieve the theoretically expected benefits in this setting.
Table 3: Results on the CLUSTER node classification task, for which positional information is needed. We compare different SOTA and Laplacian eigenvector-based methods.
Model	Test Acc. (%)
GCN [Kipf and Welling, 2017]	68.498±0.976
GIN [Xu et al., 2019]	64.716±1.553
GAT [Veličković et al., 2018]	70.587±0.447
GatedGCN [Bresson and Laurent, 2017]	73.840±0.326
SAN [Kreuzer et al., 2021]	76.691±0.650
K-Subgraph SAT [Chen et al., 2022]	77.856±0.104
EGT [Hussain et al., 2022]	79.232±0.348
GPS [Rampasek et al., 2022]	78.016±0.180
Eigenvector Methods (GPS base model)
No PE	77.423±0.241
LapPE [Dwivedi et al., 2022a]	77.250±0.280
PEG [Wang et al., 2022]	77.945±0.310
SignNet [Lim et al., 2023]	77.442±0.102
Sign Equivariant (ours)	78.201±0.118
Structural and Positional Representations. Especially for link prediction, the need for structural
node-pair representations that are not obtained from structural node representations has been discussed in several works [Srinivasan and Ribeiro, 2019, Zhang et al., 2021, Cotta et al., 2023]. As such, various methods have been developed for learning structural node-pair representations that incorporate node positional information. SEAL and other labeling-trick based methods [Zhang and Chen, 2018, Zhang et al., 2021] use added node features depending on the node-pair that we want a representation of. This is empirically successful in many tasks, but typically requires a separate subgraph extraction and forward pass through a GNN for each node-pair under consideration. Distance encoding [Li et al., 2020] uses relative distances between nodes to capture positional information. PEG [Wang et al., 2022] similarly maintains positional information by using eigenvector distances between nodes in each layer of a GNN, but does not update eigenvector representations. Identity-aware GNNs [You et al., 2021] and Neural Bellman-Ford Networks [Zhu et al., 2021] learn pair representations by conditioning on a source node from the pair.
Eigenvectors as Graph Positional Encodings. When using eigenvectors of graphs as node positional encodings for graph models like GNNs and Graph Transformers, many works have noted the need to address the sign ambiguity of the eigenvectors. This is often done by encouraging sign invariance through data augmentation—the signs of the eigenvectors are chosen randomly in each iteration of training [Dwivedi et al., 2022a,b, Kreuzer et al., 2021, Mialon et al., 2021, Kim et al., 2022, He et al., 2022, Müller et al., 2023]. In contrast, SignNet [Lim et al., 2023] enforces exact sign invariance, by processing eigenvectors with a sign invariant neural architecture; this approach has been taken by some recent works [Rampasek et al., 2022, Geisler et al., 2023, Murphy et al., 2023].
Equivariant Neural Network Design. Equivariant neural network architectures have been proposed for various types of data and symmetry groups. A common paradigm is to interleave equivariant linear maps and equivariant pointwise nonlinearities [Wood and Shawe-Taylor, 1996, Cohen and Welling, 2016, 2017, Ravanbakhsh et al., 2017, Maron et al., 2018, Kondor and Trivedi, 2018, Finzi et al., 2021, Bronstein et al., 2021, Pearce-Crump, 2022]; this is often used when the group acts as some subset of the permutation matrices. However, the sign group does not act as permutation matrices, and as we explained above this approach is not expressive for sign equivariant models.
More similarly to our approach, many equivariant machine learning works heavily leverage invariant or equivariant polynomials (or other equivariant nonlinear functions). These works include polynomials as operations within a network [Thomas et al., 2018, Puny et al., 2023], add polynomials as features [Yarotsky, 2022, Villar et al., 2021], build networks that take a similar form to equivariant polynomials [Villar et al., 2021], and/or analyze neural network expressive power by determining which equivariant polynomials a given architecture can compute [Zaheer et al., 2017, Segol and Lipman, 2019, Maron et al., 2019, 2020, Chen et al., 2020, Dym and Maron, 2021, Puny et al., 2023].
In this work, we identify and study an important method of respecting the symmetries of eigenvector data—sign equivariant models. For multi-node representation tasks, link prediction, and orthogonally equivariant tasks, sign equivariance provides a natural inductive bias; in contrast, we show that sign invariant models are provably limited in these tasks. To develop sign equivariant neural networks, we analytically characterize the sign equivariant polynomials, and then define neural networks that parameterize functions of similar form. Our neural networks are thus expressive, and inherit universal approximation guarantees of the equivariant polynomials. In several experiments, we show that our neural networks can indeed achieve the theoretically predicted benefits of sign equivariant models.
Limitations and Future Work. While we developed sign equivariant architectures in this work, we did not explore basis-change equivariant architectures, which would have the desired symmetries for inputs with repeated eigenvalues. As eigenvalue multiplicities are known to occur in many real-world graphs [Lim et al., 2023], future work in this area could be useful. Further, we give evidence that sign equivariance could help in some node-level and multi-node-level prediction tasks on graphs, but we do not have theoretical reason to believe that sign equivariance could help in graph-level representation tasks, which for instance are common in molecule processing. Our theoretical results are focused on expressive power, but we do not have results on other properties that are important for learning, such as optimization [Xu et al., 2021], stability [Wang et al., 2022, Huang et al., 2023], or generalization [Keriven and Vaiter, 2023]. Finally, while we can prove universality of our models in the non-permutation-equivariant setting, we do not know of the exact expressive power in the permutation equivariant setting. Lim et al. [2023] also faces this issue for sign invariant models; future work on analyzing and possibly improving the expressive power of these models — if they are not universal — is promising.
We would like to thank Yaron Lipman for contributing significantly early on in this work. We would like to thank Maks Ovsjanikov for noting that basis invariant models give automorphic nodes the same representation, and also Johannes Lutzeyer and Michael Murphy for helpful comments. We would also like to thank the reviewers of the Physics4ML workshop at ICLR 2023 for helpful feedback and close reading. DL is supported by an NSF Graduate Fellowship. SJ acknowledges support from NSF AI Institute NSF CCF-2112665, NSF Award 2134108, and Office of Naval Research Grant N00014-20-1-2023 (MURI ML-SCOPE).
Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research, 18(1):6446-6531, 2017.
Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In International Joint Conference on Artificial Intelligence, 2021.
Emily Alsentzer, Samuel Finlayson, Michelle Li, and Marinka Zitnik. Subgraph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 8017-8029, 2020.
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation, 55(1):407-474, 2022.
Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Association for the Advancement of Artificial Intelligence (AAAI), volume 35, pages 10737-10745, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 3391-3401, 2017.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 9061-9073, 2021.
Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 29476-29490, 2021.
A Applications of Sign Equivalence
A.1 Improving Invariant Eigenvector Networks
Neural networks that are invariant to eigenvector symmetries have been shown to empirically improve graph learning models and achieve theoretically high expressive power. SignNet [Lim et al., 2023], a sign invariant neural network, takes the form
$$
f \left(v _ {1}, \dots , v _ {k}\right) = \rho \left(\phi \left(v _ {1}\right) + \phi (- v _ {1}), \dots , \phi \left(v _ {k}\right) + \phi (- v _ {k})\right) \tag {8}
$$
for neural networks  $\rho$  and  $\phi$ . This directly enforces invariant representations, without any intermediate equivariant representations. However, many successful invariant models first have many equivariant layers before a final invariant operation as equivariant layers are more expressive: this includes convolutional neural networks [LeCun et al., 1989], message passing graph neural networks [Gilmer et al., 2017], invariant graph networks [Maron et al., 2018], and group convolutional neural networks [Cohen and Welling, 2016]. Thus, sign equivariant layers may lead to better sign invariant networks. Moreover, sign equivariant layers may improve on other aspects of SignNet, such as expressiveness of node features (Proposition 1) and efficiency (Appendix A.2)
A.2 Efficiency Gains from Sign Equivariant Networks
Here, we show that our sign equivariant models can reduce the complexity of equivariant or invariant networks for two different types of applications. Throughout, we consider functions  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$ , and we consider our permutation equivariant and sign equivariant DSS-based architecture from Section 3.4.
The time cost (in floating point operations) per layer of our DSS-based model is  $\mathcal{O}(n(kd + d^2))$ , where  $d$  is the maximum hidden dimension of the MLP and we assume constant depth MLPs. To see this, note that we can precompute  $\sum_{j=1}^{n} V_{j,}$ , so that each  $\sum_{j \neq i} V_{j,}$  can be computed in constant time by subtracting  $V_{i,}$  from the total sum. Then for each of the  $n$  rows, the MLPs require  $\mathcal{O}(kd + d^2)$  to evaluate matrix multiplications. In this process, we only form tensors of size  $\mathcal{O}(n(k + d))$ , as the inputs and outputs are of size  $\mathcal{O}(nk)$ , and the hidden layers of the MLPs form tensors of size  $\mathcal{O}(nd)$ .
A.2.1 Efficient Orthogonally Equivariant Networks
Consider the case of  $O(k)$  equivariant models  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$  such that  $f(XQ) = f(X)Q$  for all orthogonal matrices  $Q \in O(k)$ . There are many orthogonally equivariant neural architectures that are specialized to the special case of  $k = 3$ , which is very useful for applications in the physical sciences [Thomas et al., 2018, Fuchs et al., 2020]. Here we consider models that directly work for general dimension  $k$ .
Frame averaging approaches [Puny et al., 2022, Atzmon et al., 2022] require  $2^{k}$  forward passes of a base network  $f_{\theta}$ , one for each sign flip of the principal components. Letting their base network be a permutation equivariant DeepSets [Zaheer et al., 2017], this means that they require  $\mathcal{O}(n(kd + d^2)2^k)$  time to evaluate their model, where  $d$  is the hidden dimension of the base model. Note that this has an extra exponential  $2^{k}$  factor compared to our  $\mathcal{O}(n(kd + d^2))$  cost.
Another general approach with universality guarantees comes from Villar et al. [2021], who analyze invariant polynomials to develop equivariant architectures. However, their method for  $O(k)$  invariance or equivariance requires forming  $XX^{\top}$ , an  $n \times n$  matrix. Thus, the complexity is at least  $\mathcal{O}(n^{2})$ , which is a problem in applications, since oftentimes  $n$  is much larger than  $k$ . Variants of their method do not need to compute all  $\mathcal{O}(n^{2})$  inner products, but it is unclear how to maintain permutation equivariance when doing this.
A.2.2 Efficient Sign Invariant Networks
Consider again the form of SignNet [Lim et al., 2023],  $f(V) = \rho ([\phi (v_i) + \phi (-v_i)]_{i = 1,\dots ,k})$ . In the permutation equivariant version, e.g. when  $\phi$  is a DeepSets [Zaheer et al., 2017] or a message passing neural network [Gilmer et al., 2017],  $\phi$  maps from  $\mathbb{R}^n\to \mathbb{R}^{n\times d}$ , where  $d$  is the hidden dimension. Thus, computing  $\phi (v_{i}) + \phi (-v_{i})$  for all  $k$  vectors  $v_{i}$  require an  $\mathcal{O}(nkd)$  sized tensor to be formed (even if the output space of  $\phi$  is  $\mathbb{R}^n$ , a vectorized implementation computes all  $\phi (v_{i}) + \phi (-v_{i})$  in two batched inference calls to  $\phi$ , which would require  $\mathcal{O}(nkd)$  sized intermediate tensors). This is