SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Derek Lim
MIT CSAIL
dereklim@mit.edu
Joshua Robinson*
Stefanie Jegelka
TU Munich, MIT CSAIL
Haggai Maron
Technion, NVIDIA
Recent work has shown the utility of developing machine learning models that respect the structure and symmetries of eigenvectors. These works promote sign invariance, since for any eigenvector  $v$  the negation  $-v$  is also an eigenvector. However, we show that sign invariance is theoretically limited for tasks such as building orthogonally equivariant models and learning node positional encodings for link prediction in graphs. In this work, we demonstrate the benefits of sign equivariance for these tasks. To obtain these benefits, we develop novel sign equivariant neural network architectures. Our models are based on a new analytic characterization of sign equivariant polynomials and thus inherit provable expressiveness properties. Controlled synthetic experiments show that our networks can achieve the theoretically predicted benefits of sign equivariant models.
The need to process eigenvectors is ubiquitous in machine learning and the computational sciences. For instance, there is often a need to process eigenvectors of operators associated with manifolds or graphs [Belkin and Niyogi, 2003, Rustamov et al., 2007], principal components (PCA) of arbitrary datasets [Pearson, 1901], and eigenvectors arising from implicit or explicit matrix factorization methods [Levy and Goldberg, 2014, Qiu et al., 2018]. However, eigenvectors are not merely unstructured data—they have rich structure in the form of symmetries [Ovsjanikov et al., 2008].
Specifically, eigenvectors have sign and basis symmetries. An eigenvector  $v$  is sign symmetric in the sense that the sign-flipped vector  $-v$  is also an eigenvector of the same eigenvalue. Basis symmetries occur when there is a repeated eigenvalue, as then there are infinitely many choices of eigenvector basis for the same eigenspace. Prior work has developed neural networks that are invariant to these symmetries, improving empirical performance in several settings [Lim et al., 2023].
The goal of this paper is to demonstrate why sign equivariance can be useful and to characterize fundamental expressive sign equivariant architectures. Our first contribution is to show that sign equivariant models are a natural choice for several applications, whereas sign invariant architectures are provably insufficient for these applications. First, we show that sign and
basis invariant networks are theoretically limited in expressive power for learning edge representations (and more generally multi-node representations) in graphs because they learn structural node embeddings that are known to be limited for link prediction and multi-node tasks [Srinivasan and
Figure 1: Illustration of a sign equivariant function  $f$ . When column 1 of the input is negated, column 1 of the output is also negated.
*Work completed whilst at MIT.
Ribeiro, 2019, Zhang et al., 2021]. In contrast, we show that sign equivariant models can bypass this limitation by maintaining positional information in node embeddings. Furthermore, we show that sign equivariance combined with PCA can be used to parameterize expressive orthogonally equivariant point cloud models, thus giving an efficient alternative to PCA-based frame averaging [Puny et al., 2022, Atzmon et al., 2022]. In contrast, sign invariant models can only parameterize orthogonally invariant models in this framework, which excludes many important application areas.
The second contribution of this work is to develop the first sign equivariant neural network architectures, with provable expressiveness guarantees. We first present a difficulty in developing sign equivariant models: the "Geometric Deep Learning Blueprint" [Bronstein et al., 2021] suggests developing an equivariant neural network by interleaving equivariant linear maps and equivariant elementwise nonlinearities [Cohen and Welling, 2016, Ravanbakhsh et al., 2017, Finzi et al., 2021], but we show that our attempts to apply this approach are insufficient for expressive sign equivariant models. Namely, we show that sign equivariant linear maps between various input and output representations are very limited in their expressive power.
Hence, to develop our models, we derive a complete characterization of the sign equivariant polynomial functions. The form of these equivariant polynomials directly inspires our equivariant neural network architectures. Further, our architectures inherit the theoretical expressive power guarantees of the equivariant polynomials. Our characterization is also broadly useful for analysis and development of sign-symmetry-respecting architectures—for instance, we provide a new proof of the universality of SignNet [Lim et al., 2023] by showing that it can approximate all sign invariant polynomials.
To validate our theoretical results, we conduct various numerical experiments on synthetic datasets. Experiments in link prediction, n-body problems, and node clustering in graphs support our theory and demonstrate the utility of sign equivariant models.
1.1 Background
Let  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$  be a function that takes eigenvectors  $v_{1}, \ldots, v_{k} \in \mathbb{R}^{n}$  of an underlying matrix as input, and outputs representations  $f(v_{1}, \ldots, v_{k})$ . We often concatenate the eigenvectors into a matrix  $V = [v_{1}, \ldots, v_{k}] \in \mathbb{R}^{n \times k}$ , and write  $f(V)$  as the application of  $f$ . For simplicity, in this work we assume the eigenvectors come from a symmetric matrix, so they are taken to be orthonormal.
Sign and basis symmetries. Eigenvectors have symmetries, because there are many possible choices of eigenvectors of a matrix. For instance, if  $v$  is a unit-norm eigenvector of a matrix, then so is the sign-flipped  $-v$ . If the eigenvalue of  $v$  is simple, then  $-v$  is the only other choice of unit-norm eigenvector of this eigenvalue.
If  $v_{1}, \ldots, v_{m}$  are an orthonormal basis of eigenvectors for the same eigenspace (meaning they all have the same eigenvalue), then there are infinitely many other choices of orthonormal basis for this eigenspace; these other choices of basis can be written as  $VQ$ , where  $V = [v_{1} \ldots v_{m}] \in \mathbb{R}^{n \times m}$  and  $Q \in O(m)$  is an arbitrary orthogonal matrix.
We refer to these symmetries collectively as sign and basis symmetries, or more simply as eigenvector symmetries. Note that sign symmetries are a special case of basis symmetries, as  $-1$  and  $1$  are the only orthogonal  $1 \times 1$  matrices. Previous work has developed neural networks that are invariant to these symmetries—that is, networks that have the same output for any choice of sign or basis of the eigenvector inputs [Lim et al., 2023].
Sign equivariance means that if we flip the sign of an eigenvector, then the corresponding column of the output of a function  $f$  has its sign flipped. In other words, for all choices of signs  $s_1, \ldots, s_k \in \{-1, 1\}^k$ ,
$$
f \left(s _ {1} v _ {1}, \dots , s _ {k} v _ {k}\right) _ {:, j} = s _ {j} f \left(v _ {1}, \dots , v _ {k}\right) _ {:, j}, \tag {1}
$$
where  $A_{\cdot ,j}$  is the  $j$  -th column of an  $n\times k$  matrix  $A$  . See Figure 1 for an illustration. In matrix form, letting  $\mathrm{diag}(\{-1,1\} ^k)$  represent all  $k\times k$  diagonal matrices with  $-1$  or 1 on the diagonal,  $f$  is sign equivariant if
$$
f (V S) = f (V) S \quad \text {f o r a l l} S \in \operatorname {d i a g} \left(\{- 1, 1 \} ^ {k}\right). \tag {2}
$$
As  $O(1) = \{-1, 1\}$ , we can write sign equivariance as equivariance with respect to a direct product of orthogonal groups  $O(1) \times \ldots \times O(1)$ . This is different from the equivariance to a single orthogonal group  $O(d)$  considered in works on Euclidean-group equivariant networks [Thomas et al., 2018].
(a) Eigenvector 1
(b) Sign Invariant
(c) Sign Equivariant
Figure 2: (a) First nontrivial normalized Laplacian eigenvector of a graph, which is positional. Nodes  $u_{1}$  and  $u_{2}$  are far apart in the graph, but automorphic. (b) Sign invariant node features, which are structural. Nodes  $u_{1}$  and  $u_{2}$  have the same feature. (c) Sign equivariant node features, which are positional. Nodes  $u_{1}$  and  $u_{2}$  have opposite signs. A link prediction model with sign invariant node features assigns  $u_{1}$  and  $u_{2}$  the same probability of connecting to  $w$ , while sign equivariant node features could give higher probability to  $u_{1}$ .
Permutation equivariance is often also a desirable property of our functions  $f$ . We say that  $f$  is permutation equivariant if  $f(PV) = Pf(V)$  for all  $n \times n$  permutation matrices  $P$ . For instance, eigenvectors of matrices associated with simple graphs of size  $n$  have such permutation symmetries, as the ordering of nodes is arbitrary.
2 Applications of Sign Equivariance
In this section, we present several applications for which modeling networks with sign equivariant architectures is beneficial. We identify that sign invariant networks are provably insufficient for these tasks, motivating the development of sign equivariant networks to address these limitations.
2.1 Multi-Node Representations and Link Prediction
In several settings, we desire a machine learning model that computes representations for tuples of several nodes in a graph. For instance, link prediction tasks generate probabilities for pairs of nodes, and both hyperedge prediction and subgraph prediction tasks learn representations for collections of nodes in a graph [Alsentzer et al., 2020, Wang et al., 2023]. For ease of exposition, the rest of this section discusses link prediction, though the discussion applies to general multi-node tasks as well.
In link prediction, we typically want to learn structural node-pair representations, meaning adjacency-permutation equivariant functions that give a representation for each pair of nodes; more precisely, a structural node-pair representation is a map  $f: \mathbb{R}^{n \times n} \to \mathbb{R}^{n \times n}$  such that  $f(PAP^{\top}) = Pf(A)P^{\top}$ , where  $f(A)_{i,j}$  is the representation of the pair of nodes  $(i,j)$  in the graph with adjacency matrix  $A$  [Srinivasan and Ribeiro, 2019] (see Appendix A.5 for more discussion). One method to do this is to use a graph model such as a standard GNN to learn node representations  $z_i$ , and then obtain a node-pair representation for  $(i,j)$  as some function  $f_{\mathrm{decode}}(z_i,z_j)$  of  $z_i$  and  $z_j$ . However, this approach is limited because standard GNNs learn structural node encodings—that is, adjacency-permutation equivariant node features  $f_{\mathrm{node}}: \mathbb{R}^{n \times n} \to \mathbb{R}^n$  such that  $f_{\mathrm{node}}(PAP^{\top}) = Pf_{\mathrm{node}}(A)$  [Srinivasan and Ribeiro, 2019, Zhang et al., 2021]. Structural node encodings give automorphic nodes the same representation, which can be problematic since automorphic nodes can be far apart in the graph. For instance, in Figure 2,  $u_1$  and  $u_2$  are automorphic, so a link predictor based on structural node encodings would give both  $u_1$  and  $u_2$  the same probability of connecting to  $w$ , but one would expect  $u_1$  to have a higher probability of connecting to  $w$ . Most state-of-the-art link prediction methods on the Open Graph Benchmark leaderboards [Hu et al., 2020] were deliberately developed to avoid the issues of structural node encodings.
One way to surpass the limitations of structural node encodings is to use positional node embeddings, which can assign different values to automorphic nodes. Intuitively, positional encodings capture information such as distances between nodes and global position of nodes in the graph (see [Srinivasan and Ribeiro, 2019] for a formal definition). Laplacian eigenvectors are an important example of node
2This adjacency permutation equivariance is different from permutation equivariance of general vectors as defined in Section 1.1.
positional embeddings that capture much useful information of graphs [Chung, 1997, Von Luxburg, 2007]. In Figure 2 (a), the first nontrivial Laplacian eigenvector captures cluster structure in the graph, and as such assigns  $u_{1}$  and  $u_{2}$  very different values.
Pitfalls of sign and basis invariance. When processing eigenvectors of matrices associated with graphs, invariance to the symmetries of the eigenvectors has been found useful [Dwivedi et al., 2022a, Lim et al., 2023], especially for graph classification tasks. However, we show that exact invariance to these symmetries removes positional information, and thus the outputs of sign invariant or basis invariant networks are in fact structural node encodings (see Appendix A.5). Hence, eigenvector-symmetry-invariant networks cannot learn node representations that distinguish automorphic nodes, and thus face the aforementioned difficulties when used for link prediction or multi-node tasks:
Proposition 1. Let  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times d_{\mathrm{out}}}$  be a permutation equivariant function, and let  $V = [v_1, \ldots, v_k] \in \mathbb{R}^{n \times k}$  be  $k$  orthonormal eigenvectors of an adjacency matrix  $A$ . Let nodes  $i$  and  $j$  be automorphic, and let  $z_i$  and  $z_j \in \mathbb{R}^{d_{\mathrm{out}}}$  be their embeddings, i.e., the  $i$ th and  $j$ th row of  $Z = f(V)$ .
- If  $f$  is sign invariant and the eigenvalues associated with the  $v_{l}$  are simple and distinct, then  $z_{i} = z_{j}$ .
- If  $f$  is basis invariant and  $v_{1},\ldots ,v_{k}$  are a basis for some number of eigenspaces of  $A$  then  $z_{i} = z_{j}$ .
A novel link prediction approach via sign equivariance. The problem  $z_{i} = z_{j}$  arises from the sign/basis invariances, which remove crucial positional information. We instead propose using sign equivariant networks (as in Section 3) to learn node representations  $z_{i} = f(V)_{i,:} \in \mathbb{R}^{k}$ . These representations  $z_{i}$  maintain positional information for each node thanks to preserving sign information (see Figure 2 (c)). Then we use a sign invariant decoder  $f_{\mathrm{decode}}(z_i,z_j) = f_{\mathrm{decode}}(Sz_i,Sz_j)$  for  $S \in \mathrm{diag}\left(\{-1,1\}^{k}\right)$  to obtain node-pair representations. For instance, the commonly used  $f_{\mathrm{decode}} = \mathrm{MLP}(z_i \odot z_j)$ , where  $\odot$  is the elementwise product, is sign invariant. When the eigenvalues are distinct, this approach has the desired invariances (yielding structural node-pair representations) and also maintains positional information in the node embeddings; see Appendix A.5 for a proof of the invariances, and Appendix A.5.1 for an example of where sign equivariant models can be used to compute strictly more expressive node-pair representations than sign invariant models. More details and the proof of Proposition 1 are in Appendix A.4.
Our sign equivariance based approach differs substantially from existing methods for learning structural pair representations without being bottlenecked by structural node representations. Many of these methods are based on labeling tricks [Zhang et al., 2021, Wang et al., 2023], whereby the representation for a node-pair is obtained by labeling the two nodes in the pair and then processing an enclosing subgraph. Without special modifications [Zhu et al., 2021, Chamberlain et al., 2023], this requires a separate expensive subgraph extraction and forward pass for each node-pair. In contrast, our method only requires one forward pass on the original graph to compute all positional node embeddings, after which pair representations can be obtained with a cheap, parallelizable decoding.
2.2 Orthogonal Equivalence
For various applications in modelling physical systems, we desire equivariance to rigid transformations; thus, orthogonally equivariant models have been a fruitful research direction in recent years [Thomas et al., 2018, Weiler et al., 2018, Anderson et al., 2019, Deng et al., 2021]. We say that a function  $f: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$  is orthogonally equivariant if  $f(XQ) = f(X)Q$  for any  $Q \in O(k)$ , where  $O(k)$  is the set of orthogonal matrices in  $\mathbb{R}^{k \times k}$ . Orthogonal equivariance imposes infinitely many constraints on the function  $f$ . Several works have approached this problem by reducing to a finite set of constraints using so-called Principal Component Analysis (PCA) based frames [Puny et al., 2022, Atzmon et al., 2022, Xiao et al., 2020].
PCA-frame methods take an input  $X \in \mathbb{R}^{n \times k}$ , compute orthonormal eigenvectors  $R_X \in O(k)$  of the covariance matrix  $\operatorname{cov}(X) = (X - \frac{1}{n} \mathbf{11}^\top X)^\top (X - \frac{1}{n} \mathbf{11}^\top X)$  (assumed to have distinct eigenvalues), then average outputs of a base model  $h$  for each of the  $2^k$  sign-flipped inputs  $XR_X S$ , where  $S \in \mathrm{diag}(\{-1, 1\}^k)$ . We instead suggest using a sign equivariant network to parameterize an efficient  $O(k)$  equivariant model, which allows us to bypass the need to average the exponentially
3When there are repeated eigenvalues, sign invariant embeddings maintain some positional information.
Figure 3: Using sign equivariant functions  $h$  to parameterize orthogonally equivariant  $f(X) = h(XR_X)R_X^\top$ , where  $R_X$  is a choice of principal components for the point cloud. We first transform  $X$  via  $R_X$  into an orientation that is unique up to sign flips, then process  $XR_X$  using the sign equivariant model  $h$ , and finally reintegrate orientation information back into the output via  $R_X^\top$ .
Table 1: Sign invariant or equivariant polynomials and corresponding neural network architectures for different input and output spaces.  $v \in \mathbb{R}^k$  or  $V \in \mathbb{R}^{n \times k}$  are inputs to the polynomials or networks. Appendix C contains more details on the polynomials.
Constraints	Polynomials	Neural Networks
Rk→ R inv.	∑Dd1,...,dk=0 Wd1,...,dkv12d1...vk2dk	MLP(|v|)
Rn×k→ R inv.	q([V1i,j·V1i2j]i1∈[n],i2∈[n],j∈[k])	SignNet(V) = ρ([\phi(vi)+φ(-vi)]i=1,...,k)
Rk→ Rk equiv.	v ⊙ pinv(v)	v ⊙ MLP(|v|)
Rn×k→ Rn′×k equiv.	W(2)(W(1)V) ⊙ pinv(V)	[W1(l)v1,...,W(kl)vk] ⊙ SignNetl(V)
many sign-flipped inputs. For a sign equivariant network  $h$ , we define our model  $f$  to be
$$
f (X) = h \left(X R _ {X}\right) R _ {X} ^ {\top}. \tag {3}
$$
See Figure 3 for an illustration. Intuitively, this first transforms  $X$  by  $R_{X}$  into a nearly canonical orientation that is unique up to sign flips; this can be seen as writing the points in the principal components basis, or aligning the principal components of  $X$  with the coordinate axes. Then we process  $XR_{X}$  using the model  $h$  that respects the sign symmetries, and finally, we incorporate orientation information back into the output by post-multiplying by  $R_{X}^{\top}$ . Our approach only requires one forward pass through  $h$ , whereas frame averaging requires  $2^{k}$  forward passes through a base model. The following proposition shows that  $f$  is  $O(k)$  equivariant, and inherits universality properties of  $h$ .<sup>4</sup>
Proposition 2. Consider a domain  $\mathcal{X} \subseteq \mathbb{R}^{n \times k}$  such that each  $X \in \mathcal{X}$  has distinct covariance eigenvalues, and let  $R_X$  be a choice of orthonormal eigenvectors of  $\operatorname{cov}(X)$  for each  $X \in \mathcal{X}$ . If  $h: \mathcal{X} \subseteq \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$  is sign equivariant, and if  $f(X) = h(X R_X) R_X^\top$ , then  $f$  is well defined and orthogonally equivariant.
Moreover, if  $h$  is from a universal class of sign equivariant functions, then the  $f$  of the above form universally approximate  $O(k)$  equivariant functions on  $\mathcal{X}$ .
We include a proof of this result in Appendix A.6. This result also follows from Theorems 3.1 and 3.3 of Kaba et al. [2023], who show that generally we can canonicalize up to a subgroup  $K$  of a group  $G$ , and achieve  $G$ -equivariance via a  $K$ -equivariant base predictor. In our case,  $G = O(k)$  and  $K = \{-1,1\}^k$
Sign invariance only gives orthogonal invariance. In a similar way, a sign invariant model can be used to obtain an orthogonally invariant model with PCA frames, but it cannot be used for orthogonal equivariance; instead, sign equivariance is needed.
3 Sign Equivariant Polynomials and Networks
In this section, we analytically characterize the sign equivariant polynomials, and use this characterization to develop sign equivariant architectures. As equivariant polynomials universally approximate continuous equivariant functions [Yarotsky, 2022], our architectures inherit universality guarantees. We summarize our results on polynomials and neural network architectures in Table 1. Our characterization of the invariant polynomials also allows us to give an alternative proof of the universality of the sign invariant neural network SignNet [Lim et al., 2023] (see Appendix C.4).
3.1 Sign Equivariant Linear Maps
First, we consider the important case of degree one polynomials, i.e. sign equivariant linear maps from  $\mathbb{R}^{n\times k}\to \mathbb{R}^{n'\times k}$ . These maps are very limited in expressive power, as they act independently on each eigenvector.
Lemma 1. A linear map  $W: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  is sign equivariant if and only if it can be written as
$$
W (X) = \left[ W _ {1} X _ {1} \dots W _ {k} X _ {k} \right] \tag {4}
$$
for some linear maps  $W_{1},\ldots ,W_{k}:\mathbb{R}^{n}\to \mathbb{R}^{n^{\prime}}$  , where  $X_{i}\in \mathbb{R}^{n}$  is the ith column of  $X\in \mathbb{R}^{n\times k}$
See Appendix B.1 for the proof. Notably, when  $n = n' = 1$ , the linear maps are diagonal matrices.
This means a model with elementwise nonlinearities and sign equivariant linear maps will not capture any interactions between eigenvectors. For instance, when used for parameterizing orthogonally equivariant models as in Section 2.2, such a model would process each principal component direction of the point cloud independently. Hence, the popular approach —outlined in the “Geometric Deep Learning Blueprint” [Bronstein et al., 2021]—of interleaving equivariant linear maps and equivariant nonlinearities [Cohen and Welling, 2016, Zaheer et al., 2017, Kondor and Trivedi, 2018, Maron et al., 2018, 2019, Finzi et al., 2021] is not as fruitful here.
However, one may choose instead different group representations for the input and output space, but our attempts to do this do not lead to efficient models. For instance, a common method to improve expressive power of models that use equivariant linear maps is to use tensor representations [Maron et al., 2018, 2019, Finzi et al., 2021]; in our case, this would correspond having equivariant hidden representations in  $\mathbb{R}^{n\times k^m}$  for some tensor order  $m$ . This is also inefficient, as we explain in Appendix ??; we show that such an approach would have to lift to tensors of at least order 3, and that there are many sign equivariant linear maps between tensors of order 3. There is a possibility that some other group representations may allow the Geometric Deep Learning Blueprint to work better for sign equivariant networks, but we could not find any such representations.
For these reasons, we will now analyze the entire space of sign equivariant polynomials.
3.2 Sign Equivariant Polynomials
Consider polynomials  $p: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  that are sign equivariant, meaning  $p(VS) = p(V)S$  for  $S \in \mathrm{diag}(\{-1, 1\}^k)$ . We can show that a polynomial  $p$  is sign equivariant if and only if it can be written as the elementwise product of a simple (linear) sign equivariant polynomial and a general sign invariant polynomial, followed by another linear sign equivariant map.
Theorem 1. A polynomial  $p: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  is sign equivariant if and only if it can be written
$$
p (V) = W ^ {(2)} \left(\left(W ^ {(1)} V\right) \odot p _ {\text {i n v}} (V)\right) \tag {5}
$$