Title: Expressive Sign Equivariant Networks for Spectral Geometric Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Eigenvectors

spectral

geometry

universal approximation

graph

equivariance

invariance
Abstract: Recent work has shown the utility of developing machine learning models that respect the structure and symmetries of eigenvectors. These works promote sign invariance, since for any eigenvector v the negation -v is also an eigenvector. However, we show that sign invariance is theoretically limited for tasks such as building orthogonally equivariant models and learning node positional encodings for link prediction in graphs. In this work, we demonstrate the benefits of sign equivariance for these tasks. To obtain these benefits, we develop novel sign equivariant neural network architectures. Our models are based on a new analytic characterization of sign equivariant polynomials and thus inherit provable expressiveness properties. Controlled synthetic experiments show that our networks can achieve the theoretically predicted benefits of sign equivariant models.

Expressive Sign Equivariant Networks for Spectral Geometric Learning

Abstract

Recent work has shown the utility of developing machine learning models that respect the structure and symmetries of eigenvectors. These works promote sign invariance, since for any eigenvector v the negation -v is also an eigenvector. However, we show that sign invariance is theoretically limited for tasks such as building orthogonally equivariant models and learning node positional encodings for link prediction in graphs. In this work, we demonstrate the benefits of sign equivariance for these tasks. To obtain these benefits, we develop novel sign equivariant neural network architectures. Our models are based on a new analytic characterization of sign equivariant polynomials and thus inherit provable expressiveness properties. Controlled synthetic experiments show that our networks can achieve the theoretically predicted benefits of sign equivariant models.

1 Introduction

The need to process eigenvectors is ubiquitous in machine learning and the computational sciences. For instance, there is often a need to process eigenvectors of operators associated with manifolds or graphs [Belkin and Niyogi, 2003, Rustamov et al., 2007], principal components (PCA) of arbitrary datasets [Pearson, 1901], and eigenvectors arising from implicit or explicit matrix factorization methods [Levy and Goldberg, 2014, Qiu et al., 2018]. However, eigenvectors are not merely unstructured data-they have rich structure in the form of symmetries [Ovsjanikov et al., 2008].

Specifically, eigenvectors have sign and basis symmetries. An eigenvector v is sign symmetric in the sense that the sign-flipped vector -v is also an eigenvector of the same eigenvalue. Basis symmetries occur when there is a repeated eigenvalue, as then there are infinitely many choices of eigenvector basis for the same eigenspace. Prior work has developed neural networks that are invariant to these symmetries, improving empirical performance in several settings [Lim et al., 2023].

The goal of this paper is to demonstrate why sign equivariance can be useful and to characterize fundamental expressive sign equivariant architectures. Our first contribution is to show that sign equivariant models are a natural choice for several applications, whereas sign invariant architectures are provably insufficient for these applications. First, we show that sign and

Sign

Equivariant

0

basis invariant networks are theoretically limited in expressive power for learning edge representations (and more generally multi-node representations) in graphs because they learn structural node embeddings that are known to be limited for link prediction and multi-node tasks [Srinivasan and

Technion, NVIDIA

Ribeiro, 2019, Zhang et al., 2021]. In contrast, we show that sign equivariant models can bypass this limitation by maintaining positional information in node embeddings. Furthermore, we show that sign equivariance combined with PCA can be used to parameterize expressive orthogonally equivariant point cloud models, thus giving an efficient alternative to PCA-based frame averaging [Puny et al., 2022, Atzmon et al., 2022]. In contrast, sign invariant models can only parameterize orthogonally invariant models in this framework, which excludes many important application areas.

The second contribution of this work is to develop the first sign equivariant neural network architectures, with provable expressiveness guarantees. We first present a difficulty in developing sign equivariant models: the 'Geometric Deep Learning Blueprint' [Bronstein et al., 2021] suggests developing an equivariant neural network by interleaving equivariant linear maps and equivariant elementwise nonlinearities [Cohen and Welling, 2016, Ravanbakhsh et al., 2017, Finzi et al., 2021], but we show that our attempts to apply this approach are insufficient for expressive sign equivariant models. Namely, we show that sign equivariant linear maps between various input and output representations are very limited in their expressive power.

Hence, to develop our models, we derive a complete characterization of the sign equivariant polynomial functions. The form of these equivariant polynomials directly inspires our equivariant neural network architectures. Further, our architectures inherit the theoretical expressive power guarantees of the equivariant polynomials. Our characterization is also broadly useful for analysis and development of sign-symmetry-respecting architectures-for instance, we provide a new proof of the universality of SignNet [Lim et al., 2023] by showing that it can approximate all sign invariant polynomials.

To validate our theoretical results, we conduct various numerical experiments on synthetic datasets. Experiments in link prediction, n-body problems, and node clustering in graphs support our theory and demonstrate the utility of sign equivariant models.

1.1 Background

Let f : R n × k → R n × k be a function that takes eigenvectors v 1 , . . . , v k ∈ R n of an underlying matrix as input, and outputs representations f ( v 1 , . . . , v k ) . We often concatenate the eigenvectors into a matrix V = [ v 1 , . . . , v k ] ∈ R n × k , and write f ( V ) as the application of f . For simplicity, in this work we assume the eigenvectors come from a symmetric matrix, so they are taken to be orthonormal.

Sign and basis symmetries. Eigenvectors have symmetries, because there are many possible choices of eigenvectors of a matrix. For instance, if v is a unit-norm eigenvector of a matrix, then so is the sign-flipped -v . If the eigenvalue of v is simple, then -v is the only other choice of unit-norm eigenvector of this eigenvalue.

If v 1 , . . . v m are an orthonormal basis of eigenvectors for the same eigenspace (meaning they all have the same eigenvalue), then there are infinitely many other choices of orthonormal basis for this eigenspace; these other choices of basis can be written as V Q , where V = [ v 1 . . . v m ] ∈ R n × m and Q ∈ O ( m ) is an arbitrary orthogonal matrix.

We refer to these symmetries collectively as sign and basis symmetries, or more simply as eigenvector symmetries. Note that sign symmetries are a special case of basis symmetries, as -1 and 1 are the only orthogonal 1 × 1 matrices. Previous work has developed neural networks that are invariant to these symmetries-that is, networks that have the same output for any choice of sign or basis of the eigenvector inputs [Lim et al., 2023].

Sign equivariance means that if we flip the sign of an eigenvector, then the corresponding column of the output of a function f has its sign flipped. In other words, for all choices of signs s 1 , . . . , s k ∈ {-1 , 1 } k ,

where A : ,j is the j -th column of an n × k matrix A . See Figure 1 for an illustration. In matrix form, letting diag( {-1 , 1 } k ) represent all k × k diagonal matrices with -1 or 1 on the diagonal, f is sign equivariant if

As O (1) = {-1 , 1 } , we can write sign equivariance as equivariance with respect to a direct product of orthogonal groups O (1) × . . . × O (1) . This is different from the equivariance to a single orthogonal group O ( d ) considered in works on Euclidean-group equivariant networks [Thomas et al., 2018].

(a) Eigenvector 1

(b) Sign Invariant

(c) Sign Equivariant

Permutation equivariance is often also a desirable property of our functions f . We say that f is permutation equivariant if f ( PV ) = Pf ( V ) for all n × n permutation matrices P . For instance, eigenvectors of matrices associated with simple graphs of size n have such permutation symmetries, as the ordering of nodes is arbitrary.

2 Applications of Sign Equivariance

In this section, we present several applications for which modeling networks with sign equivariant architectures is beneficial. We identify that sign invariant networks are provably insufficient for these tasks, motivating the development of sign equivariant networks to address these limitations.

2.1 Multi-Node Representations and Link Prediction

In several settings, we desire a machine learning model that computes representations for tuples of several nodes in a graph. For instance, link prediction tasks generate probabilities for pairs of nodes, and both hyperedge prediction and subgraph prediction tasks learn representations for collections of nodes in a graph [Alsentzer et al., 2020, Wang et al., 2023]. For ease of exposition, the rest of this section discusses link prediction, though the discussion applies to general multi-node tasks as well.

In link prediction, we typically want to learn structural node-pair representations , meaning adjacencypermutation equivariant functions that give a representation for each pair of nodes; more precisely, a structural node-pair representation is a map f : R n × n → R n × n such that f ( PAP ⊤ ) = Pf ( A ) P ⊤ , where f ( A ) i,j is the representation of the pair of nodes ( i, j ) in the graph with adjacency matrix A [Srinivasan and Ribeiro, 2019] (see Appendix A.5 for more discussion). One method to do this is to use a graph model such as a standard GNN to learn node representations z i , and then obtain a nodepair representation for ( i, j ) as some function f decode ( z i , z j ) of z i and z j . However, this approach is limited because standard GNNs learn structural node encodings -that is, adjacency-permutation equivariant node features f node : R n × n → R n such that f node ( PAP ⊤ ) = Pf node ( A ) [Srinivasan and Ribeiro, 2019, Zhang et al., 2021]. 2 Structural node encodings give automorphic nodes the same representation, which can be problematic since automorphic nodes can be far apart in the graph. For instance, in Figure 2, u 1 and u 2 are automorphic, so a link predictor based on structural node encodings would give both u 1 and u 2 the same probability of connecting to w , but one would expect u 1 to have a higher probability of connecting to w . Most state-of-the-art link prediction methods on the Open Graph Benchmark leaderboards [Hu et al., 2020] were deliberately developed to avoid the issues of structural node encodings.

One way to surpass the limitations of structural node encodings is to use positional node embeddings , which can assign different values to automorphic nodes. Intuitively, positional encodings capture information such as distances between nodes and global position of nodes in the graph (see [Srinivasan and Ribeiro, 2019] for a formal definition). Laplacian eigenvectors are an important example of node

positional embeddings that capture much useful information of graphs [Chung, 1997, Von Luxburg, 2007]. In Figure 2 (a), the first nontrivial Laplacian eigenvector captures cluster structure in the graph, and as such assigns u 1 and u 2 very different values.

Pitfalls of sign and basis invariance. When processing eigenvectors of matrices associated with graphs, invariance to the symmetries of the eigenvectors has been found useful [Dwivedi et al., 2022a, Lim et al., 2023], especially for graph classification tasks. However, we show that exact invariance to these symmetries removes positional information , and thus the outputs of sign invariant or basis invariant networks are in fact structural node encodings (see Appendix A.5). 3 Hence, eigenvectorsymmetry-invariant networks cannot learn node representations that distinguish automorphic nodes, and thus face the aforementioned difficulties when used for link prediction or multi-node tasks:

Proposition 1. Let f : R n × k → R n × d out be a permutation equivariant function, and let V = [ v 1 , . . . , v k ] ∈ R n × k be k orthonormal eigenvectors of an adjacency matrix A . Let nodes i and j be automorphic, and let z i and z j ∈ R d out be their embeddings, i.e, the i th and j th row of Z = f ( V ) .

If f is sign invariant and the eigenvalues associated with the v l are simple and distinct, then z i = z j .

If f is basis invariant and v 1 , . . . , v k are a basis for some number of eigenspaces of A then z i = z j .

A novel link prediction approach via sign equivariance. The problem z i = z j arises from the sign/basis invariances, which remove crucial positional information. We instead propose using sign equivariant networks (as in Section 3) to learn node representations z i = f ( V ) i, : ∈ R k . These representations z i maintain positional information for each node thanks to preserving sign information (see Figure 2 (c)). Then we use a sign invariant decoder f decode ( z i , z j ) = f decode ( Sz i , Sz j ) for S ∈ diag( {-1 , 1 } k ) to obtain node-pair representations. For instance, the commonly used f decode = MLP( z i ⊙ z j ) , where ⊙ is the elementwise product, is sign invariant. When the eigenvalues are distinct, this approach has the desired invariances (yielding structural node-pair representations) and also maintains positional information in the node embeddings; see Appendix A.5 for a proof of the invariances, and Appendix A.5.1 for an example of where sign equivariant models can be used to compute strictly more expressive node-pair representations than sign invariant models. More details and the proof of Proposition 1 are in Appendix A.4.

Our sign equivariance based approach differs substantially from existing methods for learning structural pair representations without being bottlenecked by structural node representations. Many of these methods are based on labeling tricks [Zhang et al., 2021, Wang et al., 2023], whereby the representation for a node-pair is obtained by labeling the two nodes in the pair and then processing an enclosing subgraph. Without special modifications [Zhu et al., 2021, Chamberlain et al., 2023], this requires a separate expensive subgraph extraction and forward pass for each node-pair. In contrast, our method only requires one forward pass on the original graph to compute all positional node embeddings, after which pair representations can be obtained with a cheap, parallelizable decoding.

2.2 Orthogonal Equivariance

For various applications in modelling physical systems, we desire equivariance to rigid transformations; thus, orthogonally equivariant models have been a fruitful research direction in recent years [Thomas et al., 2018, Weiler et al., 2018, Anderson et al., 2019, Deng et al., 2021]. We say that a function f : R n × k → R n × k is orthogonally equivariant if f ( XQ ) = f ( X ) Q for any Q ∈ O ( k ) , where O ( k ) is the set of orthogonal matrices in R k × k . Orthogonal equivariance imposes infinitely many constraints on the function f . Several works have approached this problem by reducing to a finite set of constraints using so-called Principal Component Analysis (PCA) based frames [Puny et al., 2022, Atzmon et al., 2022, Xiao et al., 2020].

PCA-frame methods take an input X ∈ R n × k , compute orthonormal eigenvectors R X ∈ O ( k ) of the covariance matrix cov( X ) = ( X -1 n 11 ⊤ X ) ⊤ ( X -1 n 11 ⊤ X ) (assumed to have distinct eigenvalues), then average outputs of a base model h for each of the 2 k sign-flipped inputs XR X S , where S ∈ diag( {-1 , 1 } k ) . We instead suggest using a sign equivariant network to parameterize an efficient O ( k ) equivariant model, which allows us to bypass the need to average the exponentially

f

(

):

R

h

R

⊤

f

(

):

R

h

R

⊤

many sign-flipped inputs. For a sign equivariant network h , we define our model f to be

See Figure 3 for an illustration. Intuitively, this first transforms X by R X into a nearly canonical orientation that is unique up to sign flips; this can be seen as writing the points in the principal components basis, or aligning the principal components of X with the coordinate axes. Then we process XR X using the model h that respects the sign symmetries, and finally, we incorporate orientation information back into the output by post-multiplying by R ⊤ X . Our approach only requires one forward pass through h , whereas frame averaging requires 2 k forward passes through a base model. The following proposition shows that f is O ( k ) equivariant, and inherits universality properties of h . 4

Proposition 2. Consider a domain X ⊆ R n × k such that each X ∈ X has distinct covariance eigenvalues, and let R X be a choice of orthonormal eigenvectors of cov( X ) for each X ∈ X . If h : X ⊆ R n × k → R n × k is sign equivariant, and if f ( X ) = h ( XR X ) R ⊤ X , then f is well defined and orthogonally equivariant.

Moreover, if h is from a universal class of sign equivariant functions, then the f of the above form universally approximate O ( k ) equivariant functions on X .

We include a proof of this result in Appendix A.6. This result also follows from Theorems 3.1 and 3.3 of Kaba et al. [2023], who show that generally we can canonicalize up to a subgroup K of a group G , and achieve G -equivariance via a K -equivariant base predictor. In our case, G = O ( k ) and K = {-1 , 1 } k

Sign invariance only gives orthogonal invariance. In a similar way, a sign invariant model can be used to obtain an orthogonally invariant model with PCA frames, but it cannot be used for orthogonal equivariance; instead, sign equivariance is needed.

3 Sign Equivariant Polynomials and Networks

In this section, we analytically characterize the sign equivariant polynomials, and use this characterization to develop sign equivariant architectures. As equivariant polynomials universally approximate continuous equivariant functions [Yarotsky, 2022], our architectures inherit universality guarantees. We summarize our results on polynomials and neural network architectures in Table 1. Our characterization of the invariant polynomials also allows us to give an alternative proof of the universality of the sign invariant neural network SignNet [Lim et al., 2023] (see Appendix C.4).

3.1 Sign Equivariant Linear Maps

First, we consider the important case of degree one polynomials, i.e. sign equivariant linear maps from R n × k → R n ′ × k . These maps are very limited in expressive power, as they act independently on each eigenvector.

for some linear maps W 1 , . . . , W k : R n → R n ′ , where X i ∈ R n is the i th column of X ∈ R n × k .

See Appendix B.1 for the proof. Notably, when n = n ′ = 1 , the linear maps are diagonal matrices.

This means a model with elementwise nonlinearities and sign equivariant linear maps will not capture any interactions between eigenvectors. For instance, when used for parameterizing orthogonally equivariant models as in Section 2.2, such a model would process each principal component direction of the point cloud independently. Hence, the popular approach -outlined in the 'Geometric Deep Learning Blueprint' [Bronstein et al., 2021]-of interleaving equivariant linear maps and equivariant nonlinearities [Cohen and Welling, 2016, Zaheer et al., 2017, Kondor and Trivedi, 2018, Maron et al., 2018, 2019, Finzi et al., 2021] is not as fruitful here.

However, one may choose instead different group representations for the input and output space, but our attempts to do this do not lead to efficient models. For instance, a common method to improve expressive power of models that use equivariant linear maps is to use tensor representations [Maron et al., 2018, 2019, Finzi et al., 2021]; in our case, this would correspond having equivariant hidden representations in R n × k m for some tensor order m . This is also inefficient, as we explain in Appendix ?? ; we show that such an approach would have to lift to tensors of at least order 3, and that there are many sign equivariant linear maps between tensors of order 3. There is a possibility that some other group representations may allow the Geometric Deep Learning Blueprint to work better for sign equivariant networks, but we could not find any such representations.

For these reasons, we will now analyze the entire space of sign equivariant polynomials.

3.2 Sign Equivariant Polynomials

Consider polynomials p : R n × k → R n ′ × k that are sign equivariant, meaning p ( V S ) = p ( V ) S for S ∈ diag( {-1 , 1 } k ) . We can show that a polynomial p is sign equivariant if and only if it can be written as the elementwise product of a simple (linear) sign equivariant polynomial and a general sign invariant polynomial, followed by another linear sign equivariant map.

Theorem 1. A polynomial p : R n × k → R n ′ × k is sign equivariant if and only if it can be written

for sign equivariant linear W (2) and W (1) , and a sign invariant polynomial p inv : R n × k → R n ′ × k .

This reduction of sign equivariant polynomials to sign invariant polynomials combined with simple operations is convenient, as it enables us to leverage recent universal models for sign invariant functions [Lim et al., 2023]. The proof of this statement is in Appendix C, which proceeds by showing that sign equivariance leads to linear constraints on the coefficients of a polynomial, which requires the polynomial to take the form stated in the Theorem.

̸

3.3 Sign Equivariance without Permutation Symmetries

Using Theorem 1, we can now develop sign equivariant architectures. We parameterize sign equivariant functions f : R n × k → R n ′ × k as a composition of layers f l , each of the form

in which the W ( l ) i : R n → R n ′ are arbitrary linear maps, and SignNet l : R n × k → R n ′ × k is sign invariant [Lim et al., 2023]. In the case of n = n ′ = 1 , there is a simple universal form: we can write a sign equivariant function f : R k → R k as f ( v ) = v ⊙ MLP( | v | ) , where | v | is the elementwise absolute value. These two architectures are universal because they can approximate sign equivariant polynomials. Here, the sign invariant part captures interactions between eigenvectors that the equivariant linear maps cannot.

Proposition 3. Functions of the form v ↦→ v ⊙ MLP( | v | ) universally approximate continuous sign equivariant functions f : R k → R k .

Compositions f 2 ◦ f 1 of functions f l as in equation 6 universally approximate continuous sign equivariant functions f : R n × k → R n ′ × k .

3.4 Sign Equivariance and Permutation Equivariance

For models on eigenvectors that stem from graphs or point clouds, in addition to sign equivariance, we may demand permutation equivariance, i.e., f ( PV ) = Pf ( V ) for all permutation matrices P ∈ R n × n . To add permutation equivariance to our neural network architecture from Section 3.3, we use it within the framework of DeepSets for Symmetric Elements (DSS) [Maron et al., 2020]. For a hidden dimension size of d f , each layer f l : R n × k × d f → R n × k × d f of our DSS-based sign equivariant network takes the following form on row i :

̸

where f (1) l and f (2) l are sign equivariant functions as in Section 3.3. Sometimes we take d f = 1 , in which case we can use the simpler R k → R k sign equivariant networks ( v ⊙ MLP( | v | ) ) as f (1) l and f (2) l . If we have graph information, then we can do message-passing by changing the sum over j = i to a sum over a neighborhood of node i . DSS has universal approximation guarantees [Maron et al., 2020], but they only apply for groups that act as permutation matrices, whereas the sign group {-1 , 1 } k does not. Hence, the universal approximation properties of our proposed DSS-based architecture are still an open question.

4 Experiments

Our theoretical results in Section 2 predict benefits of sign equivariance in various tasks: link prediction in nearly symmetric graphs, orthogonally equivariant simulations in n-body problems, and node clustering with positional information. Next, we probe these suggested benefits empirically.

4.1 Link Prediction in Nearly Symmetric Graphs.

We begin with a synthetic link prediction task, which is carefully controlled to test the theoretically foreseen benefits of sign equivariance explained in Section 2.1. With the intuition of Figure 2 we first

either generate an Erd˝ os-Rényi [Erd˝ os et al., 1960] or Barabási-Albert [Barabási and Albert, 1999] random graph H of 1000 nodes. Then we form a larger graph G that contains two disjoint copies of H , along with 1000 uniformly-randomly added edges (both between and within copies of H ). Without the random edges, each node in one copy of H is automorphic to the corresponding node in the other copy, so we expect many nodes to be nearly automorphic with the randomly added edges.

In Table 2, we show the link prediction performance of several models that learn structural edge representations. The methods that use eigenvectors have a sign invariant final prediction for each edge. GCN [Kipf and Welling, 2017] where the node features are all ones and SignNet [Lim et al., 2023] both completely fail on the Erd˝ os-Rényi task (these two models map automorphic nodes to the same embedding), while our sign equivariant model outperforms all methods. We also try two eigenvector baselines that maintain node positional information, but do not update eigenvector representations: taking the dot product V ⊤ i, : V j, : to be the logit of a link existing, or learning a simple decoder MLP( V i, : ⊙ V j, : ) . Both perform substantially worse than our sign equivariant model, which shows that updating eigenvector representations is important here. Further, the sign equivariant model takes comparable runtime to GCN, and is significantly faster than SignNet. This is because we use networks of the form v ↦→ v ⊙ MLP( | v | ) in these experiments instead of the full SignNet-based model in equation 6. See Appendix E.2 for more details.

4.2 Orthogonal Equivariance in n-body Problems

3

5

7

9

11

Dimension

2

4

6

Seconds / epoch

Sign

Frame

3

5

7

9

11

Dimension

10

4

10

3

MSE

Sign

Frame

In this section, we empirically test the ability of our sign equivariant models to parameterize orthogonally equivariant functions on point clouds, as outlined in Section 2.2. For this purpose, we consider simulating n-body problems, following the setup in Fuchs et al. [2020] and building on the code from Puny et al. [2022]. To test the favorable scaling of our method in the dimension d of the problem against the exponential 2 d scaling of frame averaging, we generalize this problem to general dimensions d ≥ 3 . We maintain the choice of n = 5 particles, and generate new point clouds using the same procedure as in Fuchs et al. [2020] (sampling random points and initial velocities in a general dimension d ). We measure model performance via mean squared error (MSE). We use a DSS-based model that we describe in more detail in Appendix E.3.

Figure 4 illustrates the runtime and MSE. The sign equivariant model scales well with dimension-the time-per-epoch is nearly constant as we increase the dimension. In contrast, frame averaging suffers from the expected exponential slowdown with dimension, and runs out of memory on a 32GB V100 GPU for d = 11 . Considering the MSE, the equivariant model's performance closely follows that of frame averaging, i.e., we only have a small loss in accuracy with much better scalability. For d = 3 , the sign equivariant model has an MSE of .00646, compared to the .00575 of frame averaging [Puny et al., 2022]. Additional d = 3 comparisons to other baselines are included in Appendix E.3.

4.3 Node Clustering with Positional Information

As explained in Section 2.1, some applications on graph data call for positional node embeddings that can assign different representations to automorphic nodes. For instance, consider community detection or node clustering tasks on graphs, where a model makes a prediction for each node that

assigns it to a cluster. Structural encodings are insufficient for this task, as there may be automorphic or nearly-automorphic nodes that are far apart in the graph but look alike in a structural encoding. Hence, node structural encodings would guide the model to assign these nodes to the same cluster, even though they should belong to different clusters. As a concrete example, consider a graph of two clusters, and using Laplacian eigenvectors as positional encodings. The first nontrivial eigenvector will tend to assign a positive sign to one cluster and a negative sign to the other cluster. Thus, the sign information in the eigenvectors is crucial, so we expect sign equivariant models to perform well.

We test models on the CLUSTER dataset [Dwivedi et al., 2022a] for semi-supervised node clustering (viewed as node classification) in synthetic graphs. In these experiments, we build on the empirically well-performing GraphGPS model [Rampasek et al., 2022], and incorporate our sign equivariant models to update eigenvector representations within the version of GraphGPS that uses PEG [Wang et al., 2022] to process positional encodings. See Appendix E.4 for more experimental details.

As seen in Table 3, our sign equivariant models outperform all of the other GraphGPS-based eigenvector methods. Moreover, we achieve the second best performance across all methods, showing that sign equivariant models can indeed achieve the theoretically expected benefits in this setting.

5 Related Work

Structural and Positional Representations. Especially for link prediction, the need for structural node-pair representations that are not obtained from structural node representations has been discussed in several works [Srinivasan and Ribeiro, 2019, Zhang et al., 2021, Cotta et al., 2023]. As such, various methods have been developed for learning structural node-pair representations that incorporate node positional information. SEAL and other labeling-trick based methods [Zhang and Chen, 2018, Zhang et al., 2021] use added node features depending on the node-pair that we want a representation of. This is empirically successful in many tasks, but typically requires a separate subgraph extraction and forward pass through a GNN for each node-pair under consideration. Distance encoding [Li et al., 2020] uses relative distances between nodes to capture positional information. PEG [Wang et al., 2022] similarly maintains positional information by using eigenvector distances between nodes in each layer of a GNN, but does not update eigenvector representations. Identity-aware GNNs [You et al., 2021] and Neural Bellman-Ford Networks [Zhu et al., 2021] learn pair representations by conditioning on a source node from the pair.

Eigenvectors as Graph Positional Encodings. When using eigenvectors of graphs as node positional encodings for graph models like GNNs and Graph Transformers, many works have noted the need to address the sign ambiguity of the eigenvectors. This is often done by encouraging sign invariance through data augmentation-the signs of the eigenvectors are chosen randomly in each iteration of training [Dwivedi et al., 2022a,b, Kreuzer et al., 2021, Mialon et al., 2021, Kim et al., 2022, He et al., 2022, Müller et al., 2023]. In contrast, SignNet [Lim et al., 2023] enforces exact sign invariance, by processing eigenvectors with a sign invariant neural architecture; this approach has been taken by some recent works [Rampasek et al., 2022, Geisler et al., 2023, Murphy et al., 2023].

Equivariant Neural Network Design. Equivariant neural network architectures have been proposed for various types of data and symmetry groups. A common paradigm is to interleave equivariant linear maps and equivariant pointwise nonlinearities [Wood and Shawe-Taylor, 1996, Cohen and Welling, 2016, 2017, Ravanbakhsh et al., 2017, Maron et al., 2018, Kondor and Trivedi, 2018, Finzi et al., 2021, Bronstein et al., 2021, Pearce-Crump, 2022]; this is often used when the group acts as some subset of the permutation matrices. However, the sign group does not act as permutation matrices, and as we explained above this approach is not expressive for sign equivariant models.

More similarly to our approach, many equivariant machine learning works heavily leverage invariant or equivariant polynomials (or other equivariant nonlinear functions). These works include polynomials as operations within a network [Thomas et al., 2018, Puny et al., 2023], add polynomials as features [Yarotsky, 2022, Villar et al., 2021], build networks that take a similar form to equivariant polynomials [Villar et al., 2021], and/or analyze neural network expressive power by determining which equivariant polynomials a given architecture can compute [Zaheer et al., 2017, Segol and Lipman, 2019, Maron et al., 2019, 2020, Chen et al., 2020, Dym and Maron, 2021, Puny et al., 2023].

6 Conclusion

In this work, we identify and study an important method of respecting the symmetries of eigenvector data-sign equivariant models. For multi-node representation tasks, link prediction, and orthogonally equivariant tasks, sign equivariance provides a natural inductive bias; in contrast, we show that sign invariant models are provably limited in these tasks. To develop sign equivariant neural networks, we analytically characterize the sign equivariant polynomials, and then define neural networks that parameterize functions of similar form. Our neural networks are thus expressive, and inherit universal approximation guarantees of the equivariant polynomials. In several experiments, we show that our neural networks can indeed achieve the theoretically predicted benefits of sign equivariant models.

Limitations and Future Work. While we developed sign equivariant architectures in this work, we did not explore basis-change equivariant architectures, which would have the desired symmetries for inputs with repeated eigenvalues. As eigenvalue multiplicities are known to occur in many real-world graphs [Lim et al., 2023], future work in this area could be useful. Further, we give evidence that sign equivariance could help in some node-level and multi-node-level prediction tasks on graphs, but we do not have theoretical reason to believe that sign equivariance could help in graph-level representation tasks, which for instance are common in molecule processing. Our theoretical results are focused on expressive power, but we do not have results on other properties that are important for learning, such as optimization [Xu et al., 2021], stability [Wang et al., 2022, Huang et al., 2023], or generalization [Keriven and Vaiter, 2023]. Finally, while we can prove universality of our models in the non-permutation-equivariant setting, we do not know of the exact expressive power in the permutation equivariant setting. Lim et al. [2023] also faces this issue for sign invariant models; future work on analyzing and possibly improving the expressive power of these models - if they are not universal - is promising.

Acknowledgments

We would like to thank Yaron Lipman for contributing significantly early on in this work. We would like to thank Maks Ovsjanikov for noting that basis invariant models give automorphic nodes the same representation, and also Johannes Lutzeyer and Michael Murphy for helpful comments. We would also like to thank the reviewers of the Physics4ML workshop at ICLR 2023 for helpful feedback and close reading. DL is supported by an NSF Graduate Fellowship. SJ acknowledges support from NSF AI Institute NSF CCF-2112665, NSF Award 2134108, and Office of Naval Research Grant N00014-20-1-2023 (MURI ML-SCOPE).

References

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In International Joint Conference on Artificial Intelligence , 2021.

Emily Alsentzer, Samuel Finlayson, Michelle Li, and Marinka Zitnik. Subgraph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 8017-8029, 2020.

Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, and Yaron Lipman. Frame averaging for equivariant shape space learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 631-641, 2022.

Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. In Science , pages 509-512. American Association for the Advancement of Science, 1999.

Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation , 15(6):1373-1396, 2003.

Xavier Bresson and Thomas Laurent. Residual gated graph ConvNets. In preprint arXiv:1711.07553 , 2017.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. preprint arXiv:2104.13478 , 2021.

Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M Bronstein, and Max Hansmire. Graph neural networks for link prediction with subgraph sketching. In International Conference on Learning Representations (ICLR) , 2023.

Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning (ICML) , pages 34693489. PMLR, 2022.

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 10383-10395, 2020.

Fan Chung. Spectral graph theory . American Mathematical Society, 1997.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning (ICML) , pages 2990-2999. PMLR, 2016.

Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations (ICLR) , 2017.

Leonardo Cotta, Beatrice Bevilacqua, Nesreen Ahmed, and Bruno Ribeiro. Causal lifting and link prediction. preprint arXiv:2302.01198 , 2023.

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for SO(3)-equivariant networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12200-12209, 2021.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. In JMLR , 2022a.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations (ICLR) , volume 10, 2022b.

Nadav Dym and Steven J Gortler. Low dimensional invariant embeddings for universal geometric learning. preprint arXiv:2205.02956 , 2022.

Nadav Dym and Haggai Maron. On the universality of rotation equivariant point cloud networks. In International Conference on Learning Representations (ICLR) , 2021.

Paul Erd˝ os, Alfréd Rényi, et al. On the evolution of random graphs. Publ. Mathematical Institute Hungarian Academy of Science , 5(1):17-60, 1960.

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning (ICML) , pages 3318-3328. PMLR, 2021.

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3)-transformers: 3D rototranslation equivariant attention networks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 1970-1981, 2020.

William Fulton and Joe Harris. Representation theory: a first course , volume 129. Springer Science &Business Media, 2013.

Simon Geisler, Yujia Li, Daniel Mankowitz, Ali Taylan Cemgil, Stephan Günnemann, and Cosmin Paduraru. Transformers meet directed graphs. preprint arXiv:2302.00049 , 2023.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML) , pages 1263-1272. PMLR, 2017.

Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 2008.

Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann LeCun, and Xavier Bresson. A generalization of ViT/MLP-Mixer to graphs. preprint arXiv:2212.13350 , 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 22118-22133, 2020.

Yinan Huang, William Lu, Joshua Robinson, Yu Yang, Muhan Zhang, Stefanie Jegelka, and Pan Li. On the stability of expressive positional encodings for graph neural networks. arXiv preprint arXiv:2310.02579 , 2023.

Md Shamim Hussain, Mohammed J Zaki, and Dharmashankar Subramanian. Global self-attention as a replacement for graph convolution. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 655-665, 2022.

Sékou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equivariance with learned canonicalization functions. In International Conference on Machine Learning (ICML) , pages 15546-15566. PMLR, 2023.

Nicolas Keriven and Samuel Vaiter. What functions can graph neural networks compute on random graphs? the role of positional encoding. arXiv preprint arXiv:2305.14814 , 2023.

Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning (ICML) , pages 2688-2697. PMLR, 2018.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , volume 5, 2017.

Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International Conference on Machine Learning (ICML) , pages 5361-5370. PMLR, 2020.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning , pages 2747-2755. PMLR, 2018.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, 2021.

Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems (NeurIPS) , volume 2, 1989.

Shyi-Long Lee, Yeung-Long Luo, Bruce E Sagan, and Yeong-Nan Yeh. Eigenvector and eigenvalues of some special graphs. IV. Multilevel circulants. International journal of quantum chemistry , 41 (1):105-116, 1992.

Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 27, 2014.

Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 4465-4478, 2020.

Derek Lim, Joshua David Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In International Conference on Learning Representations (ICLR) , 2023.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations (ICLR) , volume 6, 2018.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International Conference on Machine Learning (ICML) , pages 4363-4371. PMLR, 2019.

Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In International Conference on Machine Learning (ICML) , pages 6734-6744. PMLR, 2020.

Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal. GraphiT: Encoding graph structure in transformers. In preprint arXiv:2106.05667 , 2021.

Luis Müller, Mikhail Galkin, Christopher Morris, and Ladislav Rampášek. Attending to graph transformers. preprint arXiv:2302.04181 , 2023.

Michael Murphy, Stefanie Jegelka, Ernest Fraenkel, Tobias Kind, David Healey, and Thomas Butler. Efficiently predicting high resolution mass spectra with graph neural networks. preprint arXiv:2301.11419 , 2023.

Ryan Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Relational pooling for graph representations. In International Conference on Machine Learning (ICML) , pages 4663-4673. PMLR, 2019.

Maks Ovsjanikov, Jian Sun, and Leonidas Guibas. Global intrinsic symmetries of shapes. In Computer graphics forum , volume 27, pages 1341-1348. Wiley Online Library, 2008.

Edward Pearce-Crump. Brauer's group equivariant neural networks. preprint arXiv:2212.08630 , 2022.

Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science , 2(11):559-572, 1901.

Daniel L Pimentel-Alarcón, Nigel Boston, and Robert D Nowak. A characterization of deterministic sampling patterns for low-rank matrix completion. IEEE Journal of Selected Topics in Signal Processing , 10(4):623-636, 2016.

Omri Puny, Matan Atzmon, Heli Ben-Hamu, Edward J Smith, Ishan Misra, Aditya Grover, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations (ICLR) , volume 10, 2022.

Omri Puny, Derek Lim, Bobak T Kiani, Haggai Maron, and Yaron Lipman. Equivariant polynomials for graph neural networks. preprint arXiv:2302.11556 , 2023.

Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying DeepWalk, LINE, PTE, and node2vec. In Proceedings of the eleventh ACM international conference on web search and data mining , pages 459-467, 2018.

Ladislav Rampasek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing. In International Conference on Machine Learning (ICML) , pages 2892-2901. PMLR, 2017.

Raif M Rustamov et al. Laplace-Beltrami eigenfunctions for deformation invariant shape representation. In Symposium on geometry processing , volume 257, pages 225-233, 2007.

Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International Conference on Machine Learning (ICML) , pages 9323-9332. PMLR, 2021.

Nimrod Segol and Yaron Lipman. On universal equivariant set networks. In International Conference on Learning Representations (ICLR) , volume 7, 2019.

Balasubramaniam Srinivasan and Bruno Ribeiro. On the equivalence between positional node embeddings and structural graph representations. In International Conference on Learning Representations (ICLR) , 2019.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3D point clouds. In NeurIPS Molecules and Materials Workshop , 2018.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR) , volume 6, 2018.

Soledad Villar, David Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, 2021.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17(4):395-416, 2007.

Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. In International Conference on Learning Representations (ICLR) , volume 10, 2022.

Xiyuan Wang, Pan Li, and Muhan Zhang. Improving graph neural networks on multi-node tasks with labeling tricks. preprint arXiv:2304.10074 , 2023.

Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Jeffrey Wood and John Shawe-Taylor. Representation theory and invariant neural networks. Discrete applied mathematics , 69(1-2):33-60, 1996.

Zelin Xiao, Hongxin Lin, Renjie Li, Lishuai Geng, Hongyang Chao, and Shengyong Ding. Endowing deep 3D models with rotation invariance based on principal component analysis. In IEEE International Conference on Multimedia and Expo (ICME) , pages 1-6. IEEE, 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR) , volume 7, 2019.

Keyulu Xu, Mozhi Zhang, Stefanie Jegelka, and Kenji Kawaguchi. Optimization of graph neural networks: Implicit acceleration by skip connections and more depth. In International Conference on Machine Learning , pages 11592-11602. PMLR, 2021.

Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation , 55(1):407-474, 2022.

Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Association for the Advancement of Artificial Intelligence (AAAI) , volume 35, pages 10737-10745, 2021.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In Advances in Neural Information Processing Systems (NeurIPS) , volume 30, pages 3391-3401, 2017.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 9061-9073, 2021.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 29476-29490, 2021.

A Applications of Sign Equivariance

A.1 Improving Invariant Eigenvector Networks

Neural networks that are invariant to eigenvector symmetries have been shown to empirically improve graph learning models and achieve theoretically high expressive power. SignNet [Lim et al., 2023], a sign invariant neural network, takes the form

for neural networks ρ and ϕ . This directly enforces invariant representations, without any intermediate equivariant representations. However, many successful invariant models first have many equivariant layers before a final invariant operation as equivariant layers are more expressive: this includes convolutional neural networks [LeCun et al., 1989], message passing graph neural networks [Gilmer et al., 2017], invariant graph networks [Maron et al., 2018], and group convolutional neural networks [Cohen and Welling, 2016]. Thus, sign equivariant layers may lead to better sign invariant networks. Moreover, sign equivariant layers may improve on other aspects of SignNet, such as expressiveness of node features (Proposition 1) and efficiency (Appendix A.2)

A.2 Efficiency Gains from Sign Equivariant Networks

Here, we show that our sign equivariant models can reduce the complexity of equivariant or invariant networks for two different types of applications. Throughout, we consider functions f : R n × k → R n × k , and we consider our permutation equivariant and sign equivariant DSS-based architecture from Section 3.4.

̸

The time cost (in floating point operations) per layer of our DSS-based model is O ( n ( kd + d 2 )) , where d is the maximum hidden dimension of the MLP and we assume constant depth MLPs. To see this, note that we can precompute ∑ n j =1 V j, : , so that each ∑ j = i V j, : can be computed in constant time by subtracting V i, : from the total sum. Then for each of the n rows, the MLPs require O ( kd + d 2 ) to evaluate matrix multiplications. In this process, we only form tensors of size O ( n ( k + d )) , as the inputs and outputs are of size O ( nk ) , and the hidden layers of the MLPs form tensors of size O ( nd ) .

A.2.1 Efficient Orthogonally Equivariant Networks

Consider the case of O ( k ) equivariant models f : R n × k → R n × k such that f ( XQ ) = f ( X ) Q for all orthogonal matrices Q ∈ O ( k ) . There are many orthogonally equivariant neural architectures that are specialized to the special case of k = 3 , which is very useful for applications in the physical sciences [Thomas et al., 2018, Fuchs et al., 2020]. Here we consider models that directly work for general dimension k .

Frame averaging approaches [Puny et al., 2022, Atzmon et al., 2022] require 2 k forward passes of a base network f θ , one for each sign flip of the principal components. Letting their base network be a permutation equivariant DeepSets [Zaheer et al., 2017], this means that they require O ( n ( kd + d 2 )2 k ) time to evaluate their model, where d is the hidden dimension of the base model. Note that this has an extra exponential 2 k factor compared to our O ( n ( kd + d 2 )) cost.

Another general approach with universality guarantees comes from Villar et al. [2021], who analyze invariant polynomials to develop equivariant architectures. However, their method for O ( k ) invariance or equivariance requires forming XX ⊤ , an n × n matrix. Thus, the complexity is at least O ( n 2 ) , which is a problem in applications, since oftentimes n is much larger than k . Variants of their method do not need to compute all O ( n 2 ) inner products, but it is unclear how to maintain permutation equivariance when doing this.

A.2.2 Efficient Sign Invariant Networks

Consider again the form of SignNet [Lim et al., 2023], f ( V ) = ρ ([ ϕ ( v i ) + ϕ ( -v i )] i =1 ,...,k ) . In the permutation equivariant version, e.g. when ϕ is a DeepSets [Zaheer et al., 2017] or a message passing neural network [Gilmer et al., 2017], ϕ maps from R n → R n × d , where d is the hidden dimension. Thus, computing ϕ ( v i ) + ϕ ( -v i ) for all k vectors v i require an O ( nkd ) sized tensor to be formed (even if the output space of ϕ is R n , a vectorized implementation computes all ϕ ( v i ) + ϕ ( -v i ) in two batched inference calls to ϕ , which would require O ( nkd ) sized intermediate tensors). This is

a multiplicative factor larger than the sign equivariant requirement of O ( n ( k + d )) sized tensors. Moreover, it would take O ( nkd 2 ) time to compute ϕ ( v i )+ ϕ ( -v i ) for each i , which is a multiplicative factor larger than the O ( n ( kd + d 2 )) time for the sign equivariant architecture.

A.3 Potential Societal Impacts

We do not foresee direct societal impacts from our work. This project is primarily theoretical and aims to improve models for two general application areas: multi-node representation learning and orthogonal equivariant models. Potential societal impacts may arise in downstream applications that may be affected by general progress in geometric machine learning, such as social network analysis and recommender systems. These two applications are known to have negative societal impacts in certain circumstances, so care must be taken in future related work to avoid major negative consequences.

A.4 Edge Representations and Link Prediction

A.4.1 Sign Invariant Link Prediction Decoders

Here, we present an ansatz for universal permutation invariant and sign invariant functions for n = 2 , that is f : R 2 × k → R d out . Note that SignNet is only known to be universal for such functions for n = 1 , where there are no permutation symmetries [Lim et al., 2023].

We will parameterize such functions as

Here, rev : R 2 → R 2 reverses the vector, so rev( a ) 1 = a 2 and rev( a ) 2 = a 1 . Moreover, φ : R 2 × 2 k → R d out is a permutation invariant neural network, so φ ( PX ) = φ ( X ) for all 2 × 2 permutation matrices P . Note that it is easy to parameterize permutation invariant functions φ in a maximally expressive way, e.g. via DeepSets [Zaheer et al., 2017]. Now, we show that this parameterization is universal:

Proposition 4. Functions f : R 2 × k → R d out of the above form are permutation invariant and sign invariant, and they universally approximate permutation invariant and sign invariant functions.

Proof. Invariance of f is easy to see; let P be a 2 × 2 permutation matrix and s i ∈ {-1 , 1 } for each i . Then

where the second to last inequality is by permutation invariance of φ . Next, we show universal approximation.

Let h : R 2 × k → R d out be a continuous permutation invariant and sign invariant function. Then by the decomposition theorem in Lim et al. [2023], we can write

for continuous functions ρ and ϕ . As a composition of continuous functions, the function ψ : B ⊆ R 2 × 2 k → R d out given by ψ ( A 1 , . . . , A k ) = ρ ( ϕ ( A 1 ) , . . . , ϕ ( A k )) is continuous, where B is the subset of R 2 × 2 k consisting of ( v 1 v ⊤ 1 , . . . , v k v ⊤ k ) such that each v i ∈ R 2 . Note that ψ is permutation invariant on B , in the sense that for any 2 × 2 permutation matrix P , we have

because if v i v ⊤ i = A i , then

by permutation invariance of h .

Now, we define our permutation invariant function φ : C ⊆ R 2 × 2 k → R d out , on the domain

We define φ by

To see that φ is permutation invariant, we need only consider the case where P = [ 0 1 1 0 ] , in which case

where in the second equality, we use the fact that A 2 , 2 j = A 1 , 2 j , j = 1 , . . . , k for A ∈ C , because A 2 , 2 j = ( v j ⊙ rev( v j )) 2 = ( v j ⊙ rev( v j )) 1 = A 1 , 2 j for some v j ∈ R 2 . Moreover, φ is clearly continuous and sign invariant. Defining f : R 2 × k → R d out using this φ , we compute that

so we are done.

If φ instead comes from a universally approximating class of permutation invariant neural networks (rather than being an arbitrary continuous permutation invariant function), then on a compact domain we can get ϵ approximation of f to h by letting φ approximate ψ to ϵ accuracy.

A.4.2 Proof of Proposition 1

Proposition 1. Let f : R n × k → R n × d out be a permutation equivariant function, and let V = [ v 1 , . . . , v k ] ∈ R n × k be k orthonormal eigenvectors of an adjacency matrix A . Let nodes i and j be automorphic, and let z i and z j ∈ R d out be their embeddings, i.e, the i th and j th row of Z = f ( V ) .

If f is sign invariant and the eigenvalues associated with the v l are simple and distinct, then z i = z j .

If f is basis invariant and v 1 , . . . , v k are a basis for some number of eigenspaces of A then z i = z j .

Proof. We only prove the basis invariance claim, as the sign invariance claim is a special case; basis invariance is sign invariance when eigenvalues are distinct.

Let P ∈ R n × n be a permutation matrix associated to an automorphism that maps node i to node j , so PAP ⊤ = A and Pe i = e j , where e l is the l th standard basis vector. Let V t = [ v r 1 , . . . , v r d t ] be the matrix whose columns are the eigenvectors v r l that are associated to eigenvalue λ i . The columns of V t are thus an orthonormal basis for the eigenspace associated to λ t . Note that for any of these eigenvectors, we have

so Pv r l is also an eigenvector of A with eigenvalue λ t . As P is orthogonal, note that Pv r 1 , . . . , P v r d t is still an orthonormal basis of the eigenspace. Thus, there exists an orthogonal matrix Q t ∈ R d t × d t such that PV t = V t Q t -see Lim et al. [2023].

Repeat the above argument to get such a Q t for each of the eigenbases V 1 , . . . , V l . We can then see that

So we are done.

A.5 Sign Invariance and Structural Node or Node-Pair Encodings

In this section, we show that when the eigenvalues λ 1 , . . . , λ k are distinct, then sign invariant functions of the orthonormal eigenvectors v 1 , . . . , v k give structural node or node-pair representations. This can also be generalized in a straightforward way to larger tuples of nodes beyond pairs, though we only consider nodes and node-pairs for ease of exposition. First, we give formal definitions.

Definition 1 (Structural Representations [Srinivasan and Ribeiro, 2019]) . Let A ∈ R n × n be the adjacency matrix of a graph on node set { 1 , . . . , n } .

A function f : R n × n → R n is a node structural representation if f ( PAP ⊤ ) = Pf ( A ) for all n × n permutation matrices P .

A function f : R n × n → R n × n is a node-pair structural representation if f ( PAP ⊤ ) = Pf ( A ) P ⊤ for all n × n permutation matrices P .

Importantly, these structural representations are permutation equivariant functions of adjacency matrices, not arbitrary matrices. For each adjacency matrix A , let V ( A ) = [ v 1 ( A ) , . . . , v k ( A )] be a choice of orthonormal eigenvectors for the first k eigenvalues λ 1 ( A ) , . . . , λ k ( A ) . We assume in this section that these first k eigenvalues are distinct for all A under consideration, so V ( A ) is defined up to sign flips. Let h : R n × k → R n be a permutation equivariant function of sets, so h ( PX ) = Ph ( X ) for all permutations matrices P . Then of course h ( PV ( A )) = Ph ( V ( A )) , but this does not make h a node structural encoding. This is because A ↦→ h ( V ( A )) is in general not a well-defined function of the adjacency, since the choice of V ( A ) is not well-defined (the choices of sign are arbitrary). If we constrain h to not depend on the signs (sign invariance), or to depend on the signs in a predictable way (sign equivariance), then we can compute structural node or node-pair encodings from eigenvectors.

We capture these observations in the below proposition. First, we define three types of functions:

Let f node : R n × k → R n be sign invariant and permutation equivariant; that is, f node ( Pv 1 s 1 , . . . , P v k s k ) = Pf node ( v 1 , . . . , v k ) for s i ∈ {-1 , 1 } and P a permutation matrix.

Let f decode : R 2 × k → R be sign invariant; that is, f decode ( Sz i , Sz j ) = f decode ( z i , z j ) for S ∈ diag( {-1 , 1 } k ) .

Let f equiv : R n × k → R n × k be a permutation equivariant and sign equivariant function; that is, f equiv ( PV ( A ) S ) = Pf equiv ( V ( A )) S for S ∈ diag( {-1 , 1 } k ) and P a permutation matrix.

Proposition 5. Let A ⊆ R n × n denote the matrices with distinct firstk eigenvalues. For A ∈ A , let V ( A ) = [ v 1 ( A ) , . . . , v k ( A )] be a choice of orthonormal eigenvectors of A , associated to the firstk (distinct) eigenvalues λ 1 ( A ) , . . . , λ k ( A ) . Then

(a) The map q node : A → R n given by q node ( A ) i = f node ( f equiv ( V ( A ))) i is well-defined and gives a structural node representation.

(b) The map q pair : A → R n × n defined by q pair ( A ) i,j = f decode ( f equiv ( V ( A )) i, : , f equiv ( V ( A )) j, : ) is well-defined and gives a structural node-pair representation.

Note that the identity mapping V ( A ) ↦→ V ( A ) is permutation equivariant and sign equivariant, so using f node or f decode directly on eigenvectors also gives structural representations. The statement

(b) means that our link prediction pipeline with sign equivariant node features and sign invariant decoding produces structural node-pair representations.

Proof. Part (a) We first show that q node : A → R n is well-defined. Suppose we had another choice of eigenvectors, so the eigenvectors we input are V ( A ) S for some S ∈ diag( {-1 , 1 } k ) . Then

where the first equality is by sign equivariance, and the second equality by sign invariance. Thus, the value of q node ( A ) is unchanged.

Now, let P be any permutation matrix. Then for each eigenvector v i ( A ) , i ∈ [ k ] , we have ( PAP ⊤ ) Pv i ( A ) = PAv i ( A ) = λ i ( A ) Pv i ( A ) , so Pv i ( A ) is an eigenvector of PAP ⊤ associated to λ i ( A ) = λ i ( PAP ⊤ ) . Hence, we denote v i ( PAP ⊤ ) = Pv i ( A ) (the choice of sign does not matter as q does not depend on the sign. Now, we have that

where the second to last equality is by permutation equivariance of f node and f equiv .

Part (b) That q pair : A → R n × n is well-defined follows from a similar argument to the q node case. Let P be a permutation matrix, and σ : [ n ] → [ n ] its underlying permutation. We compute that

A.5.1 Sign Equivariance is Provably More Expressive for Link Prediction

Our arguments in Section 2.1 and Figure 2 explain why we can expect sign equivariant models to be more powerful than sign invariant models in link prediction. To give a theoretically rigorous explanation, here we provide an example where sign equivariant models can provably compute more expressive link representations than sign invariant models.

Consider a cycle graph C 2 k for some even length 2 k , where k ≥ 3 . All nodes are automorphic in this graph, so any model based on structural node representations must assign the same representation to each node-pair. For instance, consider the eigenvalue -2 of the adjacency matrix, which is a simple eigenvalue with eigenvector [1 , -1 , 1 , -1 , . . . , 1 , -1] [Lee et al., 1992]. Then a sign invariant model will lose the sign information and map each node to the same encoding, which means that each node-pair will also have the same encoding. However, a sign equivariant model can preserve the sign of each node (for instance by learning the identity function). Then for any pair of nodes that are one hop away, it can take a dot product to compute the pair representation -1 , whereas it can take a dot product between any nodes that are two hops away to compute the pair representation 1 . Of course, using more eigenvectors would allow for more complex representations to be computed.

A.5.2 More on Sign Equivariance and Link Prediction

Key to our method is the ability to update a positional node embedding in an equivariant way, which respects the graph symmetries. To elaborate, consider the aforementioned definition of node positional encodings as samples from a permutation equivariant probability distribution over node features [Srinivasan and Ribeiro, 2019]. Laplacian eigenvector positional embeddings are samples from the distribution of orthonormal bases of the eigenspaces of the Laplacian. Our sign equivariance

based approach is possible because the randomness in Laplacian eigenvector positional encodings is exceptionally structured (consisting only of sign flips when eigenvalues are distinct). In contrast, a general way to obtain structural pair representations from node positional embeddings is to average some function over the randomness of the positional encoding (i.e., over many samples of the positional encoding) [Srinivasan and Ribeiro, 2019], but this is highly expensive, often intractable, and introduces substantial variance into the learning procedure. For instance, one may have to average samples of the n ! assignments of unique node identifiers [Murphy et al., 2019] or approximate an integral over Gaussian random features [Abboud et al., 2021].

A.6 Proof of Proposition 2, Orthogonal Equivariance via Sign Equivariance

Proposition 2. Consider a domain X ⊆ R n × d such that each X ∈ X has distinct covariance eigenvalues, and let R X be a choice of orthonormal eigenvectors of cov( X ) for each X ∈ X . If h : X ⊆ R n × d → R n × d is sign equivariant, and if f ( X ) = h ( XR X ) R ⊤ X , then f is well defined and orthogonally equivariant.

Moreover, is h is from a universal class of sign equivariant functions, then the f of the above form universally approximate O ( k ) equivariant functions on X .

Proof. First, we show that f is well defined. R X is only unique up to sign flips, as R X S is an orthonormal set of eigenvectors of cov( X ) for S ∈ diag( {-1 , 1 } k ) . However, no matter the choice of signs, f ( X ) takes the same value, since

Next, we show that f is O ( k ) equivariant. Let Q ∈ O ( k ) be any orthogonal matrix. Note that

Thus, Q ⊤ R X is an orthonormal set of eigenvectors of cov( XQ ) . This means that there is a choice of signs S ∈ diag( {-1 , 1 } k ) such that Q ⊤ R X S = R XQ . Hence, we have that

so f is O ( k ) equivariant.

Universal Approximation. Our proof of the universality of this class of functions builds on the proof of the universality of frame averaging [Puny et al., 2022]. Let f target be a continuous O ( k ) equivariant function and let ϵ > 0 be a desired approximation accuracy. Then f target is also sign equivariant (as the sign matrices S ∈ diag( {-1 , 1 } k ) are orthogonal).

Hence, by sign equivariant universality, we can choose a sign equivariant h such that ∥ h ( X ) -f target ( X ) ∥ < ϵ for all X ∈ X (where ∥·∥ is the Frobenius norm). Define the O ( k ) equivariant f ( X ) = h ( XR X ) R ⊤ X . Then for all X ∈ X we have that

So f approximates f target within ϵ accuracy on X , and we are done.

B Sign Equivariant Linear Maps

B.1 Sign Equivariant Linear Map Characterization

We first prove our result characterizing the form of the equivariant linear maps from R n × k → R n ′ × k

Lemma 1. A linear map W : R → R is sign equivariant if and only if it can be written as

. n × k n ′ × k

for some linear maps W 1 , . . . , W k : R n → R n ′ , where X i ∈ R n is the i th column of X ∈ R n × k .

Proof. For one direction, suppose W can be written as in equation 4. To see that W is sign equivariant, note that for any S ∈ diag( {-1 , 1 } k ) , we have

For the other direction, let W be a sign equivariant linear map. For any i ′ ∈ [ n ′ ] and j ′ ∈ [ k ] , we can write the action of W as

̸

where W i,j i ′ ,j ′ ∈ R are coefficients representing the linear map. Let c = j ′ be a column that is not j ′ . Further, for any row l ∈ [ n ] , let ˜ X ∈ R n × k be such that ˜ X l,c = 1 , and ˜ X is zero elsewhere. Then we have that

Now, let S ∈ diag( {-1 , 1 } k ) have a -1 in the j ′ th column and a 1 elsewhere. Then ˜ XS = ˜ X . This implies that

where in the second to last equality we used sign equivariance. This implies that W l,c i ′ ,j ′ = 0 .

Hence, for any i ′ ∈ [ n ′ ] , j ′ ∈ [ k ′ ] , we have that W ( X ) i ′ ,j ′ only depends on X j ′ , so we are done.

B.2 Sign Equivariant Linear Maps between Tensor Representations

Since the sign equivariant linear maps from R n × k → R n ′ × k are very weak, we now characterize sign equivariant linear maps between higher order tensor representations, as past work has done for other groups [Maron et al., 2018, 2019, Finzi et al., 2021]. In particular, we will consider representations R k m for natural numbers m . The action of s ∈ {-1 , 1 } k on V ∈ R k m is as follows:

for i 1 , . . . , i m ∈ [ k ] . We now prove a result showing that there are no sign equivariant linear maps between many pairs of tensor representations.

Proposition 6. If m 1 + m 2 is odd, then the only sign equivariant linear map L : R k m 1 → R k m 2 is the zero map.

Proof. Let L : R k m 2 × k m 1 be the matrix associated with a sign equivariant linear map from R k m 1 → R k m 2 . This means that for V ∈ R k m 1 , for i 1 , . . . , i m 2 ∈ [ k ] we have that

Then by sign equivariance, we have that

for s ∈ {-1 , 1 } k , which means that for all i 1 , . . . , i m 2 ∈ [ k ] ,

which means that

̸

This gives that s · L = L . Choose some arbitrary indices i 1 , . . . , i m 2 , j 1 , . . . , j m 1 ∈ [ k ] . Since m 1 + m 2 is odd, then one of these values appears an odd number of times - say it is some index p ∈ [ k ] . Let s ∈ {-1 , 1 } k have s i = 1 for i = p and s p = -1 . Then s · L = L implies that

Which implies that L is zero at these indices, and hence L = 0 everywhere, since these were arbitrarily chosen indices.

This implies that we cannot map from R k → R k 2 using sign equivariant linear maps. Thus, if we want to lift our order-one tensor input to higher tensor orders in a linear way, then we need to at least map to third order tensors in R k 3 . This requires substantial memory cost. Furthermore, we next show that even if we could map to third order tensors, learning representations of third order tensors with sign equivariant linear maps is expensive.

Proposition 7. The dimension of the space of sign equivariant linear maps from R k m 1 → R k m 2 is

Proof. This is a direct consequence of the First Projection Formula: see Fulton and Harris [2013] Section 2.2. In particular, it can be seen that the dimension of this space is equal to

by the First Projection Formula. Since trace( A ⊗ B ) = trace( A ) · trace( B ) , this is equal to

Using Proposition 7, we have numerically computed the dimension of the space of sign equivariant linear maps from R k 3 → R k 3 ; see Table 4. The dimension appears to be 15 k 3 -30 k 2 +16 k for k up to 20. In particular, when k = 8 , we compute that the space of sign equivariant linear maps from R k 3 → R k 3 is of dimension 5888, which is already quite large.

C Characterization of Sign Equivariant Polynomials

In this Appendix, we characterize the form of the sign equivariant polynomials. This is useful, because for a finite group, equivariant polynomials universally approximate equivariant continuous functions [Yarotsky, 2022]; thus, if a model universally approximates equivariant polynomials, then it universally approximates equivariant continuous functions. Using equivariant polynomials to analyze or develop equivariant machine learning models has been done successfully in many contexts [Zaheer et al., 2017, Yarotsky, 2022, Segol and Lipman, 2019, Dym and Maron, 2021, Maron et al., 2019, 2020, Villar et al., 2021, Dym and Gortler, 2022, Puny et al., 2023].

C.1 Sign Invariant Polynomials R k → R

Next, we characterize the form of sign invariant and equivariant polynomials. For simplicity, we start with the case of sign invariant polynomials p : R k → R . The sign equivariant polynomials take a very similar form. We can write any polynomial from R k to R in the form

for some coefficients W d 1 ,...,d k ∈ R and some D ∈ N . Sign invariance tells us that for any S = diag( s 1 , . . . , s k ) ∈ diag( {-1 , 1 } k ) , we must have

This holds for any v ∈ R k , so for all choices of d 1 , . . . , d k we must have

̸

Note that s d i i = 1 if d i is an even number. Hence, there are no constraints on W d 1 ,...,d k if all d i are even. On the other hand, suppose d j is odd for some j . Let s i = 1 for i = j and s j = -1 . Then the

constraint says that W d 1 ,...,d k = -W d 1 ,...,d k , so we must have W d 1 ,...,d k = 0 . To summarize, we have

Where being free means that the coefficient may take any value in R . Thus, any sign invariant p only has terms where each variable v i is raised to an even power. It is also easy to see that any polynomial p where each variable v i is raised to only even powers is sign invariant, so we have the following proposition:

Proposition 8. A polynomial p : R k → R is sign invariant if and only if it can be written

for some coefficients W d 1 ,...,d k ∈ R and D ∈ N .

In other words, p is sign invariant if and only if there exists a polynomial q : R k → R such that p ( v ) = q ( v 2 1 , . . . , v 2 k ) .

C.2 Sign Equivariant Polynomials R k → R k

The case of sign equivariant polynomials p : R k → R k is very similar. For l ∈ [ k ] , the l th output dimension of any polynomial p : R k → R k can be written

where W ( l ) d 1 ,...,d k ∈ R are coefficients (note the extra l index, so there are k times more coefficients than in the invariant case). By sign equivariance, we have

As this holds for all inputs v ∈ R k , we have the following constraints on the coefficients:

̸

where we use the fact that s l = 1 /s l since s l ∈ {-1 , 1 } . If d j is odd for j = l , then similarly to the invariant case, we can take s i = 1 for i = j and s j = -1 in the above equation to see that W ( l ) d 1 ,...,d k = 0 . If d l is even, then d l +1 is odd, so we have that W ( l ) d 1 ,...,d k = 0 by the same argument. Thus, we must have

̸

̸

Thus, the l th entry p ( v ) l only contains monomials of the term v 2 d 1 1 · · · v 2 d l +1 l · · · v 2 d k k , where each term besides v l is raised to an even power. We can factor out a v l and write such terms as v l · v 2 d 1 1 · · · v 2 d k k . It is also easy to see that any polynomial with monomials only of this form is sign equivariant. Thus, we have proven Proposition 9.

Proposition 9. A polynomial p : R k → R k is sign equivariant if and only if it can be written

In vector format, p is sign equivariant if and only if it can be written as p ( v ) = v ⊙ p inv ( v ) for a sign invariant polynomial p inv : R k → R k .

C.3 Sign Equivariant Polynomials R n × k → R n ′ × k

Finally, we will handle the case of polynomials p : R n × k → R n ′ × k equivariant to diag( {-1 , 1 } k ) . This is the case we most often deal with in practice, when we have input V = [ v 1 . . . v k ] for k eigenvectors v i ∈ R n of some n × n matrix. For a ∈ [ n ′ ] and b ∈ [ k ] , the ( a, b ) th output of a polynomial R n × k → R n ′ × k is

where the sum ranges over d i,j ∈ { 0 , . . . , D } for i ∈ [ n ] and j ∈ [ k ] , and d = ( d 1 , 1 , . . . , d n, 1 , d 1 , 2 , . . . , d n,k ) is a shorthand to index coefficients W ( a,b ) d ∈ R . By sign equivariance, we have that:

where ˜ d j = ∑ n i ′ =1 d i ′ ,j is the number of times that an entry from column j of V appears in the product ∏ n i =1 ∏ k j =1 V d i,j i,j . As this holds over all V , we thus have that

̸

By analogous arguments to the previous subsections, if ˜ d j is odd for j = b , we have that the W ( a,b ) d = 0 . Likewise, if ˜ d b is even, we have W ( a,b ) d = 0 . Thus, the constraint on W is

̸

̸

In particular, this means that the only nonzero terms in the sum that defines p ( V ) a,b have an even number of entries from column j for j = b , and an odd number of entries from column b . Thus, each term can be written as V i d ,b · p inv ( V ) d for some index i d ∈ [ n ] and sign invariant polynomial p inv . Moreover, it can be seen that any polynomial that only has terms of this form is sign equivariant. Thus, we have shown the following proposition:

Proposition 10. A polynomial p : R n × k → R n ′ × k is sign equivariant if and only if it can be written as

where p inv is a sign invariant polynomial, the sum ranges over all d , and i d ∈ [ n ] for each d .

Now, we show that this implies Theorem 1. In particular, we will write p in the form

for sign equivariant linear maps W (2) and W (1) , and a sign equivariant polynomial q inv . To do so, let ˜ D denote the number of all possible d that the sum in equation 90 ranges over. We take W (1) : R n × ˜ k → R ˜ Dn ′ × k and W (2) : R ˜ Dn ′ × k → R n ′ × k . These sign equivariant linear maps have to act independently on each column of their input, so W (1) V = [ W (1) 1 v 1 , . . . W (1) k v k ] for linear maps W (1) i : R n → R ˜ Dn ′ . We define W (1) b to be the linear map such that ( W (1) b v b ) d ,a = W ( a,b ) d V i d ,b for a ∈ [ n ′ ] . For the sign invariant polynomial q inv , we take q inv ( V ) d ,a = p inv ( V ) d .

Finally, we define W (2) to compute the sum in equation 90. In particular, for X = [ x 1 , . . . , x k ] ∈ R ˜ Dn ′ × k we write W (2) X = [ W (2) 1 x 1 , . . . , W (2) k x k ] , where ( W (2) b x b ) a = ∑ d x i d ,b . It can be seen that with these definitions of W (2) , W (1) , and q inv , we have written p in the desired form.

C.4 Sign Invariant Polynomials and SignNet

For completeness, here we state the form of the sign invariant polynomials p : R n × k → R on inputs V = [ v 1 , . . . , v k ] ∈ R n × k . The derivation very closely follows that of the sign equivariant polynomials from R n × k → R n ′ × k in Appendix C.3, so we omit this derivation.

Proposition 11. A polynomial p : R n × k → R is sign invariant if and only if it can be written

where W d = 0 for d = ( d 1 , 1 , . . . , d n, 1 , d 1 , 2 , . . . , d n,k ) only if ∑ n i =1 d i,j is even for each column j ∈ [ k ] .

̸

In particular, p is sign invariant if and only if there is a polynomial q : R n × n × k → R such that p ( V ) = q ([ V i 1 ,j · V i 2 ,j ] i 1 ∈ [ n ] ,i 2 ∈ [ n ] ,j ∈ [ k ] ) .

The polynomials V ↦→ V i 1 ,j · V i 2 ,j for i 1 , i 2 ∈ [ n ] and j ∈ [ k ] are thus generators of the ring of sign invariant polynomials from R n × k → R .

Notably, Lim et al. [2023] propose universal sign invariant neural architectures, but do not characterize or otherwise use the sign invariant polynomials. Instead, their proof of universality uses topological constructions and shows that all sign invariant continuous functions can be decomposed in a simple form-namely, ρ ([ ϕ ( v i ) + ϕ ( -v i )] i =1 ,...,k ) for continuous functions ρ and ϕ . Our characterization of sign invariant polynomials provides another path to developing and analyzing the expressive power of sign invariant architectures.

In particular, we can give an alternative proof for the universality of SignNet.

Proposition 12 (Universality of SignNet) . Let f : X ⊆ R n × k → R be a continuous sign invariant function on a compact domain X , and let ϵ > 0 . Then there exists a continuous ρ : R n 2 k → R and continuous ϕ : R n → R n 2 such that | f ( V ) -ρ ([ ϕ ( v i ) + ϕ ( -v i )] i =1 ,...,k ) | < ϵ for all V ∈ X .

Proof. First, let p be a sign invariant polynomial that approximates f to within ϵ on X . Then using Proposition 11, let q be a polynomial such that p ( V ) = q ([ V i 1 ,j · V i 2 ,j ] i 1 ∈ [ n ] ,i 2 ∈ [ n ] ,j ∈ [ k ] ) .

Define ϕ : R n → R n 2 to map a v ∈ R n to the vector of pairwise products of elements in v scaled by 1/2, that is

Then ϕ ( v ) + ϕ ( -v ) is equal to the vector of pairwise products of v . Finally, we let ρ = q , which gives that

and hence for all V ∈ X .

Given the form of the sign invariant polynomials, this proof is quite simple. However, it is technically weaker than the result of Lim et al. [2023], as they invoke the Strong Whitney Embedding Theorem and only require ϕ to map to R 2 n instead of R n 2 . Still, further arguments could probably reduce the dimension required to about 2 n in this polynomial-based proof; as the Gram matrix vv ⊤ is rank one, it can be recovered almost always from about 2 n of its entries [Pimentel-Alarcón et al., 2016].

D Sign Equivariant Architecture Universality

In this section, we prove Proposition 3 on the universality of our proposed sign equivariant architectures, which we restate here:

Proposition 3. Functions of the form v ↦→ v ⊙ MLP( | v | ) universally approximate continuous sign equivariant functions f : R k → R k .

Compositions f 2 ◦ f 1 of functions f l as in equation 6 universally approximate continuous sign equivariant functions f : R n × k → R n ′ × k .

We prove the two statements of the proposition in the next two subsections.

D.1 Universality for functions R k → R k

Proof. Let X ⊆ R k be a compact set, let ϵ > 0 , and let f target : X → R k be a continuous sign equivariant function that we wish to approximate within ϵ . Choose a sign equivariant polynomial p that approximates f target to within ϵ/ 2 on X . By compactness, we can choose a finite bound B > 0 such that | v i | < B for all v ∈ X .

By Proposition 9, we can write p ( v ) l = v l · ∑ D d 1 ,...,d k =0 W d 1 ,...,d k v 2 d 1 1 · · · v 2 d k k . By the universal approximation theorem for multilayer perceptrons, we can choose a MLP : X → R k such that approximates q ( v ) = ∑ D d 1 ,...,d k =0 W d 1 ,...,d k v 2 d 1 1 · · · v 2 d k k up to ϵ/ (2 B ) . Note that q ( | v | ) = q ( v ) , so v ↦→ MLP( | v | ) also approximates q within ϵ/ (2 B ) accuracy.

Thus, for all v ∈ X , we have that

so ∥ f -p ∥ ∞ < ϵ/ 2 on X and we are done by the triangle inequality.

D.2 Universality for functions R n × k → R n ′ × k

Recall that each layer of our sign equivariant network from R n × k → R n ′ × k takes the form

Proof. Let X ⊆ R n × k be compact, and let f target : X → R n ′ × k be a continuous sign equivariant function that we wish to approximate. Since X is compact, we can choose a finite bound B > 0 such that | V ij | < B for all V ∈ X . Let p : X ⊆ R n × k → R n ′ × k be a sign equivariant polynomial that approximates f target up to ϵ/ 2 accuracy. Using Proposition 10, we can write

for some sign invariant polynomials p inv ( V ) d . We will have one network layer f 1 approximate the summands, and have the second network layer f 2 compute the sum.

First, we absorb the coefficients W ( a,b ) d into the sign invariant part, by defining the sign invariant polynomial q inv ( V ) d ,a,b = W ( a,b ) d p inv ( V ) d , so we can write

Now, let d hidden ∈ N denote the number of all possible d that appear in the sum, multiplied by n ′ . We define f 1 : X → R d hidden × k as follows. As SignNet [Lim et al., 2023] universally approximates

sign invariant functions on compact sets, we can let SignNet 1 : X → R d hidden × k be a SignNet that approximates q inv ( V ) up to ϵ/ (2 B ) accuracy, so

For b ∈ [ k ] , we also define the weight matrices W (1) b ∈ R d hidden × n of the layer by letting the ( d , a ) th row ( W (1) b ) ( d ,a ) , : for any a ∈ [ n ] only be nonzero in the i d th index, where it is equal to 1. Thus,

Hence, the first layer takes the form

Now, for the second layer, we let SignNet( V ) i,j = 1 for all i ∈ [ n ] , j ∈ [ k ] , which can be represented exactly. Then for each column b ∈ [ k ] we will define weight matrices W (2) b such that ( W (2) b ) a, ( d ,i ) = 1 if a = i and is 0 otherwise. Then we can see that

To see that this approximates the polynomial p , for any V ∈ X we can bound

By the triangle inequality, f 2 ◦ f 1 is ϵ -close to f target , so we are done.

E Experimental Details

E.1 Miscellaneous Experimental Details

We ran the experiments on a HPC server with CPUs and GPUs. Each experiment was run on a single NVIDIA V100 GPU with 32GB memory. The runtimes for some of our experiments are included in the main paper. Our codes for our models and experiments will be open-sourced and permissively licensed.

E.2 Link Prediction in Nearly Synthetic Graphs

The base graphs H we generate are Erdös-Renyi or Barabási-Albert graphs with 1000 nodes. We use NetworkX [Hagberg et al., 2008] to generate and process the graphs. The Erdös-Renyi graphs have edge probability p = . 05 and the Barabási-Albert graphs have m = 20 new edges per new node. Let V = [ v 1 , . . . , v k ] be Laplacian eigenvectors of the graph. We take k = 16 in these experiments. The unlearned decoder baseline simply takes the predicted probability of a link between i and j to be proportional to the dot product of the eigenvectors embeddings of node i and node j ; this has no learnable parameters. In other words, the node embeddings z i and z j are taken to be V i, : and V j, : respectively, and the edge prediction is z ⊤ i z j . The learned decoder baseline takes the same z i and z j ,

but takes the edge prediction to be MLP( z i ⊙ z j ) . Every other method learns node embeddings z i and z j , and takes the edge prediction to be z ⊤ i z j .

Each model is restricted to around 25,000 learnable parameters (besides the Unlearned Decoder, which has no parameters). We train each method for 100 epochs with an Adam optimizer [Kingma and Ba, 2015] at a learning rate of .01. The train/validation/test split is 80%/10%/10%, and is chosen uniformly at random.

E.3 Details on n-body Simulations

We follow the experimental setting and build on the code of Puny et al. [2022] (no license as far as we can tell) for the n-body learning task. The code for generating the data stems from Kipf et al. [2018] (MIT License) and Fuchs et al. [2020] (MIT License). There are 3000 training trajectories, 2000 validation trajectories, and 2000 test trajectories. We modify the data generation code to apply to general dimensions d > 3 . We do not change any of the scaling factors in doing so. For each dimension d , we use the same hyperparameters for both the frame averaging model and the sign equivariant model.

In Table

E.4 Node Classification on CLUSTER

In Section 4.3, we show results for the node classification task CLUSTER [Dwivedi et al., 2022a], where the task is to cluster nodes in graphs drawn from Stochastic Block Models [Abbe, 2017]. Models are restricted to a 100k learnable parameter budget. We largely follow the experimental setting of Rampasek et al. [2022], except we report results for the eigenvector based methods on 5 runs instead of 10.

We test several eigenvector based methods within the GraphGPS framework and codebase [Rampasek et al., 2022] (MIT License), which is a state of the art Transformer / GNN hybrid. Firstly, we make use of the PEG style GraphGPS, which means that the MPNN in the l th GraphGPS layer takes as edge features e ( l ) ij = ∥ ∥ ∥ V ( l ) i -V ( l ) j ∥ ∥ ∥ 2 , where V ( l ) i ∈ R k is the eigenvector embedding of node i in layer l . This is fully O ( k ) invariant (which is much stricter than sign / basis invariance), so we relax this to just be sign invariant in our model by learning a diagonal matrix D ( l ) such that e ( l ) ij = V ( l ) ⊤ i D ( l ) V ( l ) j . Also, the standard GraphGPS only updates eigenvector representations (in a non-equivariant manner) before most of the neural network modules. When we add our sign equivariant model, we instead update eigenvector representations within each GraphGPS layer via V ( l ) = f ( l ) θ ( V ( l -1) ) for a sign equivariant f ( l ) θ : R n × k → R n × k .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a novel sign-equivariant model that is equivariant to the sign flip of eigenvectors for spectral graph learning. They demonstrated that the proposed architecture could capture the information when sign-invariant models would fail. The authors then made theoretical analyses of the sign-equivariant polynomial space and also proposed some constructible neural network architecture with sign-equivariance guaranteed by design. Experimental results on both the synthetic and real-world datasets demonstrate superior performance on link prediction, especially when automorphism exists.\n\nSTRENGTHS:\n1. The paper is well-organized and easy to follow. The authors first provided an example to demonstrate the limitation of the previous \"sign-invariant\" models and then made theoretical analyses. Experiments were followed to demonstrate superior performance over the baselines.\n\n2. The problem that the authors tried to address is novel. The authors pointed out the limitation of sign-invariant models and mathematically supported their claim in Proposition 1. The proposed \"sign-equivariant\" architecture can properly address the limitation, both theoretically and empirically.\n \n3. The mathematics are solid and the claims are well-supported. The definitions of sign-equivariance and -invariance are well-formulated. Sign-equivariance of polynomials and neural networks is proven in the appendices. Universality is also proven and guaranteed.\n\n4. Experiments on graphs with high symmetries demonstrated the effectiveness of the proposed model architecture. State-of-the-art performance was also achieved.\n\nWEAKNESSES:\n1. Proposition 1 lacks some detail (See Question 1 & 2).\n2. Limitation of the proposed architecture is not explicitly discussed.\n\nQUESTIONS:\n1. In Proposition 1, the first result claims that if $f$ is sign invariant and the eigenvalues are distinct, then the node-wise representations are the same for automorphic nodes. Consider the complete graph of 3 nodes, the adjacency matrix is\n$$\n\\begin{pmatrix}\n0&1&1\\\\\\\\\n1&0&1\\\\\\\\\n1&1&0\\\\\\\\\n\\end{pmatrix}\n$$\nWe have eigenvector $v_1=(1,1,1)$ belonging to $\\lambda=2$ and $v_2=(0,1,-1)$ belong to $\\lambda=-1$ (normalization is ignored for conciseness). Let $f$ be the pointwise absolute function, then $f$ is clearly sign-invariant (but not necessarily basis-invariant). In this graph, all three nodes are automorphic, but we have $z_1=(1,0)$ and $z_2=(1,1)$. This is probably because the eigenspace corresponding to $\\lambda=-1$ is 2-dimensional. Therefore, I believe basis-invariance is also a necessary condition. The first claim is also not directly proven in Appendix A.3.2. Please provide a more detailed justification.\n\n2. Similarly, the scenario when the dimension of the eigenvalue is greater than 1 is not explicitly discussed. In the above example, the sign-invariant model may still be capable of distinguishing the automorphic nodes. In footnote 2, the authors mentioned that sign invariant embeddings maintain \"some positional information\". The authors may provide a more accurate description using mathematical formulations similar to Proposition 1.\n\n3. What are the potential limitations of the proposed model architecture?\n\nLIMITATIONS:\nNo potential negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a sign-equivariant design for processing spectral features in geometric deep learning.\nThis is particularly useful to process the eigenvectors of graph Laplacians and generate graph positional encodings, especially for link prediction tasks.\nThe equivariant, rather than invariant, approach guarantees improved expressivity of the models, which is verified in a numberof experiments.\n\nSTRENGTHS:\nThe paper is clearly written and well motivated.\nI liked this simple yet effective design and I think the proposed method could be useful for many future works in the field.\n\nWEAKNESSES:\nI don't see major flaws in the paper.\nSee \"Questions\" for other comments.\n\nQUESTIONS:\nI don't think the claim that the Geometric Deep Learning approach fails here is correct.\nThe issue you describe is due to a bad design choice of intermediate representation within the network (i.e. the same as the input and output representation).\nNote that this choice is not common in other equivariant networks either; for instance, in group-convolution NNs (GCNNs), the input representation is an image (this is a quotient representation of $SE(2)$) but intermediate features transform like a regular representation (the output of a group convolution is a function over the full group, not only the pixels).\nThis step is fundamental to achieve universal approximation.\nThe limited expressive power you mention in line 48 is only due to the choice of intermediate representations (similarly, in a GCNN, if the intermediate features are constrained to transform as images, equivariance requires isotropic filters, which are indeed less expressive).\n\nIn any case, while a group-convolution approach is probably a poor choice here (the group $(-1, 1)^k$ has size $2^k$), smaller representations which can provide a better trade off might exist.\nYour method seems to provide a nice solution to this, so I would argue it still fits within the GDL framework but provides a more efficient solution than its naive implementation.\n\n\nIsn't computing the eigenvectors of the graph Laplacian an expensive preprocessing step?\nEstimating these features has $O(n^3)$ complexity, which seems to exceed the $O(n)$ operations in the DSS layers (or $O(n^2)$ is an attention-based or message-passing based model).\nIn Appendix A.2 you discuss the computational complexity of your method but do not include this aspect.\nCould you comment on this?\n\nCould you report the table with all baselines and their performance in Section 4.2?\nAlso, is there any more recent work on this dataset you could include?\n\nLIMITATIONS:\nI did not see an explicit discussion of the limitations in the paper, which I think would be a nice addition.\nFor example, the authors could comment further on the complexity of computing the eigenvectors.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes construction and analysis of sign equivariant neural network architectures for processing eigenvectors while respecting their symmetries. While a similar approach has been proposed by a prior work (Lim et al., 2023), this work is motivated by the fact that sign invariance, pursued in prior work, is in general not sufficient for multi-node tasks as sign invariance leads to structural node representations that does not distinguish automorphic nodes while multi-node tasks require positional node representations that provides a form of node identification (Srinivasan et al., 2019). The authors provide a construction of sign equivariant neural network architecture with analysis and provable guarantee on expressive power, and also propose its application as an alternative to PCA-based frame averaging which requires only a single evaluation of base model (while frame averaging requires 2^k) while guaranteeing equivariance to orthogonal groups in arbitrary dimensions. The developed architectures are empirically demonstrated for link prediction on graphs, n-body dynamics prediction, and node clustering on SBM graphs, which support the main claims of the paper.\n\nLim et al., Sign and Basis Invariant Networks for Spectral Graph Representation Learning (2023)\n\nSrinivasan et al., On the equivalence between positional node embeddings and structural graph representations (2019)\n\nSTRENGTHS:\nS1. The paper tackles an important and practical problem of developing expressive neural architectures that respect symmetry of eigenvectors, and successfully complements and improves key prior work (Lim et al., 2023) by achieving sign equivariance. Also, the paper is overall well written and easy to read, and the provided illustrations are clear.\n\nS2. The motivation for sign equivariance over invariance only is clearly presented in Section 2.1 where multi-node tasks are considered, and I find the discussion sound and informative.\n\nS3. In addition to multi-node tasks, a novel (as far as I can tell) and powerful alternative based on sign equivariance to PCA based frame averaging for modeling O(k) equivariant functions is provided. The approach trades off generality, as the base model (h) should satisfy sign equivariance unlike frame averaging that works with arbitrary base functions, but more efficient as it eliminates the requirement of frame averaging for 2^k forward passes.\n\nS4. The proposed parameterization of sign equivariant polynomial is novel and theoretically sound as it is grounded to the theory of equivariant polynomials. A strong theoretical guarantee on universality is provided for the non permutation equivariant case as well.\n\nS5. Overall, the experimental results overall seem sound and support the main claims of the paper.\n\nLim et al., Sign and Basis Invariant Networks for Spectral Graph Representation Learning (2023)\n\nWEAKNESSES:\nW1. The description of how sign equivariant network is applied in Appendix D.4 was a bit unclear to me. In in the equation e_ij = V_i^TDV_j, does the V refer to the output of sign equivariant module within the given GraphGPS layer? Is V passed and updated across GraphGPS layers?\n\nQUESTIONS:\nI have no specific questions for now other than W1.\n\nLIMITATIONS:\nAs far as I can tell, the authors didn't address the limitations and potential societal impact of the work. I encourage the authors to discuss them in the next revision of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on addressing the sign ambiguity problem of eigenvectors. It argues that previous sign-invariant models are insufficient for some applications, e.g., link prediction and multi-node tasks. To solve this problem, the authors propose a sign-equivariant neural network with provable expressiveness guarantees, which is more powerful than the sign-invariant models and sign-equivariant linear maps. Experiments on various tasks validate the superiority of the proposed method.\n\nSTRENGTHS:\n1. This paper is the first to study the sign-equivariance problem of eigenvectors. It finds that the signs of eigenvectors contain meaningful positional information but the sign-invariant models can only preserve the structural information. Therefore, the sign-invariant models cannot distinguish automorphic nodes and are insufficient for link prediction or multi-node tasks.\n\n2. This paper proves that the sign equivariant polynomial can be implemented by the elementwise product between a linear sign-equivariant polynomial and a general sign-invariant polynomial. The linear sign-equivariant polynomial cannot learn the interactions between eigenvectors, and the sign-invariant polynomial cannot preserve positional information. This theoretical discovery combines the advantages of both methods and addresses their shortcoming.\n\n3. Based on the characterization of sign-equivariant polynomial functions, this paper provides a general framework for analyzing the sign-symmetry models and gives a new perspective on the universality of SignNet.\n\nWEAKNESSES:\nThe major weakness of this paper is that it lacks experiments on real-world data. This paper uses three tasks, e.g., link prediction, n-body problem, and node clustering, to validate the effectiveness of the proposed method. However, all the experiments are based on synthetic data, which makes the results unconvincing. I think there are many real-world datasets for both link prediction and node clustering tasks, and the authors should give more empirical results on at least these two tasks.\n\nQUESTIONS:\nThis paper considers both sign-equivariance and permutation-equivariance. I wonder why the authors do not discuss the basis-equivariance for eigenvectors.\n\nLIMITATIONS:\nThis paper should discuss the limitation of sign-equivariance from the perspectives of efficiency and scalability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to build sign equivariant neural networks with applications of the method in O(n) equivariant modeling and graph representation learning. They show that corresponding equivariant MLPs are inexpressive, by contrast to the proposed method which is universal (non-permutation equivariant version) or plausibly much more expressive (permutation equivariant version). Experiments are performed on artifcial datasets and show that the proposed method achieve comparable or better results than alternatives.\n\nSTRENGTHS:\nThis is a well written and motivated paper. I also appreciate the detailed appendix.\n\nThe experimental protocol seems rigorous and results are presented for a few different settings.\n\nThe paper highlights interesting and potentially useful applications of sign (O(1)) equivariance.\n\nWEAKNESSES:\n(W1) I think the main result of this paper could be obtained from the results of [Villar 2021]. They already show how to use invariants to builds equivariant functions. Sign equivariance is O(1) equivariance, so it is a particular case of O(n) equivariance. This connection is not mentioned by the authors and would be important to investigate. As such polynomials may not really be needed, the results from invariant theory could be sufficient. Also, Proposition 11 from the aforementioned work may offer a way to obtain the desired permutation equivariance with universality.\n\n(W2) More generally, given the previous comment, I feel like the paper should situate itself more within the literature on O(n) equvariance, since sign and basis equivariance are that. I appreciate that some of the applications considered included processing eigenvectors, which is different from most of what is done in that literature, but this was already put forward by [Lim 2023].\n\n(W3) The experiment on CLUSTERS is interesting but the improvement to other methods is not too important and has not convinced me of the practical usefulness of the method. Especially, since CLUSTERS is a synthetic task. It would also be great to have results on real-world graphs to see if the method brings a performance increase there.\n\n(W4) It seems like the proposal for orthogonal equivariance is a realization of the partial canonicalization framework of [Kaba 2023]. Partial canonicalization is performed with respect to SO(3), with the reminder handled by a sign equivariant function. Proposition 2 follows from their results. The results on the N-body problem seem worst compared to learned canonicalization, which should be included for comparison. There are also many other methods outside of those presented in [Puny 2022] that could be compared against.\n\n(W5) The paper [Sachs 1983] is relevant to the Proposition 1 presented in this paper and should be mentioned.\n\n(W6) I think the application on link prediction is the most promising aspect of the submission. The authors should compare with the setup/methods that was proposed by [Sartorras 2021] in their autoencoding experiment. In that paper, the method was used to solve a similar issue caused by automorphic nodes.\n\nQUESTIONS:\n(Q1) Can the authors provide a comparaison of their theoretical results with [Villar 2021] and discuss how their work compares?\n\n(Q2) Would you elaborate on some limitations of this work and potential future directions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}