SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| f (v) _ {i} - p (v) _ {i} \right| = \left| v _ {i} \cdot \operatorname {M L P} (\left| v \right|) _ {i} - v _ {i} \cdot \sum_ {d = 1} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} v _ {1} ^ {2 d _ {1}} \dots v _ {k} ^ {2 d _ {k}} \right| (96) \\ = | v _ {i} | | \operatorname {M L P} (| v |) _ {i} - \sum_ {d = 1} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} v _ {1} ^ {2 d _ {1}} \dots v _ {k} ^ {2 d _ {k}} | (97) \\ \leq B \cdot | \operatorname {M L P} (| v |) _ {i} - \sum_ {d = 1} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} v _ {1} ^ {2 d _ {1}} \dots v _ {k} ^ {2 d _ {k}} | (98) \\ <   \epsilon / 2, (99) \\ \end{array}
$$
so  $\| f - p \|_{\infty} < \epsilon / 2$  on  $\mathcal{X}$  and we are done by the triangle inequality.
D.2 Universality for functions  $\mathbb{R}^{n\times k}\to \mathbb{R}^{n'\times k}$
Recall that each layer of our sign equivariant network from  $\mathbb{R}^{n\times k}\to \mathbb{R}^{n'\times k}$  takes the form
$$
f _ {l} (V) = \left[ W _ {1} ^ {(l)} v _ {1}, \dots , W _ {k} ^ {(l)} v _ {k} \right] \odot \operatorname {S i g n N e t} _ {l} (V).
$$
Proof. Let  $\mathcal{X} \subseteq \mathbb{R}^{n \times k}$  be compact, and let  $f_{\mathrm{target}}: \mathcal{X} \to \mathbb{R}^{n' \times k}$  be a continuous sign equivariant function that we wish to approximate. Since  $\mathcal{X}$  is compact, we can choose a finite bound  $B > 0$  such that  $|V_{ij}| < B$  for all  $V \in \mathcal{X}$ . Let  $p: \mathcal{X} \subseteq \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  be a sign equivariant polynomial that approximates  $f_{\mathrm{target}}$  up to  $\epsilon / 2$  accuracy. Using Proposition 10, we can write
$$
p (V) _ {a, b} = \sum_ {d _ {i, j} = 0} ^ {D} \mathbf {W} _ {\mathbf {d}} ^ {(a, b)} V _ {i _ {\mathbf {d}}, b} \cdot p _ {\mathrm {i n v}} (V) _ {\mathbf {d}},
$$
for some sign invariant polynomials  $p_{\mathrm{inv}}(V)_{\mathbf{d}}$ . We will have one network layer  $f_{1}$  approximate the summands, and have the second network layer  $f_{2}$  compute the sum.
First, we absorb the coefficients  $\mathbf{W}_{\mathbf{d}}^{(a,b)}$  into the sign invariant part, by defining the sign invariant polynomial  $q_{\mathrm{inv}}(V)_{\mathbf{d},a,b} = \mathbf{W}_{\mathbf{d}}^{(a,b)}p_{\mathrm{inv}}(V)_{\mathbf{d}}$ , so we can write
$$
p (V) _ {a, b} = \sum_ {d _ {i, j} = 0} ^ {D} V _ {i _ {\mathbf {d}}, b} \cdot q _ {\mathrm {i n v}} (V) _ {\mathbf {d}, a, b}.
$$
Now, let  $d_{\mathrm{hidden}} \in \mathbb{N}$  denote the number of all possible  $\mathbf{d}$  that appear in the sum, multiplied by  $n'$ . We define  $f_1: \mathcal{X} \to \mathbb{R}^{d_{\mathrm{hidden}} \times k}$  as follows. As SignNet [Lim et al., 2023] universally approximates
sign invariant functions on compact sets, we can let  $\mathrm{SignNet}_1: \mathcal{X} \to \mathbb{R}^{d_{\mathrm{hidden}} \times k}$  be a SignNet that approximates  $q_{\mathrm{inv}}(V)$  up to  $\epsilon/(2B)$  accuracy, so
$$
\left| \operatorname {S i g n N e t} _ {1} (V) _ {(\mathbf {d}, a), b} - q _ {\text {i n v}} (V) _ {\mathbf {d}, a, b} \right| <   \frac {\epsilon}{2 B \cdot d _ {\text {h i d d e n}}}. \tag {100}
$$
For  $b \in [k]$ , we also define the weight matrices  $W_{b}^{(1)} \in \mathbb{R}^{d_{\mathrm{hidden}} \times n}$  of the layer by letting the  $(\mathbf{d}, a)$ th row  $(W_{b}^{(1)})_{(\mathbf{d}, a);}$  for any  $a \in [n]$  only be nonzero in the  $i_{\mathbf{d}}$ th index, where it is equal to 1. Thus,
$$
\left(W _ {b} ^ {(1)} v _ {b}\right) _ {\left(\mathbf {d}, a\right)} = V _ {i _ {\mathbf {d}}, b}. \tag {101}
$$
Hence, the first layer takes the form
$$
f _ {1} (V) _ {(\mathbf {d}, a),:} = \left[ V _ {i _ {\mathbf {d}}, 1} \cdot \operatorname {S i g n N e t} _ {1} (V) _ {(\mathbf {d}, a), 1} \quad \dots \quad V _ {i _ {\mathbf {d}}, k} \cdot \operatorname {S i g n N e t} _ {1} (V) _ {(\mathbf {d}, a), k} \right] \in \mathbb {R} ^ {k}. \tag {102}
$$
Now, for the second layer, we let  $\operatorname{SignNet}(V)_{i,j} = 1$  for all  $i \in [n], j \in [k]$ , which can be represented exactly. Then for each column  $b \in [k]$  we will define weight matrices  $W_b^{(2)}$  such that  $(W_b^{(2)})_{a,(\mathbf{d},i)} = 1$  if  $a = i$  and is 0 otherwise. Then we can see that
$$
f _ {2} \circ f _ {1} (V) _ {a, b} = \sum_ {\mathbf {d}} V _ {i _ {\mathbf {d}}, b} \cdot \operatorname {S i g n N e t} _ {1} (V) _ {(d, a), b}. \tag {103}
$$
To see that this approximates the polynomial  $p$ , for any  $V \in \mathcal{X}$  we can bound
$$
\begin{array}{l} \left| p (V) _ {a, b} - f _ {2} \circ f _ {1} (V) _ {a, b} \right| = \left| \sum_ {\mathbf {d}} V _ {i _ {\mathbf {d}, b}} \cdot \left(q _ {\text {i n v}} (V) _ {\mathbf {d}, a, b} - \operatorname {S i g n N e t} _ {1} (V) _ {(\mathbf {d}, a), b}\right) \right| (104) \\ \leq \sum_ {\mathbf {d}} \left| V _ {i _ {\mathbf {d}}, b} \right| \left| \left(q _ {\text {i n v}} (V) _ {\mathbf {d}, a, b} - \operatorname {S i g n N e t} _ {1} (V) _ {(\mathbf {d}, a), b}\right) \right| (105) \\ \leq B \sum_ {\mathbf {d}} \left| \left(q _ {\text {i n v}} (V) _ {\mathbf {d}, a, b} - \operatorname {S i g n N e t} _ {1} (V) _ {(\mathbf {d}, a), b}\right) \right| (106) \\ <   B \sum_ {\mathbf {d}} \frac {\epsilon}{2 B d _ {\text {h i d d e n}}} (107) \\ \leq \frac {\epsilon}{2} (108) \\ \end{array}
$$
By the triangle inequality,  $f_{2} \circ f_{1}$  is  $\epsilon$ -close to  $f_{\mathrm{target}}$ , so we are done.
E Experimental Details
E.1 Miscellaneous Experimental Details
We ran the experiments on a HPC server with CPUs and GPUs. Each experiment was run on a single NVIDIA V100 GPU with 32GB memory. The runtimes for some of our experiments are included in the main paper. Our codes for our models and experiments will be open-sourced and permissively licensed.
E.2 Link Prediction in Nearly Synthetic Graphs
The base graphs  $H$  we generate are Erdős-Renyi or Barabási-Albert graphs with 1000 nodes. We use NetworkX [Hagberg et al., 2008] to generate and process the graphs. The Erdős-Renyi graphs have edge probability  $p = .05$  and the Barabási-Albert graphs have  $m = 20$  new edges per new node. Let  $V = [v_{1},\dots,v_{k}]$  be Laplacian eigenvectors of the graph. We take  $k = 16$  in these experiments. The unlearned decoder baseline simply takes the predicted probability of a link between  $i$  and  $j$  to be proportional to the dot product of the eigenvectors embeddings of node  $i$  and node  $j$ ; this has no learnable parameters. In other words, the node embeddings  $z_{i}$  and  $z_{j}$  are taken to be  $V_{i,j}$  and  $V_{j,i}$ , respectively, and the edge prediction is  $z_{i}^{\top}z_{j}$ . The learned decoder baseline takes the same  $z_{i}$  and  $z_{j}$ ,
but takes the edge prediction to be  $\mathrm{MLP}(z_i\odot z_j)$ . Every other method learns node embeddings  $z_{i}$  and  $z_{j}$ , and takes the edge prediction to be  $z_{i}^{\top}z_{j}$ .
Each model is restricted to around 25,000 learnable parameters (besides the Unlearned Decoder, which has no parameters). We train each method for 100 epochs with an Adam optimizer [Kingma and Ba, 2015] at a learning rate of .01. The train/validation/test split is  $80\% / 10\% / 10\%$ , and is chosen uniformly at random.
E.3 Details on n-body Simulations
We follow the experimental setting and build on the code of Puny et al. [2022] (no license as far as we can tell) for the n-body learning task. The code for generating the data stems from Kipf et al. [2018] (MIT License) and Fuchs et al. [2020] (MIT License). There are 3000 training trajectories, 2000 validation trajectories, and 2000 test trajectories. We modify the data generation code to apply to general dimensions  $d > 3$ . We do not change any of the scaling factors in doing so. For each dimension  $d$ , we use the same hyperparameters for both the frame averaging model and the sign equivariant model.
Table 5: n-body simulation results for dimension  $d = 3$ . Lower MSE is better. Results are from [Satorras et al., 2021, Puny et al., 2022, Kaba et al., 2023].
Model	Test MSE
Linear	.0819
SE(3) Transformer [Fuchs et al., 2020]	.0244
TFN [Thomas et al., 2018]	.0155
Radial Field [Köhler et al., 2020]	.0104
EGNN [Satorras et al., 2021]	.0071
FA-GNN [Puny et al., 2022]	.0057
CN-GNN [Kaba et al., 2023]	.0043
Sign Equivariant (Ours)	.0065
In Table
E.4 Node Classification on CLUSTER
In Section 4.3, we show results for the node classification task CLUSTER [Dwivedi et al., 2022a], where the task is to cluster nodes in graphs drawn from Stochastic Block Models [Abbe, 2017]. Models are restricted to a 100k learnable parameter budget. We largely follow the experimental setting of Rampasek et al. [2022], except we report results for the eigenvector based methods on 5 runs instead of 10.
We test several eigenvector based methods within the GraphGPS framework and codebase [Rampasek et al., 2022] (MIT License), which is a state of the art Transformer / GNN hybrid. Firstly, we make use of the PEG style GraphGPS, which means that the MPNN in the  $l$ th GraphGPS layer takes as edge features  $e_{ij}^{(l)} = \left\| V_i^{(l)} - V_j^{(l)}\right\|^2$ , where  $V_i^{(l)} \in \mathbb{R}^k$  is the eigenvector embedding of node  $i$  in layer  $l$ . This is fully  $O(k)$  invariant (which is much stricter than sign / basis invariance), so we relax this to just be sign invariant in our model by learning a diagonal matrix  $D^{(l)}$  such that  $e_{ij}^{(l)} = V_i^{(l)\top}D^{(l)}V_j^{(l)}$ . Also, the standard GraphGPS only updates eigenvector representations (in a non-equivariant manner) before most of the neural network modules. When we add our sign equivariant model, we instead update eigenvector representations within each GraphGPS layer via  $V^{(l)} = f_{\theta}^{(l)}(V^{(l-1)})$  for a sign equivariant  $f_{\theta}^{(l)}: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$ .