SlowGuess/NeurIPS_2023_OCR huggingface dataset:
a multiplicative factor larger than the sign equivariant requirement of  $\mathcal{O}(n(k + d))$  sized tensors. Moreover, it would take  $\mathcal{O}(nkd^2)$  time to compute  $\phi (v_{i}) + \phi (-v_{i})$  for each  $i$ , which is a multiplicative factor larger than the  $\mathcal{O}(n(kd + d^2))$  time for the sign equivariant architecture.
A.3 Potential Societal Impacts
We do not foresee direct societal impacts from our work. This project is primarily theoretical and aims to improve models for two general application areas: multi-node representation learning and orthogonal equivariant models. Potential societal impacts may arise in downstream applications that may be affected by general progress in geometric machine learning, such as social network analysis and recommender systems. These two applications are known to have negative societal impacts in certain circumstances, so care must be taken in future related work to avoid major negative consequences.
A.4 Edge Representations and Link Prediction
A.4.1 Sign Invariant Link Prediction Decoders
Here, we present an ansatz for universal permutation invariant and sign invariant functions for  $n = 2$ , that is  $f: \mathbb{R}^{2 \times k} \to \mathbb{R}^{d_{\mathrm{out}}}$ . Note that SignNet is only known to be universal for such functions for  $n = 1$ , where there are no permutation symmetries [Lim et al., 2023].
We will parameterize such functions as
$$
f \left(v _ {1}, \dots v _ {k}\right) = \varphi \left(v _ {1} \odot v _ {1}, v _ {1} \odot \operatorname {r e v} \left(v _ {1}\right), \dots , v _ {k} \odot v _ {k}, v _ {k} \odot \operatorname {r e v} \left(v _ {k}\right)\right). \tag {9}
$$
Here,  $\operatorname{rev}:\mathbb{R}^2\to \mathbb{R}^2$  reverses the vector, so  $\mathrm{rev}(a)_1 = a_2$  and  $\mathrm{rev}(a)_2 = a_1$ . Moreover,  $\varphi :$ $\mathbb{R}^{2\times 2k}\rightarrow \mathbb{R}^{d_{\mathrm{out}}}$  is a permutation invariant neural network, so  $\varphi (PX) = \varphi (X)$  for all  $2\times 2$  permutation matrices  $P$ . Note that it is easy to parameterize permutation invariant functions  $\varphi$  in a maximally expressive way, e.g. via DeepSets [Zaheer et al., 2017]. Now, we show that this parameterization is universal:
Proposition 4. Functions  $f: \mathbb{R}^{2 \times k} \to \mathbb{R}^{d_{\mathrm{out}}}$  of the above form are permutation invariant and sign invariant, and they universally approximate permutation invariant and sign invariant functions.
Proof. Invariance of  $f$  is easy to see; let  $P$  be a  $2 \times 2$  permutation matrix and  $s_i \in \{-1, 1\}$  for each  $i$ . Then
$$
\begin{array}{l} f \left(P v _ {1} s _ {1}, \dots , P v _ {k} s _ {k}\right) = \varphi \left(\left(P v _ {1} s _ {1}\right) \odot \left(P v _ {1} s _ {1}\right), \left(P v _ {1} s _ {1}\right) \odot \operatorname {r e v} \left(P v _ {1} s _ {1}\right), \dots\right) (10) \\ = \varphi \left(P \left(v _ {1} s _ {1} \odot v _ {1} s _ {1}\right), P \left(v _ {1} s _ {1} \odot \operatorname {r e v} \left(v _ {1} s _ {1}\right)\right), \dots\right) (11) \\ = \varphi \left(P \left(v _ {1} \odot v _ {1}\right), P \left(v _ {1} \odot \operatorname {r e v} \left(v _ {1}\right)\right), \dots\right) (12) \\ = \varphi \left(v _ {1} \odot v _ {1}, v _ {1} \odot \operatorname {r e v} \left(v _ {1}\right), \dots\right) (13) \\ = f \left(v _ {1}, \dots , v _ {k}\right), (14) \\ \end{array}
$$
where the second to last inequality is by permutation invariance of  $\varphi$ . Next, we show universal approximation.
Let  $h: \mathbb{R}^{2 \times k} \to \mathbb{R}^{d_{\mathrm{out}}}$  be a continuous permutation invariant and sign invariant function. Then by the decomposition theorem in Lim et al. [2023], we can write
$$
h \left(v _ {1}, \dots , v _ {k}\right) = \rho \left(\phi \left(v _ {1} v _ {1} ^ {\top}\right), \dots , \phi \left(v _ {k} v _ {k} ^ {\top}\right)\right), \tag {15}
$$
for continuous functions  $\rho$  and  $\phi$ . As a composition of continuous functions, the function  $\psi : B \subseteq \mathbb{R}^{2 \times 2k} \to \mathbb{R}^{d_{\mathrm{out}}}$  given by  $\psi(A_1, \ldots, A_k) = \rho(\phi(A_1), \ldots, \phi(A_k))$  is continuous, where  $B$  is the subset of  $\mathbb{R}^{2 \times 2k}$  consisting of  $(v_1 v_1^\top, \ldots, v_k v_k^\top)$  such that each  $v_i \in \mathbb{R}^2$ . Note that  $\psi$  is permutation invariant on  $B$ , in the sense that for any  $2 \times 2$  permutation matrix  $P$ , we have
$$
\psi \left(P A _ {1} P ^ {\top}, \dots , P A _ {k} P ^ {\top}\right) = \psi \left(A _ {1}, \dots , A _ {k}\right), \tag {16}
$$
because if  $v_{i}v_{i}^{\top} = A_{i}$ , then
$$
\psi \left(P A _ {1} P ^ {\top}, \dots , P A _ {k} P ^ {\top}\right) = h \left(P v _ {1}, \dots , P v _ {k}\right) = h \left(v _ {1}, \dots , v _ {k}\right) = \psi \left(A _ {1}, \dots , A _ {k}\right), \tag {17}
$$
by permutation invariance of  $h$ .
Now, we define our permutation invariant function  $\varphi : C \subseteq \mathbb{R}^{2 \times 2k} \to \mathbb{R}^{d_{\mathrm{out}}}$ , on the domain
$$
C = \left\{\left[ v _ {1} \odot v _ {1}, v _ {1} \odot \operatorname {r e v} \left(v _ {1}\right), \dots , v _ {k} \odot v _ {k}, v _ {k} \odot \operatorname {r e v} \left(v _ {k}\right) \right]: v _ {i} \in \mathbb {R} ^ {2} \right\}. \tag {18}
$$
We define  $\varphi$  by
$$
\varphi (A) = \psi \left(\left[ \begin{array}{l l} A _ {1, 1} & A _ {2, 2} \\ A _ {2, 2} & A _ {2, 1} \end{array} \right], \left[ \begin{array}{l l} A _ {1, 3} & A _ {2, 4} \\ A _ {2, 4} & A _ {2, 3} \end{array} \right], \dots , \left[ \begin{array}{l l} A _ {1, 2 k - 1} & A _ {2, 2 k} \\ A _ {2, 2 k} & A _ {2, 2 k - 1} \end{array} \right]\right). \tag {19}
$$
To see that  $\varphi$  is permutation invariant, we need only consider the case where  $P = \left[ \begin{array}{ll}0 & 1\\ 1 & 0 \end{array} \right]$ , in which case
$$
\begin{array}{l} \varphi (P A) = \psi \left(\left[ \begin{array}{l l} A _ {2, 1} & A _ {1, 2} \\ A _ {1, 2} & A _ {1, 1} \end{array} \right], \left[ \begin{array}{l l} A _ {2, 3} & A _ {1, 4} \\ A _ {1, 4} & A _ {1, 3} \end{array} \right], \dots , \left[ \begin{array}{l l} A _ {2, 2 k - 1} & A _ {1, 2 k} \\ A _ {1, 2 k} & A _ {1, 2 k - 1} \end{array} \right]\right) (20) \\ = \psi \left(P \left[ \begin{array}{l l} A _ {1, 1} & A _ {2, 2} \\ A _ {2, 2} & A _ {2, 1} \end{array} \right] P ^ {\top}, P \left[ \begin{array}{l l} A _ {1, 3} & A _ {2, 4} \\ A _ {2, 4} & A _ {2, 3} \end{array} \right] P ^ {\top}, \dots , P \left[ \begin{array}{l l} A _ {1, 2 k - 1} & A _ {2, 2 k} \\ A _ {2, 2 k} & A _ {2, 2 k - 1} \end{array} \right] P ^ {\top}\right) (21) \\ = \psi \left(\left[ \begin{array}{l l} A _ {1, 1} & A _ {2, 2} \\ A _ {2, 2} & A _ {2, 1} \end{array} \right], \left[ \begin{array}{l l} A _ {1, 3} & A _ {2, 4} \\ A _ {2, 4} & A _ {2, 3} \end{array} \right], \dots , \left[ \begin{array}{l l} A _ {1, 2 k - 1} & A _ {2, 2 k} \\ A _ {2, 2 k} & A _ {2, 2 k - 1} \end{array} \right]\right) \quad (\psi \text {p e r m . i n v .}) (22) \\ = \varphi (A), (23) \\ \end{array}
$$
where in the second equality, we use the fact that  $A_{2,2j} = A_{1,2j}, j = 1,\ldots,k$  for  $A \in C$ , because  $A_{2,2j} = (v_j \odot \mathrm{rev}(v_j))_2 = (v_j \odot \mathrm{rev}(v_j))_1 = A_{1,2j}$  for some  $v_j \in \mathbb{R}^2$ . Moreover,  $\varphi$  is clearly continuous and sign invariant. Defining  $f: \mathbb{R}^{2 \times k} \to \mathbb{R}^{d_{\mathrm{out}}}$  using this  $\varphi$ , we compute that
$$
\begin{array}{l} f \left(v _ {1}, \dots v _ {k}\right) = \varphi \left(v _ {1} \odot v _ {1}, v _ {1} \odot \operatorname {r e v} \left(v _ {1}\right), \dots , v _ {k} \odot v _ {k}, v _ {k} \odot \operatorname {r e v} \left(v _ {k}\right)\right) (24) \\ = \psi \left(\left[ \begin{array}{c c} v _ {1, 1} ^ {2} & v _ {1, 1} v _ {1, 2} \\ v _ {1, 1} v _ {1, 2} & v _ {1, 2} ^ {2} \end{array} \right], \dots , \left[ \begin{array}{c c} v _ {k, 1} ^ {2} & v _ {k, 1} v _ {k, 2} \\ v _ {k, 1} v _ {k, 2} & v _ {k, 2} ^ {2} \end{array} \right]\right) (25) \\ = \psi \left(v _ {1} v _ {1} ^ {\top}, \dots , v _ {k} v _ {k} ^ {\top}\right) (26) \\ = h \left(v _ {1}, \dots , v _ {k}\right), (27) \\ \end{array}
$$
so we are done.
If  $\varphi$  instead comes from a universally approximating class of permutation invariant neural networks (rather than being an arbitrary continuous permutation invariant function), then on a compact domain we can get  $\epsilon$  approximation of  $f$  to  $h$  by letting  $\varphi$  approximate  $\psi$  to  $\epsilon$  accuracy.
A.4.2 Proof of Proposition 1
Proof. We only prove the basis invariance claim, as the sign invariance claim is a special case; basis invariance is sign invariance when eigenvalues are distinct.
Let  $P \in \mathbb{R}^{n \times n}$  be a permutation matrix associated to an automorphism that maps node  $i$  to node  $j$ , so  $PAP^{\top} = A$  and  $Pe_i = e_j$ , where  $e_l$  is the  $l$ th standard basis vector. Let  $V_t = [v_{r_1}, \ldots, v_{r_{d_t}}]$  be the matrix whose columns are the eigenvectors  $v_{r_l}$  that are associated to eigenvalue  $\lambda_i$ . The columns of  $V_t$  are thus an orthonormal basis for the eigenspace associated to  $\lambda_t$ . Note that for any of these eigenvectors, we have
$$
A \left(P v _ {r _ {l}}\right) = P A P ^ {\top} \left(P v _ {r _ {l}}\right) = P A v _ {r _ {l}} = P \lambda_ {i} v _ {r _ {l}} = \lambda_ {t} \left(P v _ {r _ {l}}\right), \tag {28}
$$
so  $Pv_{r_l}$  is also an eigenvector of  $A$  with eigenvalue  $\lambda_t$ . As  $P$  is orthogonal, note that  $Pv_{r_1},\ldots ,Pv_{r_{d_t}}$  is still an orthonormal basis of the eigenspace. Thus, there exists an orthogonal matrix  $Q_{t}\in \mathbb{R}^{d_{t}\times d_{t}}$  such that  $PV_{t} = V_{t}Q_{t}$  -see Lim et al. [2023].
Repeat the above argument to get such a  $Q_{t}$  for each of the eigenbases  $V_{1},\ldots ,V_{l}$ . We can then see that
$$
\begin{array}{l} z _ {j} = f \left(V _ {1}, \dots , V _ {l}\right) _ {j,:} \\ = f \left(V _ {1} Q _ {1}, \dots , V _ {l} Q _ {l}\right) _ {j,:} \\ = f \left(P V _ {1}, \dots , P V _ {l}\right) _ {j,:} \\ = \left(P f \left(V _ {1}, \dots , V _ {l}\right)\right) _ {j: i} \\ = f \left(V _ {1}, \dots , V _ {l}\right) _ {i:} \\ = z _ {i}. \\ \end{array}
$$
basis invariance
$$
\text {c h o i c e} Q _ {t}
$$
permutation equivariance
choice of  $P$
So we are done.
A.5 Sign Invariance and Structural Node or Node-Pair Encodings
In this section, we show that when the eigenvalues  $\lambda_1,\ldots ,\lambda_k$  are distinct, then sign invariant functions of the orthonormal eigenvectors  $v_{1},\dots,v_{k}$  give structural node or node-pair representations. This can also be generalized in a straightforward way to larger tuples of nodes beyond pairs, though we only consider nodes and node-pairs for ease of exposition. First, we give formal definitions.
Definition 1 (Structural Representations [Srinivasan and Ribeiro, 2019]). Let  $A \in \mathbb{R}^{n \times n}$  be the adjacency matrix of a graph on node set  $\{1, \ldots, n\}$ .
A function  $f: \mathbb{R}^{n \times n} \to \mathbb{R}^n$  is a node structural representation if  $f(PAP^\top) = Pf(A)$  for all  $n \times n$  permutation matrices  $P$ .
A function  $f: \mathbb{R}^{n \times n} \to \mathbb{R}^{n \times n}$  is a node-pair structural representation if  $f(PAP^{\top}) = Pf(A)P^{\top}$  for all  $n \times n$  permutation matrices  $P$ .
Importantly, these structural representations are permutation equivariant functions of adjacency matrices, not arbitrary matrices. For each adjacency matrix  $A$ , let  $V(A) = [v_{1}(A),\dots,v_{k}(A)]$  be a choice of orthonormal eigenvectors for the first  $k$  eigenvalues  $\lambda_1(A),\ldots ,\lambda_k(A)$ . We assume in this section that these first  $k$  eigenvalues are distinct for all  $A$  under consideration, so  $V(A)$  is defined up to sign flips. Let  $h:\mathbb{R}^{n\times k}\to \mathbb{R}^n$  be a permutation equivariant function of sets, so  $h(PX) = Ph(X)$  for all permutations matrices  $P$ . Then of course  $h(PV(A)) = Ph(V(A))$ , but this does not make  $h$  a node structural encoding. This is because  $A\mapsto h(V(A))$  is in general not a well-defined function of the adjacency, since the choice of  $V(A)$  is not well-defined (the choices of sign are arbitrary). If we constrain  $h$  to not depend on the signs (sign invariance), or to depend on the signs in a predictable way (sign equivariance), then we can compute structural node or node-pair encodings from eigenvectors.
We capture these observations in the below proposition. First, we define three types of functions:
- Let  $f_{\mathrm{node}}: \mathbb{R}^{n \times k} \to \mathbb{R}^n$  be sign invariant and permutation equivariant; that is,  $f_{\mathrm{node}}(Pv_1s_1, \ldots, Pv_ks_k) = Pf_{\mathrm{node}}(v_1, \ldots, v_k)$  for  $s_i \in \{-1, 1\}$  and  $P$  a permutation matrix.
- Let  $f_{\mathrm{decode}}: \mathbb{R}^{2 \times k} \to \mathbb{R}$  be a sign invariant; that is,  $f_{\mathrm{decode}}(Sz_i, Sz_j) = f_{\mathrm{decode}}(z_i, z_j)$  for  $S \in \operatorname{diag}\left(\{-1, 1\}^k\right)$ .
- Let  $f_{\mathrm{equiv}}: \mathbb{R}^{n \times k} \to \mathbb{R}^{n \times k}$  be a permutation equivariant and sign equivariant function; that is,  $f_{\mathrm{equiv}}(PV(A)S) = Pf_{\mathrm{equiv}}(V(A))S$  for  $S \in \operatorname{diag}(\{-1, 1\}^k)$  and  $P$  a permutation matrix.
Proposition 5. Let  $\mathcal{A} \subseteq \mathbb{R}^{n \times n}$  denote the matrices with distinct first- $k$  eigenvalues. For  $A \in \mathcal{A}$ , let  $V(A) = [v_{1}(A), \ldots, v_{k}(A)]$  be a choice of orthonormal eigenvectors of  $A$ , associated to the first- $k$  (distinct) eigenvalues  $\lambda_{1}(A), \ldots, \lambda_{k}(A)$ . Then
(a) The map  $q_{\mathrm{node}}: \mathcal{A} \to \mathbb{R}^n$  given by  $q_{\mathrm{node}}(A)_i = f_{\mathrm{node}}(f_{\mathrm{equiv}}(V(A)))_i$  is well-defined and gives a structural node representation.
(b) The map  $q_{\mathrm{pair}}: \mathcal{A} \to \mathbb{R}^{n \times n}$  defined by  $q_{\mathrm{pair}}(A)_{i,j} = f_{\mathrm{decode}}(f_{\mathrm{equiv}}(V(A))_{i,:}, f_{\mathrm{equiv}}(V(A))_{j,:})$  is well-defined and gives a structural node-pair representation.
Note that the identity mapping  $V(A) \mapsto V(A)$  is permutation equivariant and sign equivariant, so using  $f_{\mathrm{node}}$  or  $f_{\mathrm{decode}}$  directly on eigenvectors also gives structural representations. The statement
(b) means that our link prediction pipeline with sign equivariant node features and sign invariant decoding produces structural node-pair representations.
Proof. Part (a) We first show that  $q_{\mathrm{node}}: \mathcal{A} \to \mathbb{R}^n$  is well-defined. Suppose we had another choice of eigenvectors, so the eigenvectors we input are  $V(A)S$  for some  $S \in \operatorname{diag}(\{-1, 1\}^k)$ . Then
$$
f _ {\text {n o d e}} \left(f _ {\text {e q u i v}} (V (A) S)\right) = f _ {\text {n o d e}} \left(f _ {\text {e q u i v}} (V (A)) S\right) = f _ {\text {n o d e}} \left(f _ {\text {e q u i v}} (V (A))\right), \tag {29}
$$
where the first equality is by sign equivariance, and the second equality by sign invariance. Thus, the value of  $q_{\mathrm{node}}(A)$  is unchanged.
Now, let  $P$  be any permutation matrix. Then for each eigenvector  $v_{i}(A), i \in [k]$ , we have  $(PAP^{\top})Pv_{i}(A) = PAv_{i}(A) = \lambda_{i}(A)Pv_{i}(A)$ , so  $Pv_{i}(A)$  is an eigenvector of  $PAP^{\top}$  associated to  $\lambda_{i}(A) = \lambda_{i}(PAP^{\top})$ . Hence, we denote  $v_{i}(PAP^{\top}) = Pv_{i}(A)$  (the choice of sign does not matter as  $q$  does not depend on the sign. Now, we have that
$$
\begin{array}{l} q _ {\text {n o d e}} \left(P A P ^ {\top}\right) = f _ {\text {n o d e}} \left(f _ {\text {e q u i v}} \left(V \left(P A P ^ {\top}\right)\right)\right) (30) \\ = f _ {\text {n o d e}} \left(f _ {\text {e q u i v}} (P V (A))\right) (31) \\ = P f _ {\text {n o d e}} \left(f _ {\text {e q u i v}} (V (A))\right) (32) \\ = P q _ {\text {n o d e}} (A) (33) \\ \end{array}
$$
where the second to last equality is by permutation equivariance of  $f_{\mathrm{node}}$  and  $f_{\mathrm{equiv}}$ .
Part (b) That  $q_{\mathrm{pair}}: \mathcal{A} \to \mathbb{R}^{n \times n}$  is well-defined follows from a similar argument to the  $q_{\mathrm{node}}$  case. Let  $P$  be a permutation matrix, and  $\sigma: [n] \to [n]$  its underlying permutation. We compute that
$$
\begin{array}{l} q _ {\text {p a i r}} \left(P A P ^ {\top}\right) _ {i, j} = f _ {\text {d e c o d e}} \left(f _ {\text {e q u i v}} \left(V \left(P A P ^ {\top}\right)\right) _ {i,:}, f _ {\text {e q u i v}} \left(V \left(P A P ^ {\top}\right)\right) _ {j,:}\right) (34) \\ = f _ {\text {d e c o d e}} \left(f _ {\text {e q u i v}} (P V (A)) _ {i,:}, f _ {\text {e q u i v}} (P V (A)) _ {j,:}\right) (35) \\ = f _ {\text {d e c o d e}} \left(\left[ P f _ {\text {e q u i v}} (V (A)) \right] _ {i,:}, \left[ P f _ {\text {e q u i v}} (V (A)) \right] _ {j,:}\right) (36) \\ = f _ {\text {d e c o d e}} \left(f _ {\text {e q u i v}} (V (A)) _ {\sigma^ {- 1} (i),:,} f _ {\text {e q u i v}} (V (A)) _ {\sigma^ {- 1} (j),:}\right) (37) \\ = q _ {\text {p a i r}} (A) _ {\sigma^ {- 1} (i), \sigma^ {- 1} (j)} (38) \\ = \left(P q _ {\text {p a i r}} (A) P ^ {\top}\right) _ {i, j} (39) \\ \end{array}
$$
A.5.1 Sign Equivariance is Provably More Expressive for Link Prediction
Our arguments in Section 2.1 and Figure 2 explain why we can expect sign equivariant models to be more powerful than sign invariant models in link prediction. To give a theoretically rigorous explanation, here we provide an example where sign equivariant models can provably compute more expressive link representations than sign invariant models.
Consider a cycle graph  $C_{2k}$  for some even length  $2k$ , where  $k \geq 3$ . All nodes are automorphic in this graph, so any model based on structural node representations must assign the same representation to each node-pair. For instance, consider the eigenvalue  $-2$  of the adjacency matrix, which is a simple eigenvalue with eigenvector  $[1, -1, 1, -1, \dots, 1, -1]$  [Lee et al., 1992]. Then a sign invariant model will lose the sign information and map each node to the same encoding, which means that each node-pair will also have the same encoding. However, a sign equivariant model can preserve the sign of each node (for instance by learning the identity function). Then for any pair of nodes that are one hop away, it can take a dot product to compute the pair representation  $-1$ , whereas it can take a dot product between any nodes that are two hops away to compute the pair representation 1. Of course, using more eigenvectors would allow for more complex representations to be computed.
A.5.2 More on Sign Equivariance and Link Prediction
Key to our method is the ability to update a positional node embedding in an equivariant way, which respects the graph symmetries. To elaborate, consider the aforementioned definition of node positional encodings as samples from a permutation equivariant probability distribution over node features [Srinivasan and Ribeiro, 2019]. Laplacian eigenvector positional embeddings are samples from the distribution of orthonormal bases of the eigenspaces of the Laplacian. Our sign equivariance
based approach is possible because the randomness in Laplacian eigenvector positional encodings is exceptionally structured (consisting only of sign flips when eigenvalues are distinct). In contrast, a general way to obtain structural pair representations from node positional embeddings is to average some function over the randomness of the positional encoding (i.e., over many samples of the positional encoding) [Srinivasan and Ribeiro, 2019], but this is highly expensive, often intractable, and introduces substantial variance into the learning procedure. For instance, one may have to average samples of the  $n!$  assignments of unique node identifiers [Murphy et al., 2019] or approximate an integral over Gaussian random features [Abboud et al., 2021].
A.6 Proof of Proposition 2, Orthogonal Equivalence via Sign Equivalence
Proposition 2. Consider a domain  $\mathcal{X} \subseteq \mathbb{R}^{n \times d}$  such that each  $X \in \mathcal{X}$  has distinct covariance eigenvalues, and let  $R_X$  be a choice of orthonormal eigenvectors of  $\operatorname{cov}(X)$  for each  $X \in \mathcal{X}$ . If  $h: \mathcal{X} \subseteq \mathbb{R}^{n \times d} \to \mathbb{R}^{n \times d}$  is sign equivariant, and if  $f(X) = h(X R_X) R_X^\top$ , then  $f$  is well defined and orthogonally equivariant.
Moreover, is  $h$  is from a universal class of sign equivariant functions, then the  $f$  of the above form universally approximate  $O(k)$  equivariant functions on  $\mathcal{X}$ .
Proof. First, we show that  $f$  is well defined.  $R_{X}$  is only unique up to sign flips, as  $R_{X}S$  is an orthonormal set of eigenvectors of  $\operatorname{cov}(X)$  for  $S \in \mathrm{diag}(\{-1,1\}^k)$ . However, no matter the choice of signs,  $f(X)$  takes the same value, since
$$
\begin{array}{l} h \left(X R _ {X} S\right) \left(R _ {X} S\right) ^ {\top} = h \left(X R _ {X} S\right) S ^ {\top} R _ {X} ^ {\top} (40) \\ = h \left(X R _ {X}\right) S S ^ {\top} R _ {X} ^ {\top} \quad \text {s i g n e q u i v a r i a n c e} (41) \\ = h \left(X R _ {X}\right) R _ {X} ^ {\top}. (42) \\ \end{array}
$$
Next, we show that  $f$  is  $O(k)$  equivariant. Let  $Q \in O(k)$  be any orthogonal matrix. Note that
$$
\operatorname {c o v} (X Q) = \left(X Q - \frac {1}{n} \mathbf {1 1} ^ {\top} X Q\right) ^ {\top} \left(X Q - \frac {1}{n} \mathbf {1 1} ^ {\top} X Q\right) = Q ^ {\top} \operatorname {c o v} (X) Q. \tag {43}
$$
Thus,  $Q^{\top}R_{X}$  is an orthonormal set of eigenvectors of  $\operatorname{cov}(XQ)$ . This means that there is a choice of signs  $S \in \mathrm{diag}(\{-1,1\}^{k})$  such that  $Q^{\top}R_{X}S = R_{XQ}$ . Hence, we have that
$$
\begin{array}{l} f (X Q) = h \left(X Q R _ {X Q}\right) R _ {X Q} ^ {\top} (44) \\ = h \left(X Q Q ^ {\top} R _ {X} S\right) \left(Q ^ {\top} R _ {X} S\right) ^ {\top} (45) \\ = h \left(X R _ {X}\right) S S ^ {\top} R _ {X} ^ {\top} Q \quad \text {s i g n e q u i v a r i a n c e} (46) \\ = h \left(X R _ {X}\right) R _ {X} ^ {\top} Q (47) \\ = f (X) Q ^ {\top}, (48) \\ \end{array}
$$
so  $f$  is  $O(k)$  equivariant.
Universal Approximation. Our proof of the universality of this class of functions builds on the proof of the universality of frame averaging [Puny et al., 2022]. Let  $f_{\mathrm{target}}$  be a continuous  $O(k)$  equivariant function and let  $\epsilon > 0$  be a desired approximation accuracy. Then  $f_{\mathrm{target}}$  is also sign equivariant (as the sign matrices  $S \in \mathrm{diag}(\{-1,1\}^k)$  are orthogonal).
Hence, by sign equivariant universality, we can choose a sign equivariant  $h$  such that  $\| h(X) - f_{\mathrm{target}}(X)\| < \epsilon$  for all  $X \in \mathcal{X}$  (where  $\| \cdot \|$  is the Frobenius norm). Define the  $O(k)$  equivariant  $f(X) = h(XR_X)R_X^\top$ . Then for all  $X \in \mathcal{X}$  we have that
$$
\begin{array}{l} \left\| f _ {\text {t a r g e t}} (X) - f (X) \right\| = \left\| f _ {\text {t a r g e t}} (X) - h \left(X R _ {X}\right) R _ {X} ^ {\top} \right\| (49) \\ = \left\| f _ {\text {t a r g e t}} (X) R _ {X} R _ {X} ^ {\top} - h \left(X R _ {X}\right) R _ {X} ^ {\top} \right\| \quad R _ {X} \text {o r t h o g o n a l} (50) \\ = \left\| f _ {\text {t a r g e t}} \left(X R _ {X}\right) R _ {X} ^ {\top} - h \left(X R _ {X}\right) R _ {X} ^ {\top} \right\| \quad \text {o r t h o g o n a l e q u i v a r i a n c e} (51) \\ = \left\| f _ {\text {t a r g e t}} \left(X R _ {X}\right) - h \left(X R _ {X}\right) \right\| \quad R _ {X} \text {o r t h o g o n a l} (52) \\ <   \epsilon . (53) \\ \end{array}
$$
So  $f$  approximates  $f_{\mathrm{target}}$  within  $\epsilon$  accuracy on  $\mathcal{X}$  , and we are done.
B Sign Equivariant Linear Maps
B.1 Sign Equivariant Linear Map Characterization
We first prove our result characterizing the form of the equivariant linear maps from  $\mathbb{R}^{n\times k}\to \mathbb{R}^{n'\times k}$ . Lemma 1. A linear map  $W:\mathbb{R}^{n\times k}\to \mathbb{R}^{n'\times k}$  is sign equivariant if and only if it can be written as
$$
W (X) = \left[ W _ {1} X _ {1} \dots W _ {k} X _ {k} \right] \tag {54}
$$