SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. For one direction, suppose  $W$  can be written as in equation 4. To see that  $W$  is sign equivariant, note that for any  $S \in \mathrm{diag}(\{-1,1\}^k)$ , we have
$$
W (X S) = \left[ s _ {1} W _ {1} X _ {1} \dots s _ {k} W _ {k} X _ {k} \right] = \left[ W _ {1} X _ {1} \dots W _ {k} X _ {k} \right] S = W (X) S. \tag {55}
$$
For the other direction, let  $W$  be a sign equivariant linear map. For any  $i' \in [n']$  and  $j' \in [k]$ , we can write the action of  $W$  as
$$
W (X) _ {i ^ {\prime}, j ^ {\prime}} = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {k} W _ {i ^ {\prime}, j ^ {\prime}} ^ {i, j} X _ {i, j}, \tag {56}
$$
where  $W_{i',j'}^{i,j} \in \mathbb{R}$  are coefficients representing the linear map. Let  $c \neq j'$  be a column that is not  $j'$ . Further, for any row  $l \in [n]$ , let  $\tilde{X} \in \mathbb{R}^{n \times k}$  be such that  $\tilde{X}_{l,c} = 1$ , and  $\tilde{X}$  is zero elsewhere. Then we have that
$$
W (\tilde {X}) _ {i ^ {\prime}, j ^ {\prime}} = W _ {i ^ {\prime}, j ^ {\prime}} ^ {l, c}. \tag {57}
$$
Now, let  $S \in \mathrm{diag}(\{-1, 1\}^k)$  have a  $-1$  in the  $j'$ th column and a 1 elsewhere. Then  $\tilde{X} S = \tilde{X}$ . This implies that
$$
\begin{array}{l} W _ {i ^ {\prime}, j ^ {\prime}} ^ {l, c} = W (\tilde {X}) _ {i ^ {\prime}, j ^ {\prime}} (58) \\ = W \left(\tilde {X} S\right) _ {i ^ {\prime}, j ^ {\prime}} (59) \\ = - W (\tilde {X}) _ {i ^ {\prime}, j ^ {\prime}} (60) \\ = - W _ {i ^ {\prime}, j ^ {\prime}} ^ {l, c}, (61) \\ \end{array}
$$
where in the second to last equality we used sign equivariance. This implies that  $W_{i',j'}^{l,c} = 0$ .
Hence, for any  $i' \in [n']$ ,  $j' \in [k']$ , we have that  $W(X)_{i',j'}$  only depends on  $X_{j'}$ , so we are done.
B.2 Sign Equivariant Linear Maps between Tensor Representations
Since the sign equivariant linear maps from  $\mathbb{R}^{n\times k}\to \mathbb{R}^{n'\times k}$  are very weak, we now characterize sign equivariant linear maps between higher order tensor representations, as past work has done for other groups [Maron et al., 2018, 2019, Finzi et al., 2021]. In particular, we will consider representations  $\mathbb{R}^{k^m}$  for natural numbers  $m$ . The action of  $s\in \{-1,1\} ^k$  on  $V\in \mathbb{R}^{k^m}$  is as follows:
$$
(s \cdot V) _ {i _ {1}, \dots , i _ {m}} = s _ {i _ {1}} \dots s _ {i _ {m}} V _ {i _ {1}, \dots , i _ {m}}, \tag {62}
$$
for  $i_1, \ldots, i_m \in [k]$ . We now prove a result showing that there are no sign-equivariant linear maps between many pairs of tensor representations.
Proposition 6. If  $m_1 + m_2$  is odd, then the only sign equivariant linear map  $L: \mathbb{R}^{k^{m_1}} \to \mathbb{R}^{k^{m_2}}$  is the zero map.
Proof. Let  $L: \mathbb{R}^{k^{m_2} \times k^{m_1}}$  be the matrix associated with a sign equivariant linear map from  $\mathbb{R}^{k^{m_1}} \to \mathbb{R}^{k^{m_2}}$ . This means that for  $V \in \mathbb{R}^{k^{m_1}}$ , for  $i_1, \ldots, i_{m_2} \in [k]$  we have that
$$
(L V) _ {i _ {1}, \dots , i _ {m _ {2}}} = \sum_ {j _ {1}, \dots , j _ {m _ {1}} = 1} ^ {k} L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}} V _ {j _ {1}, \dots , j _ {m _ {1}}}. \tag {63}
$$
Then by sign equivariance, we have that
$$
L (s \cdot V) = s \cdot (L V), \tag {64}
$$
for  $s \in \{-1, 1\}^k$ , which means that for all  $i_1, \ldots, i_{m_2} \in [k]$
$$
\sum_ {j _ {1}, \dots , j _ {m _ {1}} = 1} ^ {k} L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}} s _ {j _ {1}} \dots s _ {j _ {m _ {1}}} V _ {j _ {1}, \dots , j _ {m _ {1}}} \tag {65}
$$
$$
= s _ {i _ {1}} \dots s _ {i _ {m _ {2}}} \sum_ {j _ {1}, \dots , j _ {m _ {1}} = 1} ^ {k} L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}} V _ {j _ {1}, \dots , j _ {m _ {1}}}, \tag {66}
$$
$$
\sum_ {j _ {1}, \dots , j _ {m _ {1}} = 1} ^ {k} L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}} s _ {i _ {1}} \dots s _ {i _ {m _ {2}}} s _ {j _ {1}} \dots s _ {j _ {m _ {1}}} V _ {j _ {1}, \dots , j _ {m _ {1}}} \tag {67}
$$
$$
= \sum_ {j _ {1}, \dots , j _ {m _ {1}} = 1} ^ {k} L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}} V _ {j _ {1}, \dots , j _ {m _ {1}}}. \tag {68}
$$
This gives that  $s \cdot L = L$ . Choose some arbitrary indices  $i_1, \ldots, i_{m_2}, j_1, \ldots, j_{m_1} \in [k]$ . Since  $m_1 + m_2$  is odd, then one of these values appears an odd number of times — say it is some index  $p \in [k]$ . Let  $s \in \{-1, 1\}^k$  have  $s_i = 1$  for  $i \neq p$  and  $s_p = -1$ . Then  $s \cdot L = L$  implies that
$$
- L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}} = L _ {i _ {1}, \dots , i _ {m _ {2}}, j _ {1}, \dots , j _ {m _ {1}}}. \tag {69}
$$
Which implies that  $L$  is zero at these indices, and hence  $L = 0$  everywhere, since these were arbitrarily chosen indices.
This implies that we cannot map from  $\mathbb{R}^k\to \mathbb{R}^{k^2}$  using sign equivariant linear maps. Thus, if we want to lift our order-one tensor input to higher tensor orders in a linear way, then we need to at least map to third order tensors in  $\mathbb{R}^{k^3}$ . This requires substantial memory cost. Furthermore, we next show that even if we could map to third order tensors, learning representations of third order tensors with sign equivariant linear maps is expensive.
Proposition 7. The dimension of the space of sign equivariant linear maps from  $\mathbb{R}^{k^{m_1}}\to \mathbb{R}^{k^{m_2}}$  is
$$
\frac {1}{2 ^ {k}} \sum_ {s \in \{- 1, 1 \} ^ {k}} \left(s _ {1} + \dots + s _ {k}\right) ^ {m _ {1} + m _ {2}}. \tag {70}
$$
Proof. This is a direct consequence of the First Projection Formula: see Fulton and Harris [2013] Section 2.2. In particular, it can be seen that the dimension of this space is equal to
$$
\frac {1}{2 ^ {k}} \sum_ {S \in \operatorname {D i a g} \left(\{- 1, 1 \} ^ {k}\right)} \operatorname {t r a c e} \left(S ^ {\otimes \left(m _ {1} + m _ {2}\right)}\right) \tag {71}
$$
by the First Projection Formula. Since  $\operatorname{trace}(A \otimes B) = \operatorname{trace}(A) \cdot \operatorname{trace}(B)$ , this is equal to
$$
\frac {1}{2 ^ {k}} \sum_ {S \in \operatorname {D i a g} (\{- 1, 1 \} ^ {k})} \operatorname {t r a c e} (S) ^ {m _ {1} + m _ {2}} = \frac {1}{2 ^ {k}} \sum_ {s \in \{- 1, 1 \} ^ {k}} \left(s _ {1} + \dots + s _ {k}\right) ^ {m _ {1} + m _ {2}}. \tag {72}
$$
Using Proposition 7, we have numerically computed the dimension of the space of sign equivariant linear maps from  $\mathbb{R}^{k^3}\to \mathbb{R}^{k^3}$ ; see Table 4. The dimension appears to be  $15k^{3} - 30k^{2} + 16k$  for  $k$  up to 20. In particular, when  $k = 8$ , we compute that the space of sign equivariant linear maps from  $\mathbb{R}^{k^3}\rightarrow \mathbb{R}^{k^3}$  is of dimension 5888, which is already quite large.
Table 4: Dimension of the space of sign equivariant linear maps between third-order tensor representations  $\mathbb{R}^{k^3}\to \mathbb{R}^{k^3}$ .
k	Dimension
1	1
2	32
3	183
4	544
5	1,205
6	2,256
7	3,787
8	5,888
9	8,649
10	12,160
11	16,511
12	21,792
13	28,093
14	35,504
15	44,115
16	54,016
17	65,297
18	78,048
19	92,359
20	108,320
C Characterization of Sign Equivariant Polynomials
In this Appendix, we characterize the form of the sign equivariant polynomials. This is useful, because for a finite group, equivariant polynomials universally approximate equivariant continuous functions [Yarotsky, 2022]; thus, if a model universally approximates equivariant polynomials, then it universally approximates equivariant continuous functions. Using equivariant polynomials to analyze or develop equivariant machine learning models has been done successfully in many contexts [Zaheer et al., 2017, Yarotsky, 2022, Segol and Lipman, 2019, Dym and Maron, 2021, Maron et al., 2019, 2020, Villar et al., 2021, Dym and Gortler, 2022, Puny et al., 2023].
C.1 Sign Invariant Polynomials  $\mathbb{R}^k\to \mathbb{R}$
Next, we characterize the form of sign invariant and equivariant polynomials. For simplicity, we start with the case of sign invariant polynomials  $p: \mathbb{R}^k \to \mathbb{R}$ . The sign equivariant polynomials take a very similar form. We can write any polynomial from  $\mathbb{R}^k$  to  $\mathbb{R}$  in the form
$$
p (v) = \sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} v _ {1} ^ {d _ {1}} \dots v _ {k} ^ {d _ {k}} \tag {73}
$$
for some coefficients  $\mathbf{W}_{d_1,\ldots ,d_k}\in \mathbb{R}$  and some  $D\in \mathbb{N}$ . Sign invariance tells us that for any  $S = \mathrm{diag}(s_1,\dots ,s_k)\in \mathrm{diag}(\{-1,1\} ^k)$ , we must have
$$
\sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} v _ {1} ^ {d _ {1}} \dots v _ {k} ^ {d _ {k}} = p (v) = p (S v) = \sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} s _ {1} ^ {d _ {1}} \dots s _ {k} ^ {d _ {k}} v _ {1} ^ {d _ {1}} \dots v _ {k} ^ {d _ {k}}. \tag {74}
$$
This holds for any  $v \in \mathbb{R}^k$ , so for all choices of  $d_1, \ldots, d_k$  we must have
$$
\mathbf {W} _ {d _ {1}, \dots , d _ {k}} = s _ {1} ^ {d _ {1}} \dots s _ {k} ^ {d _ {k}} \mathbf {W} _ {d _ {1}, \dots , d _ {k}}, \quad \text {f o r a l l} (s _ {1}, \dots , s _ {k}) \in \{- 1, 1 \} ^ {k}. \tag {75}
$$
Note that  $s_i^{d_i} = 1$  if  $d_i$  is an even number. Hence, there are no constraints on  $\mathbf{W}_{d_1,\ldots,d_k}$  if all  $d_i$  are even. On the other hand, suppose  $d_j$  is odd for some  $j$ . Let  $s_i = 1$  for  $i \neq j$  and  $s_j = -1$ . Then the
constraint says that  $\mathbf{W}_{d_1,\dots,d_k} = -\mathbf{W}_{d_1,\dots,d_k}$ , so we must have  $\mathbf{W}_{d_1,\dots,d_k} = 0$ . To summarize, we have
$$
\mathbf {W} _ {d _ {1}, \dots , d _ {k}} = \left\{ \begin{array}{l l} \text {f r e e} & d _ {i} \text {e v e n f o r e a c h} i \\ 0 & \text {e l s e} \end{array} \right. \tag {76}
$$
Where being free means that the coefficient may take any value in  $\mathbb{R}$ . Thus, any sign invariant  $p$  only has terms where each variable  $v_{i}$  is raised to an even power. It is also easy to see that any polynomial  $p$  where each variable  $v_{i}$  is raised to only even powers is sign invariant, so we have the following proposition:
Proposition 8. A polynomial  $p: \mathbb{R}^k \to \mathbb{R}$  is sign invariant if and only if it can be written
$$
p (v) = \sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} v _ {1} ^ {2 d _ {1}} \dots v _ {k} ^ {2 d _ {k}}, \tag {77}
$$
for some coefficients  $\mathbf{W}_{d_1,\dots,d_k} \in \mathbb{R}$  and  $D \in \mathbb{N}$ .
In other words,  $p$  is sign invariant if and only if there exists a polynomial  $q: \mathbb{R}^k \to \mathbb{R}$  such that  $p(v) = q(v_1^2, \ldots, v_k^2)$ .
C.2 Sign Equivariant Polynomials  $\mathbb{R}^k\to \mathbb{R}^k$
The case of sign equivariant polynomials  $p: \mathbb{R}^k \to \mathbb{R}^k$  is very similar. For  $l \in [k]$ , the  $l$ th output dimension of any polynomial  $p: \mathbb{R}^k \to \mathbb{R}^k$  can be written
$$
p (v) _ {l} = \sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} v _ {1} ^ {d _ {1}} \dots v _ {k} ^ {d _ {k}}, \tag {78}
$$
where  $\mathbf{W}_{d_1,\ldots ,d_k}^{(l)}\in \mathbb{R}$  are coefficients (note the extra  $l$  index, so there are  $k$  times more coefficients than in the invariant case). By sign equivariance, we have
$$
s _ {l} \cdot p (v) _ {l} = p (S v) _ {l} \tag {79}
$$
$$
s _ {l} \cdot \sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} v _ {1} ^ {d _ {1}} \dots v _ {k} ^ {d _ {k}} = \sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} s _ {1} ^ {d _ {1}} \dots s _ {k} ^ {d _ {k}} v _ {1} ^ {d _ {1}} \dots v _ {k} ^ {d _ {k}}. \tag {80}
$$
As this holds for all inputs  $v \in \mathbb{R}^k$ , we have the following constraints on the coefficients:
$$
s _ {l} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} = s _ {1} ^ {d _ {1}} \dots s _ {k} ^ {d _ {k}} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} \tag {81}
$$
$$
\mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} = s _ {l} \cdot s _ {1} ^ {d _ {1}} \dots s _ {k} ^ {d _ {k}} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)}, \tag {82}
$$
where we use the fact that  $s_l = 1 / s_l$  since  $s_l \in \{-1, 1\}$ . If  $d_j$  is odd for  $j \neq l$ , then similarly to the invariant case, we can take  $s_i = 1$  for  $i \neq j$  and  $s_j = -1$  in the above equation to see that  $\mathbf{W}_{d_1, \ldots, d_k}^{(l)} = 0$ . If  $d_l$  is even, then  $d_l + 1$  is odd, so we have that  $\mathbf{W}_{d_1, \ldots, d_k}^{(l)} = 0$  by the same argument. Thus, we must have
$$
\mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} = \left\{ \begin{array}{l l} \text {f r e e} & d _ {l} \text {o d d}, \text {a n d} d _ {i} \text {e v e n f o r e a c h} i \neq l \\ 0 & \text {e l s e} \end{array} . \right. \tag {83}
$$
Thus, the  $l$ th entry  $p(v)_l$  only contains monomials of the term  $v_1^{2d_1} \cdots v_l^{2d_l + 1} \cdots v_k^{2d_k}$ , where each term besides  $v_l$  is raised to an even power. We can factor out a  $v_l$  and write such terms as  $v_l \cdot v_1^{2d_1} \cdots v_k^{2d_k}$ . It is also easy to see that any polynomial with monomials only of this form is sign equivariant. Thus, we have proven Proposition 9.
Proposition 9. A polynomial  $p: \mathbb{R}^k \to \mathbb{R}^k$  is sign equivariant if and only if it can be written
$$
p (v) _ {l} = v _ {l} \cdot \left(\sum_ {d _ {1}, \dots , d _ {k} = 0} ^ {D} \mathbf {W} _ {d _ {1}, \dots , d _ {k}} ^ {(l)} v _ {1} ^ {2 d _ {1}} \dots v _ {k} ^ {2 d _ {k}}\right). \tag {84}
$$
In vector format,  $p$  is sign equivariant if and only if it can be written as  $p(v) = v \odot p_{\mathrm{inv}}(v)$  for a sign invariant polynomial  $p_{\mathrm{inv}}: \mathbb{R}^k \to \mathbb{R}^k$ .
C.3 Sign Equivariant Polynomials  $\mathbb{R}^{n\times k}\to \mathbb{R}^{n^{\prime}\times k}$
Finally, we will handle the case of polynomials  $p: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  equivariant to  $\mathrm{diag}(\{-1, 1\}^k)$ . This is the case we most often deal with in practice, when we have input  $V = [v_1 \ldots v_k]$  for  $k$  eigenvectors  $v_i \in \mathbb{R}^n$  of some  $n \times n$  matrix. For  $a \in [n']$  and  $b \in [k]$ , the  $(a, b)$ th output of a polynomial  $\mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  is
$$
p (V) _ {a, b} = \sum_ {d _ {i, j} = 0} ^ {D} \mathbf {W} _ {\mathbf {d}} ^ {(a, b)} \prod_ {i = 1} ^ {n} \prod_ {j = 1} ^ {k} V _ {i, j} ^ {d _ {i, j}}, \tag {85}
$$
where the sum ranges over  $d_{i,j} \in \{0, \dots, D\}$  for  $i \in [n]$  and  $j \in [k]$ , and  $\mathbf{d} = (d_{1,1}, \ldots, d_{n,1}, d_{1,2}, \ldots, d_{n,k})$  is a shorthand to index coefficients  $\mathbf{W}_{\mathbf{d}}^{(a,b)} \in \mathbb{R}$ . By sign equiv-ariance, we have that:
$$
s _ {b} \cdot p (V) _ {a, b} = p (V S) _ {a, b} \tag {86}
$$
$$
s _ {b} \cdot \sum_ {d _ {i, j} = 0} ^ {D} \mathbf {W} _ {\mathbf {d}} ^ {(a, b)} \prod_ {i = 1} ^ {n} \prod_ {j = 1} ^ {k} V _ {i, j} ^ {d _ {i, j}} = \sum_ {d _ {i, j} = 0} ^ {D} \mathbf {W} _ {\mathbf {d}} ^ {(a, b)} s _ {1} ^ {\tilde {d} _ {1}} \dots s _ {k} ^ {\tilde {d} _ {k}} \prod_ {i = 1} ^ {n} \prod_ {j = 1} ^ {k} V _ {i, j} ^ {d _ {i, j}}, \tag {87}
$$
where  $\tilde{d}_j = \sum_{i' = 1}^n d_{i',j}$  is the number of times that an entry from column  $j$  of  $V$  appears in the product  $\prod_{i = 1}^{n}\prod_{j = 1}^{k}V_{i,j}^{d_{i,j}}$ . As this holds over all  $V$ , we thus have that
$$
\mathbf {W} _ {\mathbf {d}} ^ {(a, b)} = s _ {b} \cdot s _ {1} ^ {\tilde {d} _ {1}} \dots s _ {k} ^ {\tilde {d} _ {k}} \cdot \mathbf {W} _ {\mathbf {d}} ^ {(a, b)}. \tag {88}
$$
By analogous arguments to the previous subsections, if  $\tilde{d}_j$  is odd for  $j \neq b$ , we have that the  $\mathbf{W}_{\mathbf{d}}^{(a,b)} = 0$ . Likewise, if  $\tilde{d}_b$  is even, we have  $\mathbf{W}_{\mathbf{d}}^{(a,b)} = 0$ . Thus, the constraint on  $\mathbf{W}$  is
$$
\mathbf {W} _ {\mathbf {d}} ^ {(a, b)} = \left\{ \begin{array}{l l} \text {f r e e} & \sum_ {i} d _ {i, b} \text {o d d}, \text {a n d} \sum_ {i} d _ {i, j} \text {e v e n f o r e a c h} j \neq b \\ 0 & \text {e l s e} \end{array} . \right. \tag {89}
$$
In particular, this means that the only nonzero terms in the sum that defines  $p(V)_{a,b}$  have an even number of entries from column  $j$  for  $j \neq b$ , and an odd number of entries from column  $b$ . Thus, each term can be written as  $V_{i_{\mathbf{d}},b} \cdot p_{\mathrm{inv}}(V)_{\mathbf{d}}$  for some index  $i_{\mathbf{d}} \in [n]$  and sign invariant polynomial  $p_{\mathrm{inv}}$ . Moreover, it can be seen that any polynomial that only has terms of this form is sign equivariant. Thus, we have shown the following proposition:
Proposition 10. A polynomial  $p: \mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  is sign equivariant if and only if it can be written as
$$
p (V) _ {a, b} = \sum_ {d _ {i, j} = 0} ^ {D} \mathbf {w} _ {\mathbf {d}} ^ {(a, b)} V _ {i _ {\mathbf {d}}, b} \cdot p _ {\text {i n v}} (V) _ {\mathbf {d}}, \tag {90}
$$
where  $p_{\mathrm{inv}}$  is a sign invariant polynomial, the sum ranges over all  $\mathbf{d}$ , and  $i_{\mathbf{d}} \in [n]$  for each  $\mathbf{d}$ .
Now, we show that this implies Theorem 1. In particular, we will write  $p$  in the form
$$
p (V) = W ^ {(2)} \left(\left(W ^ {(1)} V\right) \odot q _ {\text {i n v}} (V)\right), \tag {91}
$$
for sign equivariant linear maps  $W^{(2)}$  and  $W^{(1)}$ , and a sign equivariant polynomial  $q_{\mathrm{inv}}$ . To do so, let  $\tilde{D}$  denote the number of all possible  $\mathbf{d}$  that the sum in equation 90 ranges over. We take  $W^{(1)}: \mathbb{R}^{n \times \tilde{k}} \to \mathbb{R}^{\tilde{D} n' \times k}$  and  $W^{(2)}: \mathbb{R}^{\tilde{D} n' \times k} \to \mathbb{R}^{n' \times k}$ . These sign equivariant linear maps have to act independently on each column of their input, so  $W^{(1)}V = [W_1^{(1)}v_1, \ldots, W_k^{(1)}v_k]$  for linear maps  $W_i^{(1)}: \mathbb{R}^n \to \mathbb{R}^{\tilde{D} n'}$ . We define  $W_b^{(1)}$  to be the linear map such that  $(W_b^{(1)}v_b)_{\mathbf{d},a} = W_\mathbf{d}^{(a,b)}V_{i_\mathbf{d},b}$  for  $a \in [n']$ . For the sign invariant polynomial  $q_{\mathrm{inv}}$ , we take  $q_{\mathrm{inv}}(V)_{\mathbf{d},a} = p_{\mathrm{inv}}(V)_{\mathbf{d}}$ .
Finally, we define  $W^{(2)}$  to compute the sum in equation 90. In particular, for  $X = [x_{1},\ldots ,x_{k}]\in \mathbb{R}^{\tilde{D} n^{\prime}\times k}$  we write  $W^{(2)}X = [W_1^{(2)}x_1,\dots ,W_k^{(2)}x_k]$ , where  $(W_b^{(2)}x_b)_a = \sum_{\mathbf{d}}x_{i_{\mathbf{d}},b}$ . It can be seen that with these definitions of  $W^{(2)},W^{(1)}$ , and  $q_{\mathrm{inv}}$ , we have written  $p$  in the desired form.
C.4 Sign Invariant Polynomials and SignNet
For completeness, here we state the form of the sign invariant polynomials  $p: \mathbb{R}^{n \times k} \to \mathbb{R}$  on inputs  $V = [v_{1}, \ldots, v_{k}] \in \mathbb{R}^{n \times k}$ . The derivation very closely follows that of the sign equivariant polynomials from  $\mathbb{R}^{n \times k} \to \mathbb{R}^{n' \times k}$  in Appendix C.3, so we omit this derivation.
Proposition 11. A polynomial  $p: \mathbb{R}^{n \times k} \to \mathbb{R}$  is sign invariant if and only if it can be written
$$
p (V) = \sum_ {d _ {i, j} = 0} ^ {D} \mathbf {W} _ {\mathbf {d}} \prod_ {i = 1} ^ {n} \prod_ {j = 1} ^ {k} V _ {i, j} ^ {d _ {i, j}}, \tag {92}
$$
where  $\mathbf{W}_{\mathbf{d}} \neq 0$  for  $\mathbf{d} = (d_{1,1}, \ldots, d_{n,1}, d_{1,2}, \ldots, d_{n,k})$  only if  $\sum_{i=1}^{n} d_{i,j}$  is even for each column  $j \in [k]$ .
In particular,  $p$  is sign invariant if and only if there is a polynomial  $q: \mathbb{R}^{n \times n \times k} \to \mathbb{R}$  such that  $p(V) = q([V_{i_1,j} \cdot V_{i_2,j}]_{i_1 \in [n], i_2 \in [n], j \in [k]})$ .
The polynomials  $V \mapsto V_{i_1,j} \cdot V_{i_2,j}$  for  $i_1, i_2 \in [n]$  and  $j \in [k]$  are thus generators of the ring of sign invariant polynomials from  $\mathbb{R}^{n \times k} \to \mathbb{R}$ .
Notably, Lim et al. [2023] propose universal sign invariant neural architectures, but do not characterize or otherwise use the sign invariant polynomials. Instead, their proof of universality uses topological constructions and shows that all sign invariant continuous functions can be decomposed in a simple form—namely,  $\rho ([\phi (v_i) + \phi (-v_i)]_{i = 1,\dots ,k})$  for continuous functions  $\rho$  and  $\phi$ . Our characterization of sign invariant polynomials provides another path to developing and analyzing the expressive power of sign invariant architectures.
In particular, we can give an alternative proof for the universality of SignNet.
Proposition 12 (Universality of SignNet). Let  $f: \mathcal{X} \subseteq \mathbb{R}^{n \times k} \to \mathbb{R}$  be a continuous sign invariant function on a compact domain  $\mathcal{X}$ , and let  $\epsilon > 0$ . Then there exists a continuous  $\rho: \mathbb{R}^{n^2k} \to \mathbb{R}$  and continuous  $\phi: \mathbb{R}^n \to \mathbb{R}^{n^2}$  such that  $|f(V) - \rho([\phi(v_i) + \phi(-v_i)]_{i=1,\ldots,k})| < \epsilon$  for all  $V \in \mathcal{X}$ .
Proof. First, let  $p$  be a sign invariant polynomial that approximates  $f$  to within  $\epsilon$  on  $\mathcal{X}$ . Then using Proposition 11, let  $q$  be a polynomial such that  $p(V) = q([V_{i_1,j}\cdot V_{i_2,j}]_{i_1\in [n],i_2\in [n],j\in [k]})$ .
Define  $\phi : \mathbb{R}^n \to \mathbb{R}^{n^2}$  to map a  $v \in \mathbb{R}^n$  to the vector of pairwise products of elements in  $v$  scaled by  $1/2$ , that is
$$
\phi (v) = \frac {1}{2} \operatorname {v e c} \left(v v ^ {\top}\right) \tag {93}
$$
Then  $\phi(v) + \phi(-v)$  is equal to the vector of pairwise products of  $v$ . Finally, we let  $\rho = q$ , which gives that
$$
p (V) = \rho ([ \phi (v _ {i}) + \phi (- v _ {i}) ] _ {i = 1, \dots , k}), \tag {94}
$$
$$
| f (V) - \rho ([ \phi (v _ {i}) + \phi (- v _ {i}) ] _ {i = 1, \dots , k}) | = | f (V) - p (V) | <   \epsilon \tag {95}
$$
for all  $V\in \mathcal{X}$
Given the form of the sign invariant polynomials, this proof is quite simple. However, it is technically weaker than the result of Lim et al. [2023], as they invoke the Strong Whitney Embedding Theorem and only require  $\phi$  to map to  $\mathbb{R}^{2n}$  instead of  $\mathbb{R}^{n^2}$ . Still, further arguments could probably reduce the dimension required to about  $2n$  in this polynomial-based proof; as the Gram matrix  $vv^\top$  is rank one, it can be recovered almost always from about  $2n$  of its entries [Pimentel-Alarcón et al., 2016].
D Sign Equivariant Architecture Universality
In this section, we prove Proposition 3 on the universality of our proposed sign equivariant architectures, which we restate here:
We prove the two statements of the proposition in the next two subsections.
D.1 Universality for functions  $\mathbb{R}^k\to \mathbb{R}^k$
Proof. Let  $\mathcal{X} \subseteq \mathbb{R}^k$  be a compact set, let  $\epsilon > 0$ , and let  $f_{\mathrm{target}}: \mathcal{X} \to \mathbb{R}^k$  be a continuous sign equivariant function that we wish to approximate within  $\epsilon$ . Choose a sign equivariant polynomial  $p$  that approximates  $f_{\mathrm{target}}$  to within  $\epsilon/2$  on  $\mathcal{X}$ . By compactness, we can choose a finite bound  $B > 0$  such that  $|v_i| < B$  for all  $v \in \mathcal{X}$ .
By Proposition 9, we can write  $p(v)_l = v_l \cdot \sum_{d_1,\ldots,d_k=0}^{D} \mathbf{W}_{d_1,\ldots,d_k} v_1^{2d_1} \cdots v_k^{2d_k}$ . By the universal approximation theorem for multilayer perceptrons, we can choose a MLP:  $\mathcal{X} \to \mathbb{R}^k$  such that approximates  $q(v) = \sum_{d_1,\ldots,d_k=0}^{D} \mathbf{W}_{d_1,\ldots,d_k} v_1^{2d_1} \cdots v_k^{2d_k}$  up to  $\epsilon/(2B)$ . Note that  $q(|v|) = q(v)$ , so  $v \mapsto \mathrm{MLP}(|v|)$  also approximates  $q$  within  $\epsilon/(2B)$  accuracy.
Thus, for all  $v \in \mathcal{X}$ , we have that