SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 8 offers an alternative approach to dealing with sign ambiguity in addition to Algorithm 1, though it cannot generalize to the basis ambiguity case. The time complexity of Algorithm 8 is  $\mathcal{O}(n^2\log n)$ .
K Proofs
K.1 Proof of Theorem 1
We first prove that  $U\Lambda^{\frac{1}{2}}$  is a real-valued matrix.
Lemma 1. Suppose  $\hat{A}$  is the normalized adjacency matrix of a graph  $\mathcal{G}$ , and  $\hat{A} = U\Lambda U^{\top}$  is its spectral decomposition. Then,  $U\Lambda^{\frac{1}{2}} \in \mathbb{R}^{n \times n}$ .
Proof. Let  $\pmb{\Lambda} = \mathrm{diag}(\pmb{\lambda})$ . It suffices to show that  $\lambda_{i}\geq 0$  for  $i = 1,2,\ldots ,n$
Let  $\mathcal{G} = (\mathbb{V},\mathbb{E})$ , where  $\mathbb{V}$  is the vertex set and  $\mathbb{E}$  is the edge set. For node  $i$ , we denote the degree of node  $i$  by  $d_{i}$ . For any  $\pmb{x}\in \mathbb{R}^n$ , we have
$$
\boldsymbol {x} ^ {\top} \hat {\boldsymbol {A}} \boldsymbol {x} = \boldsymbol {x} ^ {\top} (\boldsymbol {I} + \tilde {\boldsymbol {A}}) \boldsymbol {x} = \sum_ {i \in \mathbb {V}} x _ {i} ^ {2} + \sum_ {(i, j) \in \mathbb {E}} \frac {2 x _ {i} x _ {j}}{\sqrt {d _ {i} d _ {j}}} = \sum_ {(i, j) \in \mathbb {E}} \left(\frac {x _ {i}}{\sqrt {d _ {i}}} + \frac {x _ {j}}{\sqrt {d _ {j}}}\right) ^ {2} \geq 0,
$$
thus the Rayleigh quotient of  $\hat{A}$  is bounded by  $\frac{\boldsymbol{x}^{\top}\hat{A}\boldsymbol{x}}{\boldsymbol{x}^{\top}\boldsymbol{x}}\geq 0$ . The Rayleigh quotient gives the lower bound of eigenvalues of  $\hat{A}$ , therefore we have  $\lambda_{i}\geq 0$ , and this completes the proof.
Then we give the proof of Theorem 1.
Proof. We can rewrite  $f$  such that it is a continuous set invariant function on the set consisting of the rows of its input:
$$
f ([ \boldsymbol {X}, \hat {\boldsymbol {A}} ]) = f ([ \boldsymbol {X}, \boldsymbol {U} \boldsymbol {\Lambda} \boldsymbol {U} ^ {\top} ]) = f \left(\left[ \boldsymbol {X}, (\boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}) (\boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}) ^ {\top} \right]\right) = F ([ \boldsymbol {X}, \boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}} ]).
$$
Using the permutation invariance property of  $f$ , we can verify that  $F$  is set invariant by observing:
$$
\begin{array}{l} F ([ \boldsymbol {P} \boldsymbol {X}, \boldsymbol {P} \boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}} ]) = f \left(\left[ \boldsymbol {P} \boldsymbol {X}, (\boldsymbol {P} \boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}) (\boldsymbol {P} \boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}}) ^ {\top} \right]\right) \\ = f ([ \boldsymbol {P} \boldsymbol {X}, \boldsymbol {P} \hat {\boldsymbol {A}} \boldsymbol {P} ^ {\top} ]) = f ([ \boldsymbol {X}, \hat {\boldsymbol {A}} ]) = F ([ \boldsymbol {X}, \boldsymbol {U} \boldsymbol {\Lambda} ^ {\frac {1}{2}} ]). \\ \end{array}
$$
Thus a universal network on sets can approximate  $F$  to an arbitrary precision.
K.2 Proof of Theorem 2
Proof. On the one hand, if  $f(hx) = gf(x)$  holds for some  $h \in H$  and  $g \in G$ , then we have  $\mathcal{A}\big(f(hx)\big) = h\mathcal{A}\big(f(x)\big)$  by the equivariance property of  $\mathcal{A}$  and  $\mathcal{A}\big(gf(x)\big) = \mathcal{A}\big(f(x)\big)$  by the invariance property of  $\mathcal{A}$ . However, since  $\mathcal{A}$  is a mapping and  $f(hx) = gf(x)$ , there must be  $\mathcal{A}\big(f(hx)\big) = \mathcal{A}\big(gf(x)\big)$  and thus  $\mathcal{A}\big(f(x)\big) = h\mathcal{A}\big(f(x)\big)$ . Now we have  $x \neq hx$  and  $\mathcal{A}\big(f(x)\big) = h\mathcal{A}\big(f(x)\big)$ , contradicting the universality property of  $\mathcal{A}$ .
On the other hand, if there does not exist  $h \in H$  and  $g \in G$  such that  $x \neq hx$  and  $f(hx) = gf(x)$ , we can construct a canonization of  $x$  as follows. First arbitrarily choose  $y_0$  such that  $f(x) = y_0$ , and let  $A\big(f(x)\big) = y_0$ . For any  $h \in H$  such that  $x \neq hx$ , by the equivariance of  $f$  we know that  $f(hx) = hy_0$ , thus we let  $A\big(f(hx)\big) = hy_0$ . Since  $f(hx) \neq gf(x)$  for any  $g$ , we know that  $y_0 \neq hy_0$ , thus the universality property of  $A$  holds.  $A$  is also invariant, since  $A\big(f(x)\big)$  is uniquely determined; and equivariant, since  $A\big(f(hx)\big) = hy_0 = hA\big(f(x)\big)$  for any  $h$ . We can repeat this process for all equivalence classes in  $\mathcal{X}$  and obtain an invariant, equivariant and universal canonization for all inputs.
K.3 Proof of Corollary 1
Proof. Under sign ambiguity, we have  $H = S_{n}$  and  $G = \{+1, -1\}$ . By Theorem 2,  $\mathbf{u}$  is canonizable if and only if there does not exist a permutation  $\sigma$  such that  $\mathbf{u} \neq \sigma(\mathbf{u})$  and  $\mathbf{u} = \pm \sigma(\mathbf{u})$ . This is equivalent to say that there does not exist a permutation matrix  $P \in \mathbb{R}^{n \times n}$  such that  $\mathbf{u} = -P\mathbf{u}$ .
K.4 Proof of Corollary 2
Proof. Under basis ambiguity, we have  $H = S_{n}$  and  $G = O(d)$ . By Theorem 2,  $U$  is canonizable if and only if there does not exist a permutation  $\sigma$  and an orthonormal matrix  $Q \in O(d)$  such that  $U \neq \sigma(U)$  and  $U = \sigma(U)Q$ . This is equivalent to say that there does not exist a permutation matrix  $P \in \mathbb{R}^{n \times n}$  such that  $U \neq PU$  and  $U$  and  $PU$  span the same  $d$ -dimensional subspace  $V \subseteq \mathbb{R}^n$ . Note here we used a lemma from Lim et al. [29]: for any orthonormal bases  $V$  and  $W$  of the same subspace, there exists an orthogonal  $Q \in O(d)$  such that  $VQ = W$ .
K.5 Proof of Theorem 3
Proof. Without loss of generality, we can always assume that the angles  $\{\alpha_i\}$  are sorted (if they are not, we can simply rearrange  $e_i$  to make them sorted and proceed in the same way):
$$
\begin{array}{l} \left| \boldsymbol {P} e _ {1} \right| = \dots = \left| \boldsymbol {P} e _ {n _ {1}} \right| > \left| \boldsymbol {P} e _ {n _ {1} + 1} \right| = \dots = \left| \boldsymbol {P} e _ {n _ {1} + n _ {2}} \right| > \dots \\ > \left| P e _ {n _ {1} + \dots + n _ {k - 1} + 1} \right| = \dots = \left| P e _ {n _ {1} + \dots + n _ {k}} \right|, \\ \end{array}
$$
where  $k$  is the number of distinct lengths of  $Pe_i$ ,  $\sum_{i=1}^k n_i = n$ . Here we divide  $e_i$  into  $k$  groups according to the angles between them and the eigenspace, with each group sharing the same  $|Pe_i|$ . Define  $j = (1, 1, \dots, 1) \in \mathbb{R}^n$ , then  $x_i$  can be expressed as
$$
\boldsymbol {x} _ {i} = \boldsymbol {e} _ {n _ {1} + \dots + n _ {i - 1} + 1} + \dots + \boldsymbol {e} _ {n _ {1} + \dots + n _ {i}} + c \boldsymbol {j}, 1 \leq i \leq k.
$$
The sign of  $\pmb{u}_0$  is selected based on the sign of  $\pmb{u}^\top \pmb{x}_h (\neq 0)$ , which is sign-equivariant. No matter what the sign of  $\pmb{u}$  is, we will always choose the one that maximizes  $\pmb{u}^\top \pmb{x}_h$ , thus  $\pmb{u}_0$  is sign-invariant.
Suppose the entries of the input eigenvector  $\pmb{u}$  is permutated by  $\sigma \in S_{n}$ , where  $S_{n}$  is the permutation group of order  $n$ . Then we have  $u_{i}^{\prime} = u_{\sigma (i)}(1\leq i\leq n)$  and
$$
\begin{array}{l} \left. \left| u ^ {\prime} u ^ {\prime} ^ {\top} e _ {\sigma (1)} \right| = \dots = \left| u ^ {\prime} u ^ {\prime} ^ {\top} e _ {\sigma (n _ {1})} \right| > \left| u ^ {\prime} u ^ {\prime} ^ {\top} e _ {\sigma (n _ {1} + 1)} \right| = \dots = \left| u ^ {\prime} u ^ {\prime} ^ {\top} e _ {\sigma (n _ {1} + n _ {2})} \right| > \dots \right. \\ > \left| \boldsymbol {u} ^ {\prime} \boldsymbol {u} ^ {\prime} ^ {\top} \boldsymbol {e} _ {\sigma \left(n _ {1} + \dots + n _ {k - 1} + 1\right)} \right| = \dots = \left| \boldsymbol {u} ^ {\prime} \boldsymbol {u} ^ {\prime} ^ {\top} \boldsymbol {e} _ {\sigma \left(n _ {1} + \dots + n _ {k}\right)} \right|, \\ \end{array}
$$
thus  $(x_{i}^{\prime})_{i} = (x_{i})_{\sigma (i)}$ , i.e., the vectors  $\pmb{x}_i(1\leq i\leq k)$  are permutation-equivariant. Let  $\pmb{x}_h^\prime$  be defined as in Section 3.3.1 (the number  $h$  is permutation-invariant because each  $\pmb{u}^{\prime \top}\pmb{x}_{i}^{\prime}$  is permutation-invariant). The sign of  $\pmb{u}_{0}^{\prime}$  is determined by the sign of the dot product of  $\pmb{u}^{\prime}$  and  $\pmb{x}_h^\prime$ , both of which are permutation-equivariant. This indicates that the sign of  $\pmb{u}^{\prime \top}\pmb{x}_h^\prime$  (and thus the sign of  $\pmb{u}_0^\prime$ ) is
permutation-invariant (i.e., unique). Since  $\pmb{u}_0' = \pmb{u}'$  or  $\pmb{u}_0' = -\pmb{u}'$ ,  $\pmb{u}_0'$  is permutation-equivariant as well.
If there exists a permutation  $\sigma$  (acting on entries of  $\mathbf{u}$ ) such that  $\mathbf{u} \neq \sigma(\mathbf{u})$  but they have the same canonical form, then either  $\mathbf{u} = +\sigma(\mathbf{u})$  or  $\mathbf{u} = -\sigma(\mathbf{u})$ . The former is impossible since we already assumed they are not equal, and the latter violates Assumption 1, leading to a contradiction. Thus the canonization of  $\mathbf{u}$  is universal.
K.6 Proof of Theorem 4
where  $k$  is the number of distinct lengths of  $P e_{i}$ ,  $\sum_{i=1}^{k} n_{i} = n$ . Here we divide  $e_{i}$  into  $k$  groups according to the angles between them and the eigenspace, with each group sharing the same  $|P e_{i}|$ . Define  $j = (1, 1, \ldots, 1) \in \mathbb{R}^{n}$ , then  $x_{i}$  can be expressed as
Notice that  $|\pmb{Pe}_i| = |\pmb{u}\pmb{u}^\top \pmb{e}_i| = u_i|\pmb{u}|$ , thus
$$
\left| u _ {1} \right| = \dots = \left| u _ {n _ {1}} \right| > \dots > \left| u _ {n _ {1} + \dots + n _ {k - 1} + 1} \right| = \dots = \left| u _ {n _ {1} + \dots + n _ {k}} \right|.
$$
Suppose  $\pmb{u}$  violates Assumption 1. Then for any  $1\leq j\leq k$  , we have
$$
u _ {n _ {1} + \dots + n _ {j - 1} + 1} + \dots + u _ {n _ {1} + \dots + n _ {j}} = 0.
$$
Let  $\tilde{\pmb{u}}_j\coloneqq (u_{n_1 + \dots +n_{j - 1} + 1},\ldots ,u_{n_1 + \dots +n_j})^\top$ . The above equations show that (1) the absolute value of the entries of  $\tilde{\pmb{u}}_j$  are all equal; (2) the sum of the entries of  $\tilde{\pmb{u}}_j$  is 0. Thus for any entry of  $\tilde{\pmb{u}}_j$ , either it is 0 or its positive and negative value appears in pairs. In conclusion,  $+\tilde{\pmb{u}}_j$  and  $-\tilde{\pmb{u}}_j$  are equal up to a permutation for all  $j$ , thus  $+u$  and  $-u$  are equal up to a permutation. By Theorem 1, it is not canonizable.
On the other hand, suppose  $\pmb{u}$  satisfies Assumption 1. Then there exists  $1\leq j\leq k$  such that
$$
u _ {n _ {1} + \dots + n _ {j - 1} + 1} + \dots + u _ {n _ {1} + \dots + n _ {j}} \neq 0.
$$
Let  $\tilde{\pmb{u}}_j\coloneqq (u_{n_1 + \dots +n_{j - 1} + 1},\ldots ,u_{n_1 + \dots +n_j})^\top$  .  $\tilde{\pmb{u}}_j$  contains all entries of  $\pmb{u}$  with absolute value  $|u_{n_1 + \dots +n_j}|$  . Since the sum of  $\tilde{\pmb{u}}_j$  is non-zero, the numbers of positive and negative entries in  $\tilde{\pmb{u}}_j$  are different. No matter how we permute  $+u$  and  $-\pmb{u}$  , their corresponding entries in  $\tilde{\pmb{u}}_j$  will not align. Thus  $+u$  and  $-\pmb{u}$  are not equal up to any permutation. By Theorem 1, it is canonizable.
K.7 Proof of Theorem 5
We first point out that the orthogonal projector  $P = U U^{\top}$  is invariant to the choice of basis.
Lemma 2. Let  $\mathbf{U} = [\mathbf{u}_1, \ldots, \mathbf{u}_d] \in \mathbb{R}^{n \times d}$  and  $\mathbf{V} = [\mathbf{v}_1, \ldots, \mathbf{v}_d] \in \mathbb{R}^{n \times d}$  be two sets of orthonormal vectors that span the same  $d$ -dimensional subspace  $V \subseteq \mathbb{R}^n$ . Then  $\mathbf{U}\mathbf{U}^\top = \mathbf{V}\mathbf{V}^\top$ .
Proof. We have  $\boldsymbol{U}^{\top}\boldsymbol{U} = \boldsymbol{V}^{\top}\boldsymbol{V} = \boldsymbol{I}$  by the definition of  $\boldsymbol{U}$  and  $\boldsymbol{V}$ . Let  $\boldsymbol{U} = \boldsymbol{V}\boldsymbol{Q}$  where  $\boldsymbol{Q}$  is an invertible matrix, then
$$
\begin{array}{l} \boldsymbol {U} \boldsymbol {U} ^ {\top} = \boldsymbol {U} (\boldsymbol {U} ^ {\top} \boldsymbol {U}) ^ {- 1} \boldsymbol {U} ^ {\top} \\ = V Q \left(Q ^ {\top} V ^ {\top} V Q\right) ^ {- 1} Q ^ {\top} V ^ {\top} \\ = \boldsymbol {V} \boldsymbol {Q} \boldsymbol {Q} ^ {- 1} (\boldsymbol {V} ^ {\top} \boldsymbol {V}) ^ {- 1} (\boldsymbol {Q} ^ {\top}) ^ {- 1} \boldsymbol {Q} ^ {\top} \boldsymbol {V} ^ {\top} = \boldsymbol {V} \boldsymbol {V} ^ {\top}. \\ \end{array}
$$
Lemma 2 shows that  $\mathbf{Pe}_i$  (and thus  $\mathbf{x}_i$ ) is invariant to the choice of basis in  $V$ . If we permute  $e_1, \ldots, e_n$  by  $\sigma \in S_n$ , then the sorted sequence of  $|\mathbf{Pe}_i|$  is also permuted by  $\sigma$ . Thus the elements of each  $\mathbf{x}_i$  is permuted by  $\sigma$  as well. This shows the choice of  $\mathbf{x}_i$  is permutation-equivariant.
Then we prove Theorem 5.
Proof. Without loss of generality, we can always assume that the angles  $\{\alpha_{i}\}$  are sorted (if they are not, we can simply rearrange  $e_i$  to make them sorted and proceed in the same way):
where  $k$  is the number of distinct lengths of  $Pe_i$ ,  $\sum_{i=1}^{k} n_i = n$ . Here we divide  $e_i$  into  $k$  groups according to the angles between them and the eigenspace, with each group sharing the same  $|Pe_i|$ . Define  $j = (1, 1, \dots, 1) \in \mathbb{R}^n$ , then  $x_i$  can be expressed as
We have already shown the existence of the maximum value of  $f(\pmb{u})$ . To show that basis-invariance of  $U_0$ , it suffices to show the uniqueness of the maximum point of  $f(\pmb{u})$ . Thus no matter what basis of  $\pmb{U}$  is, Algorithm 2 always yields the same output.
Notice that  $f(-\pmb{u}) = -f(\pmb{u})$ , thus either the maximum value of  $f(\pmb{u})$  is positive, or  $f(\pmb{u}) = \pmb{u}^\top \pmb{x}_i \equiv 0$ . However,  $f(\pmb{u}) \equiv 0$  implies that  $\pmb{x}_i$  is perpendicular to  $\langle \pmb{u}_1, \dots, \pmb{u}_{i-1} \rangle^\perp$ , violating Assumption 3. Thus we conclude the maximum value of  $f(\pmb{u})$  is positive.
Suppose there exists  $\pmb{u}^{\prime}\neq \pmb{u}^{\prime \prime}$  such that  $f(\pmb {u}^{\prime}) = f(\pmb{u}^{\prime \prime})$  takes maximum value. Consider the vector  $\alpha \pmb {u}' + \alpha \pmb{u}''$  where  $\alpha = \sqrt{\frac{1}{2(1 + \pmb{u}'^\top\pmb{u}'')} > \frac{1}{2}}$ . Obviously,  $\alpha \pmb {u}' + \alpha \pmb{u}''\in \langle \pmb {u}_1,\dots ,\pmb{u}_{i - 1}\rangle^\perp$ ,  $|\alpha \pmb{u}^{\prime} + \alpha \pmb{u}^{\prime \prime}| = 1$ , and  $f(\alpha \pmb{u}^{\prime} + \alpha \pmb{u}^{\prime \prime}) = 2\alpha \pmb{u}^{\prime \top}\pmb{x}_i > \pmb{u}^{\prime \top}\pmb{x}_i$ . This leads to a contradiction. Therefore, the choice of  $\pmb{u}_i$  is unique.
The permutation-equivariance of Algorithm 2 can be proved by observing that each step of Algorithm 2 is permutation-equivariant. Since each  $\boldsymbol{x}_i$  is permutation-equivariant, its eigenprojection on the subspace  $V$  (and thus  $U_0$ ) is also permutation-equivariant.
If there exists a permutation  $\sigma$  (acting on rows of  $U$ ) such that  $U \neq \sigma(U)$  but they have the same canonical form, then  $U$  and  $\sigma(U)$  spans the same subspace. On the one hand,  $U \neq \sigma(U)$  means that at least one of the eigenvectors in  $U$  is not  $\sigma$ -invariant. On the other hand, since all  $u_1, \ldots, u_d$  are permutation-equivariant but unchanged after  $\sigma$ , they are all  $\sigma$ -invariant. This leads to a contradiction, since it is impossible to have a non- $\sigma$ -invariant eigenvector in a  $\sigma$ -invariant eigenspace.
K.8 Proof of Theorem 7
We first prove the following lemmas.
Lemma 3. Let  $\mathbf{R} \in \mathbb{R}^{n \times n}$  be a random matrix, and each entry of  $\mathbf{R}$  is sampled independently from the standard Gaussian distribution  $N(0,1)$ . Then with probability 1,  $\mathbf{R}$  has full rank.
Proof. We denote the first column of  $\mathbf{R}$  by  $\mathbf{R}_{:,1}$ . It is linearly independent because  $\mathbf{R}_{:,1} = \mathbf{0}$  with probability 1. Then we view  $\mathbf{R}_{:,1}$  as fixed, and consider the second column  $\mathbf{R}_{:,2}$ . The probability that  $\mathbf{R}_{:,2}$  falls into the span of  $\mathbf{R}_{:,1}$  is 0, thus with probability 1,  $\mathbf{R}_{:,1}$  and  $\mathbf{R}_{:,2}$  are linearly independent.
Generally, let us consider the  $k$ -th column  $\mathbf{R}_{\cdot ,k}$ . The first  $k - 1$  columns of  $\mathbf{R}$  forms a subspace in  $\mathbb{R}^n$  whose Lebesgue measure is 0. Thus  $\mathbf{R}_{\cdot ,k}$  falls into this subspace with probability 0. By inference, we have all the columns of  $\mathbf{R}$  are linearly independent with probability 1, i.e.,  $P(\mathrm{rank}(\mathbf{R}) = n) = 1$ .
Lemma 4. Let  $\mathbf{A} \in \mathbb{R}^{s \times n}$ ,  $\mathbf{B} \in \mathbb{R}^{s \times m}$  be two matrices. Then the equation  $\mathbf{AX} = \mathbf{B}$  has a solution iff.  $\operatorname{rank}(\mathbf{A}) = \operatorname{rank}([\mathbf{A}, \mathbf{B}])$ .
Proof. First we prove the necessity. Suppose  $AX = B$  has a solution. Then,
$$
[ \boldsymbol {A} _ {:; 1}, \boldsymbol {A} _ {:; 2}, \dots , \boldsymbol {A} _ {:; n} ] \boldsymbol {X} _ {:; i} = \boldsymbol {B} _ {:; i},
$$
where  $M_{\vdots,i}$  denotes the  $i$ -th column of matrix  $M$ . This means each column of  $B$  can be expressed as a linear combination of the columns of  $A$ , and therefore each column of  $[A,B]$  can be expressed as a linear combination of the columns of  $A$ .
On the other hand, it is obvious that each column of  $\mathbf{A}$  can be expressed as a linear combination of the columns of  $[A,B]$ . Thus we have  $\mathrm{rank}(\mathbf{A}) = \mathrm{rank}([\mathbf{A},\mathbf{B}])$ .
Then we prove the sufficiency. Since  $\mathrm{rank}(\mathbf{A}) = \mathrm{rank}([A,B])$ , and each column of  $\mathbf{A}$  can be expressed as a linear combination of columns of  $[A,B]$ , we have the columns of  $\mathbf{A}$  and the columns of  $[A,B]$  are equivalent. Therefore, each column of  $\mathbf{B}$  can be expressed as a linear combination of columns of  $\mathbf{A}$ , i.e.,  $Ax = B_{:,i}$  has a solution for  $i = 1,\dots,m$ . Thus the equation  $AX = B$  has a solution.
Then we give the proof of Theorem 7.
Proof. For any prediction  $Z \in \mathbb{R}^{n \times d'}$ , we wish to prove that with probability 1, there exists parameters of a linear GNN with RNI  $W \in \mathbb{R}^{d \times d'}$  such that
$$
[ \mathbf {X}, \mathbf {R} ] \boldsymbol {W} = \boldsymbol {Z}. \tag {7}
$$
By Lemma 4, the necessary and sufficient condition that Equation 7 has a solution  $\mathbf{W}$  is  $\mathrm{rank}([\mathbf{X},\mathbf{R}]) = \mathrm{rank}([\mathbf{X},\mathbf{R},\mathbf{Z}])$
By Lemma 3, with probability 1,  $\mathrm{rank}(\mathbf{R}) = n$ , therefore  $\mathrm{rank}([\mathbf{X},\mathbf{R}]) = \mathrm{rank}([\mathbf{X},\mathbf{R},\mathbf{Z}]) = n$ .
Thus, in conclusion, with probability 1, there exists parameters of a linear GNN with RNI  $\pmb{W} \in \mathbb{R}^{d \times d'}$  such that the GNN produces  $\pmb{Z}$ .
We can also prove linear GNNs' equivariance by observing that for any permutation matrix  $P \in \mathbb{R}^{n \times n}$ ,
$$
[ P X, \mathbf {R} ] W = P [ X, \mathbf {R} ] W = P Z,
$$
where we used  $\mathbf{PR} = \mathbf{R}$  because each entry of  $\mathbf{PR}$  is also sampled from the standard Gaussian matrix  $N(0,1)$ .
K.9 Proof of Theorem 14
The following lemmas are used in our proof.
Lemma 5 (Newton's Identities). Let  $x_{1}, x_{2}, \ldots, x_{n}$  be variables, denote for  $k \geq 1$  by  $P_{k}$  the  $k$ -th power sum:
$$
P _ {k} = \sum_ {i = 1} ^ {n} x _ {i} ^ {k} = x _ {1} ^ {k} + \dots + x _ {n} ^ {k},
$$
and for  $k \geq 0$  denote by  $e_k$  the elementary symmetric polynomial. Then we have
$$
P _ {k} = (- 1) ^ {k - 1} k e _ {k} + \sum_ {i = 1} ^ {k - 1} (- 1) ^ {k - 1 + i} e _ {k - i} P _ {i},
$$
for all  $n \geq 1$  and  $n \geq k \geq 1$ .
Lemma 6 (Vieta's Formulas). Let  $f(x) = a_{n}x^{n} + a_{n - 1}x^{n - 1} + \dots +a_{1}x + a_{0}$  be a polynomial of degree  $n$ ,  $e_k$  be the elementary symmetric polynomial. Then we have
$$
e _ {1} = - \frac {a _ {n - 1}}{a _ {n}}, \quad e _ {2} = \frac {a _ {n - 2}}{a _ {n}}, \quad \ldots , \quad e _ {n} = (- 1) ^ {n} \frac {a _ {0}}{a _ {n}}.
$$
We first show that  $h$  exists.
Lemma 7. Let  $\mathbf{u} \in \mathbb{R}^n$ . Assume that there does not exist a permutation matrix  $\mathbf{P} \in \mathbb{R}^{n \times n}$  such that  $\mathbf{u} = -\mathbf{P}\mathbf{u}$ . Then there exists a positive odd integer  $h \leq n$  such that  $\sum_{i=1}^{n} u_i^h \neq 0$ .
Proof. Suppose the opposite holds, i.e.,  $P_{i} = 0$  for all odd  $1 \leq i \leq n$ , where  $P_{i}$  is the  $i$ -th power sum of entries of  $\mathbf{u}$ . Let  $u_{1}, u_{2}, \ldots, u_{n}$  be the roots of the polynomial  $f(x) = a_{n}x^{n} + a_{n - 1}x^{n - 1} + \dots + a_{1}x + a_{0}$ .
If  $n$  is even, by Lemma 5,
$$
0 = P _ {1} = e _ {1},
$$
$$
0 = P _ {3} = e _ {1} P _ {2} - e _ {2} P _ {1} + 3 e _ {3},
$$
$$
0 = P _ {5} = e _ {1} P _ {4} - e _ {2} P _ {3} + e _ {3} P _ {2} - e _ {4} P _ {1} + 5 e _ {5},
$$
$$
0 = P _ {n - 1} = e _ {1} P _ {n - 2} - e _ {2} P _ {n - 3} + \dots + (n - 1) e _ {n - 1},
$$
which gives us  $e_1 = e_3 = \dots = e_{n-1} = 0$ . Then, by Lemma 6, we have  $a_{n-1} = a_{n-3} = \dots = a_1 = 0$ . This indicates that  $f(x) = g(x^2)$  for some polynomial  $g(x)$ . Similarly, if  $n$  is odd, by Lemma 5,
$$
0 = P _ {n} = e _ {1} P _ {n - 1} - e _ {2} P _ {n - 2} + \dots + n e _ {n},
$$
which gives us  $e_1 = e_3 = \dots = e_n = 0$ . Then, by Lemma 6, we have  $a_{n-1} = a_{n-3} = \dots = a_0 = 0$ . This indicates that  $f(x) = xg(x^2)$  for some polynomial  $g(x)$ . Either way, all  $n$  roots of  $f(x)$  are symmetric with respect to the  $y$ -axis. Then there must exist a permutation matrix such that  $\mathbf{u} = -\mathbf{P}\mathbf{u}$ , leading to a contradiction. Thus Lemma 7 holds.
Then we prove Theorem 14.
Proof. By Lemma 7, we have shown the existence of  $h$ . Since flipping the sign of  $\mathbf{u}$  also flips the sign of  $\sum_{i=1}^{n} u_i^h$  (because  $h$  is odd), Algorithm 8 uniquely decides the sign of  $\mathbf{u}$ . Since the algorithm outputs either  $\mathbf{u}$  or  $-\mathbf{u}$ , it is also permutation-equivariant.
K.10 Proof of Theorem 15
We prove that under mild conditions, the loss of expressive power induced by truncating RSE can be upper bounded, as shown in the following theorem.
Theorem 15. Let  $\Omega \subset \mathbb{R}^{n\times d}\times \mathbb{R}^{n\times n}$  be a compact set of graphs,  $[\pmb {X},\hat{\pmb{A}} ]\in \Omega$  . Let NN be a universal neural network on sets. Given an invariant graph function  $f$  defined over  $\Omega$  that can be  $\varepsilon$  -approximated by an  $L_{p}$  -Lipschitz continuous function and arbitrary  $\varepsilon >0$  , for any integer  $0 < k\leq n$  , there exist parameters of NN such that for all graphs  $[\pmb {X},\hat{\pmb{A}} ]\in \Omega$
$$
\left| f ([ \pmb {X}, \hat {\pmb {A}} ]) - \mathrm {N N} ([ \pmb {X}, (\pmb {U} \pmb {\Lambda} ^ {\frac {1}{2}}) _ {: -, k:}, \pmb {0} ]) \right| <   \sqrt {n - k} L _ {p} \lambda_ {n - k} + \varepsilon .
$$
Here the  $L_{p}$ -Lipschitz continuity of  $f$  is defined using the Frobenius norm on the input domain,  $0 \leq \lambda_{1} \leq \dots \leq \lambda_{n} \leq 2$  are the eigenvalues of  $\hat{A}$ ,  $\mathbf{0} \in \mathbb{R}^{n \times (n - k)}$ .
We can see from Theorem 15 that the upper bound of the loss of expressive power decreases when  $k$  increases, and when  $k = n$ , the network becomes universal. We give its proof as follows.
By Lemma 1, we know that  $\lambda_{i}\geq 0$  for  $i = 1,2,\ldots ,n$ . Next we prove that  $\lambda_{i}\leq 2$ .
Lemma 8. Suppose  $\hat{\mathbf{A}}$  is the normalized adjacency matrix of a graph  $\mathcal{G}$ , and  $\lambda_1 < \dots < \lambda_n$  are its eigenvalues. Then  $\lambda_i \leq 2$ , for  $i = 1, 2, \ldots, n$ .
Proof. In the proof of Lemma 1, we proved  $\pmb{x}^{\top}(\pmb{I} + \tilde{\pmb{A}})\pmb{x} \geq 0$ . Similarly, we have