SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {s g n} \left(\sum_ {j = 1} ^ {M} o _ {j} \operatorname {s g n} \left(\left\langle w _ {j}, x \right\rangle - b _ {j}\right)\right) = \operatorname {s g n} \left(\sum_ {j = 1} ^ {M} o _ {j} ^ {\prime} \operatorname {s g n} \left(\left\langle w _ {j} ^ {\prime}, x \right\rangle - b _ {j}\right)\right), \forall x \in \mathcal {X}.
$$
Further, note that we can also multiply the output weights by the constant  $\sum_{j=1}^{M} o_j'$  without changing the sign and thus the classification. Overall, we can define a network  $N^{\star}$  equivalent to  $N$  in classification with
$$
N ^ {\star} (x) := \sum_ {j = 1} ^ {M} \frac {1}{\sum_ {j = 1} ^ {M} o _ {j} ^ {\prime}} o _ {j} ^ {\prime} \operatorname {s g n} (\langle w _ {j} ^ {\prime}, x \rangle - b _ {j}).
$$
Now, note that the limiting classifier can be written as  $\mathcal{F}_{\theta}(x) = \mathrm{sgn}(\mathbb{E}_{\mathbf{a}_{\theta},\mathbf{b}_{\theta}}\mathrm{sgn}(\langle \mathbf{a}_{\theta},x\rangle = \mathbf{b}_{\theta}))$  . We can then define its distribution as one that samples from the weights and bias  $w_{j}^{\prime}$ $b_{j}$  with probabilities according to  $\frac{1}{\sum_{j = 1}^{M}o_{j}^{\prime}} o_{j}^{\prime}$  . This implies that  $\mathcal{F}_{\theta}(x) = N^{\star}(x) = N(x)$  , making our  $\mathcal{F}_{\theta}$  achieve zero-error in the task. Note that if  $p^\dagger \coloneqq M$  is the number of parameters in  $N$  , we can generate the linear coefficients in  $\mathcal{F}_{\theta}(x)$  with at most  $p^\dagger +O(1)$  -simply take the external noise from a uniform distribution and map it to the probabilities in the output layer to pick the coefficients.
Finally, note that we have to also characterize the probability that  $\mathbf{y}_{\theta}^{(m)}$  outputs a different answer from  $\mathcal{F}_{\theta}$ . The common tool for this is Hoeffding's inequality, which lets us upper-bound it for some  $x \in \mathcal{X}$  with
$$
P \left(\mathbf {y} _ {\theta} ^ {(m)} \neq \mathcal {F} _ {\theta} (x)\right) \leq \exp \left\{- 2 \varepsilon_ {x} ^ {2} m ^ {2} \right\},
$$
where  $\varepsilon_{x} = P\big(\mathbf{y}_{\theta}^{(m)} = \mathcal{F}_{\theta}(x)\big) - 0.5$ . Note that  $\varepsilon_{x}$  depends on  $x$  and our definition of the minimum bias  $\epsilon$  in the main text is precisely the lowest of all such  $\varepsilon_{x}$ . Thus, as stated we finally have that for all  $x \in \mathcal{X}$
$$
P \left(\mathbf {y} _ {\theta} ^ {(m)} = \mathcal {F} _ {\theta} (x)\right) > 1 - \exp \{- 2 \varepsilon^ {2} m ^ {2} \}.
$$
Note that the above is true for any  $\mathcal{F}_{\theta}$ , including the universal and invariant ones later discussed in the main text.
Further taking the usual assumption that  $\mathcal{X}$  is compact.
Proof of Theorem 2. Let us start with Proposition 3, a central observation needed in Theorem 2. Put into words Proposition 3 says that we can transfer the action of  $G$  to the weights of the linear classifier. For the reader more familiar with group representation theory, the result follows immediately from noting that compact groups admit unitary representations, see [28] for a good resource on the matter.
Proposition 3. If  $G$  is a compact group and  $g \cdot x$  is an action of  $g \in G$  on  $x \in \mathbb{R}^d$ , there exists a bijective mapping  $\alpha \colon G \to G$  such that
$$
\langle g \cdot x, w \rangle = \langle x, \alpha (g) \cdot w \rangle , \forall x, w \in \mathbb {R} ^ {d}, \forall g \in G.
$$
Proof. Since we are dealing with actions on a finite-dimensional vector space, the action  $\cdot$  of  $g$  can be associated with a linear representation  $\rho$  of  $G$ , i.e.,  $\rho(g) \in \mathbb{R}^{d \times d}$ . That is, if  $[x]$  is the column vector of  $x$ , we have that  $\rho(g)[x] = g \cdot x$ . Now, since  $G$  is compact and  $\mathbb{R}^d$  is finite-dimensional,  $G$  also admits a unitary linear representation  $\rho_u$  with associated action  $\cdot_u$ . Then, it is easy to see that
$$
\langle g \cdot_ {u} x, w \rangle = \langle \rho_ {u} (g) [ x ], w \rangle = \langle x, \rho_ {u} (g) ^ {- 1} [ w ] \rangle = \langle x, g ^ {- 1} \cdot_ {u} [ w ] \rangle , \forall w \in \mathbb {R} ^ {d}.
$$
Thus, since inversion defines a bijection on  $G$ , Proposition 3 holds for the unitary representation action. Now, note that—as any other group action— $\cdot_u$  defines a homomorphism between  $G$  and the symmetric group of  $\mathbb{R}^d$ . Then, for any other action  $\cdot$  there exists a bijective function  $\beta \colon G \to G$  such that  $g \cdot x = \beta(g) \cdot_u x, \forall x \in \mathbb{R}^d$ . Hence, we can write
$$
\langle g \cdot x, w \rangle = \left\langle \beta (g) \cdot_ {u} x, w \right\rangle = \left\langle \beta (g) ^ {- 1} \cdot_ {u} x, w \right\rangle = \left\langle \beta^ {- 1} (\beta (g) ^ {- 1}) \cdot x, w \right\rangle , \forall x, w \in \mathbb {R} ^ {d}, \forall g \in G.
$$
Finally, since  $\beta$  and group inversion are bijective, we finish the proof by defining the bijective mapping  $\alpha(g) \coloneqq \beta^{-1}(\beta(g)^{-1})$  for every  $g \in G$ .
Now, we can proceed to prove the universality part of Theorem 2. Since the task admits a smooth separator, there exists a universal limiting classifier  $\mathcal{F}_{\theta}$  for it. That is, for every  $x \in \mathcal{X}$
$$
\mathcal {F} _ {\theta} (x) = y (x).
$$
Since the task is  $G$ -invariant, we know that for every  $g \in G$
$$
\mathcal {F} _ {\theta} (x) = y (g \cdot x).
$$
Note that since  $G$  is compact, it admits a unique normalized Haar measure  $\lambda^4$ . Then, the insight comes from considering a random group action  $\mathbf{g} \sim \lambda$ . By Fubini's theorem and Proposition 3, we have
$$
\mathcal {F} _ {\theta} (x) = \operatorname {E} _ {\mathbf {g}} \left[ \mathcal {F} _ {\theta} (\mathbf {g} \cdot x) \right] = \operatorname {s g n} \left(\operatorname {E} _ {\mathbf {g}, \mathbf {a} _ {\theta}, \mathbf {b} _ {\theta}} \left[ \operatorname {s g n} \left(\langle x, \alpha (\mathbf {g}) \cdot \mathbf {a} _ {\theta} \rangle - \mathbf {b} _ {\theta}\right) \right]\right).
$$
Now, we can define a limiting classifier  $\mathcal{F}_{\theta^{\star}}$  that samples coefficients according to  $\mathbf{b}_{\theta^{\star}} \stackrel{d}{=} \mathbf{b}_{\theta}$  and  $\mathbf{a}_{\theta^{\star}} \stackrel{d}{=} \mathbf{g} \cdot \mathbf{a}_{\theta}$ . From above, we know that  $\mathcal{F}_{\theta^{\star}}(x) = \mathcal{F}_{\theta}(x), \forall x \in \mathcal{X}$ . Since  $\mathcal{F}_{\theta}$  is universal,  $\mathcal{F}_{\theta^{\star}}$  is also universal. Finally, since  $\mathbf{g}$  is sampled according to the Haar measure, we have that
$$
\mathbf {a} _ {\theta^ {*}} \stackrel {{d}} {{=}} g \cdot \mathbf {a} _ {\theta^ {*}}, \forall g \in G.
$$
At last, it is easy to see from Proposition 3 that the action on the weights can be transferred to the input as well, meaning that every model is probabilistic invariant, i.e.,  $\mathcal{F}_{\theta}(x) = \mathcal{F}_{\theta}(g\cdot x),\forall x\in \mathcal{X},\forall g\in G,\forall \theta \in \mathbb{R}^{p}$ , which finalizes the proof.
Proposition 1 (Infinitely  $G$ -invariant RLCs). Let  $(x,y)$  be a  $G$ -invariant task with infinitely  $G$ -invariant data (Assumption 3) with a smooth separator as in Assumption 1. Then, the set of RLCs with an infinitely  $G$ -invariant distribution in the linear classifier weights, i.e., as in Assumption 3  $(a_{\theta i})_{i=1}^{\infty} \stackrel{d}{=} g_{\infty} \cdot (a_{\theta i})_{i=1}^{\infty}, \forall g_{\infty} \in G_{\infty}$ , where  $a_{\theta} \stackrel{d}{=} (a_{\theta i})_{i \in S}$ , for  $S \in \binom{\mathbb{N}}{d}$ , and absolutely continuous randomness source (cf. Assumption 2) is probabilistic  $G$ -invariant and universal for  $(x,y)$  as in Theorem 2.
4The reader can think of  $\lambda$  as a uniform distribution over  $G$
Proof of Proposition 1. From Proposition 3 it also follows that the model is  $G$ -invariant. Let us proceed to prove it is universal. Since the task admits a smooth separator, from Theorem 1 there exists an RLC  $\mathcal{F}_{\theta}$  such that
$$
\mathbb {E} _ {\mathbf {x}} \left[ \mathcal {F} _ {\theta} (\mathbf {x}) - y (\mathbf {x}) \right] = 0.
$$
Since the data is infinitely  $G$ -invariant, we have that for the infinite  $G$ -invariant sequence  $(\mathbf{x}_i)_{i=1}^{\infty}$
$$
\mathbb {E} _ {\mathbf {x}} [ \mathcal {F} _ {\theta} (\mathbf {x}) - y (\mathbf {x}) ] = \mathbb {E} _ {(\mathbf {x} _ {i}) _ {i \in S}} [ \mathcal {F} _ {\theta} ((\mathbf {x} _ {i}) _ {i \in S}) - y ((\mathbf {x} _ {i}) _ {i \in S}) ] = 0, \text {f o r} S \in \left( \begin{array}{c} \mathbb {N} \\ d \end{array} \right),
$$
with  $\binom{\mathbb{N}}{d}$  being the set of all  $d$ -size subsets of  $\mathbb{N}$ . Let  $\lambda$  be the unique normalized Haar measure of  $G$ . As in the proof of Theorem 2, we can take a random group action  $\mathbf{g} \sim \lambda$  and have from Proposition 3 and Fubini's theorem that
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x} _ {i}) _ {i \in S}} [ \mathcal {F} _ {\theta} ((\mathbf {x} _ {i}) _ {i \in S}) - y ((\mathbf {x} _ {i}) _ {i \in S}) ] = \mathbb {E} _ {((\mathbf {x} _ {i}) _ {i \in S}) _ {i \in S}, \mathbf {g}} [ \mathcal {F} _ {\theta} (\mathbf {g} \cdot ((\mathbf {x} _ {i}) _ {i \in S})) ] \\ = 0, \text {f o r} S \in \left( \begin{array}{c} \mathbb {N} \\ d \end{array} \right). \\ \end{array}
$$
Now, let us define a task  $(\mathbf{x}', y)$  on a higher dimension  $d' \geq d$ ,  $\operatorname{supp}(\mathbf{x}) \subset \operatorname{supp}(\mathbf{x}') \subseteq \mathbb{R}^{d'}$  where  $y(\mathbf{x}') = y(\mathbf{x}'_{[1:d]})$ . Further,  $\mathbf{g}'$  is a random group action (sampled from the Haar measure) from  $G'$ , the homomorphism of  $G$  into the dimension  $d'$ . From above, together with Proposition 3 we have that
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x} _ {i}) _ {i \in S ^ {\prime}}} [ \mathcal {F} _ {\theta} ((\mathbf {x} _ {i}) _ {i \in S ^ {\prime}}) - y ((\mathbf {x} _ {i}) _ {i \in S ^ {\prime}}) ] = \mathbb {E} _ {((\mathbf {x} _ {i}) _ {i \in S ^ {\prime}}) _ {i \in S ^ {\prime}}, \mathbf {g} ^ {\prime}} [ \mathcal {F} _ {\theta} (\mathbf {g} ^ {\prime} \cdot ((\mathbf {x} _ {i}) _ {i \in S ^ {\prime}})) ] \\ = \mathbb {E} _ {\left((\mathbf {x} _ {i}) _ {i \in S ^ {\prime}}\right) _ {i \in S ^ {\prime}}} \left[ \operatorname {s g n} \left(\mathbb {E} _ {\mathbf {a} _ {\theta}, \mathbf {b} _ {\theta}, \mathbf {g} ^ {\prime}} \left[ \operatorname {s g n} \left(\langle x, \alpha (\mathbf {g} ^ {\prime}) \cdot \mathbf {a} _ {\theta} ^ {\prime} \rangle - \mathbf {b} _ {\theta} ^ {\prime}\right) \right]\right) \right] \\ = 0, \text {f o r} S ^ {\prime} \in \left( \begin{array}{c} \mathbb {N} \backslash \{1, \ldots , d \} \\ d ^ {\prime} \end{array} \right). \\ \end{array}
$$
Now, from above, since the lower dimensions  $\{1,\dots ,n\}$  are  $G$  -invariant if all higher dimensions are  $G$  -invariant, it is easy to see that there is an infinitely  $G$  -invariant sequence of weights  $(\mathbf{a}_{\theta_i})_{i = 1}^{\infty}$  that achieves zero error.
Proposition 2 (Universality and resource consumption of RSetCs). Let  $(x,y)$  be a permutation-invariant task with a smooth separator as in Assumption 1 and infinitely permutation-invariant data as in Assumption 3. Then, RSetCs as in Definition 1 with absolutely continuous randomness source (cf. Assumption 2) are probabilistic  $G$ -invariant and universal for  $(x,y)$  (as in Theorem 2). Further, the number of parameters needed by RSetCs in this task will depend only on the smallest finite absolute moments of the weight and bias distributions in the zero-error solution, i.e., the ones given by  $f_{\theta_f^*}(\pmb{u},\pmb{u}_i)$  and  $g_{\theta_g^*}(\pmb{u},\pmb{u}_b)$ .
Proof of Proposition 2. The result follows directly from the combination of de Finetti's theorem [9], Kallenberg's noise transfer theorem [16] and the recent results on the capacity of neural networks to generate distributions from noise [34]. Let us outline it in more detail. From de Finetti's theorem [9] we know that any infinite sequence of exchangeable random variables can be expressed by i.i.d. random variables conditioned on a common latent measure. Combining this with Kallenberg's noise transfer theorem we have that the weights and biases can be written as  $f(\mathbf{u}, \mathbf{u}_i)$  and  $g(\mathbf{u}, \mathbf{u}_b)$  where  $f, g$  are measurable maps and the noises are sampled from a uniform distribution. Finally, the work of [34] showed that we can replace the uniform noise with absolutely continuous noise and  $f, g$  with a sufficiently expressive ReLU multi-layer perceptron. Moreover, Theorem 2.1 in [34] says that the number of parameters depends only on the output dimension and the smallest finite absolute moments of the distributions. Since the output dimension  $k$  is constant for us, it depends solely on the latter and we finalize the proof.
Theorem 3. Let  $(x,y)$  be a graph isomorphism-invariant task that either i) has a smooth separator as in Assumption 1 or ii) is an inner-product decision graph problem as in Definition 3. Further, the task has infinitely graph isomorphism-invariant data as in Assumption 3. Then, there exists an RGraphC as in Definition 2 with absolutely continuous randomness source (cf. Assumption 2) that is probabilistic  $G$ -invariant and universal for  $(x,y)$  (as in Theorem 2). Further, the number of parameters of this RGraphC will depend only on the smallest finite absolute moments of its weight and bias distributions, i.e., the ones given by  $f_{\theta_f}(\pmb{u},\pmb{u}_i)$  and  $g_{\theta_g}(\pmb{u},\pmb{u}_b)$ .
Proof of Theorem 3. Note that it suffices to show that there exists an RLC that takes as input graphs of size  $d$  and decides correctly tasks in Definition 3 with probability  $> 0.5$  —this implies a universal limiting classifier
$\mathcal{F}_{\theta}$ . Then, we can replace the smoothness assumption in Theorem 2 and Proposition 1 with Definition 3 and we will have that there exists an infinitely jointly exchangeable sequence that is universal for tasks as in Definition 3. Finally, we follow Proposition 2's proof by simply replacing de Finetti's with Aldous-Hoover's theorem. Thus, let us now proceed to prove the main part of this theorem, i.e., there exists an RLC that takes as input graphs of size  $d$  and decides correctly tasks in Definition 3 with probability  $>0.5$ .
Define an RLC that samples the linear coefficients as follows.
- Let  $\mathbf{a}_{\theta}^{\prime}$  be a random variable such that  $\mathrm{supp}(\mathbf{a}_{\theta}^{\prime}) = S$ . It is important to note that we do not require that this random vector to have the same distribution as the input, or any other distribution on  $S$ . We simply need it to be supported on  $S$ .
- Now, let  $t \sim \operatorname{Bernoulli}(0.5 + \gamma)$  for some  $\gamma > 0$ . We define  $\mathbf{a}_{\theta} := t \cdot \mathbf{a}_{\theta}'$  and  $\mathbf{b}_{\theta} := t \cdot b$ . Our random prediction can then be rewritten as  $\operatorname{sgn}(t \cdot (\langle \mathbf{a}_{\theta}', x \rangle - b))$ .
Let us now calculate our probability of success. For convenience, let us define  $\mathrm{sgn}(0) = 1^5$ . Then, the probability of a positive graph  $x^{+}$ , i.e.,  $y(x^{+}) = 1$ , being classified correctly is
$$
\begin{array}{l} P \left(\operatorname {s g n} \left(\left\langle \mathbf {a} _ {\theta}, x ^ {+} \right\rangle - \mathbf {b} _ {\theta}\right) = 1\right) = P \left(\operatorname {s g n} \left(t \cdot \left(\left\langle \mathbf {a} _ {\theta} ^ {\prime}, x ^ {+} \right\rangle - b\right)\right) = 1\right) \\ = 0. 5 - \gamma + P \left(\left\langle \mathbf {a} _ {\theta} ^ {\prime}, x ^ {+} \right\rangle \geq b\right) \cdot (0. 5 + \gamma), \\ \end{array}
$$
while the probability of a negative graph  $x^{-}$ , i.e.,  $y(x^{-}) = -1$ , being classified correctly is
$$
P \left(\operatorname {s g n} \left(\left\langle \mathbf {a} _ {\theta}, x ^ {-} \right\rangle - \mathbf {b} _ {\theta}\right) = 1\right) = P \left(\operatorname {s g n} \left(t \cdot \left(\left\langle \mathbf {a} _ {\theta} ^ {\prime}, x ^ {-} \right\rangle - b\right)\right) = 1\right) = 0. 5 + \gamma .
$$
Then, since  $\mathcal{X}$  is a finite set for any input distribution  $\mathbf{x}$  there exists a constant  $\gamma$  where  $0 < \gamma /(0.5 + \gamma) < P(\langle \mathbf{a}_{\theta}^{\prime},x\rangle \geq b)$  for all  $x\in \mathcal{X}$ . Given such gamma, every input has a probability of success greater than 0.5, which implies that  $\mathcal{F}_{\theta}(x) = y(x),\forall x\in \mathcal{X}$  as we wanted to show.
B RLCs for spherical data
Here we want to show how the same ideas presented in Sections 3.1 and 3.2 can be applied to spherical data using Freedman's theorem [13]. More specifically, we let our input be  $d$ -dimensional vectors, i.e.,  $\mathrm{supp}(\mathbf{x}) \subseteq \mathbb{R}^d$  and our task invariant to the orthogonal group, i.e.,  $y(x) = y(g \cdot x), \forall x \in \mathrm{supp}(\mathbf{x}), \forall g \in G$ , where  $G \coloneqq O(d)$ . We refer to tasks like this as tasks over spherical data since it is invariant to rotations and reflections.
Just as de Finetti, Aldous and Hoover informed us about sets and graphs, Freedman did it for spherical data [13]. It follows from his work that an infinite sequence invariant to the action of the orthogonal group can be represented as a random scalar multiplying a sequence of i.i.d. standard Gaussian distributions. Leveraging this, we can define the following very simple invariant model for spherical data (assuming Assumption 3).
Definition 4 (Randomized Sphere Classifiers). A Randomized Sphere Classifier (RSphereC) uses 2 neural networks  $f_{\theta_f} \colon \mathbb{R}^2 \to \mathbb{R}$  and  $g_{\theta_g} \colon \mathbb{R}^2 \to \mathbb{R}$  together with 3 sources of randomness:  $\mathbf{u}, \mathbf{u}_a$  and  $\mathbf{u}_b$ . The random linear classifier coefficients are generated with
$$
\boldsymbol {a} _ {\theta} \stackrel {{d}} {{=}} f _ {\theta_ {f}} (\boldsymbol {u}, \boldsymbol {u} _ {a}) \cdot [ \mathcal {N} (0, 1), \dots \mathcal {N} (0, 1) ], a n d \boldsymbol {b} _ {\theta} \stackrel {{d}} {{=}} g _ {\theta_ {g}} (\boldsymbol {u}, \boldsymbol {u} _ {b}),
$$
where  $[\mathcal{N}(0,1),\dots \mathcal{N}(0,1)]$  is a vector of d i.i.d. standard Gaussians.
Proposition 4 (Universality and resource consumption of RSphereCs). Let  $(\pmb{x},\pmb{y})$  be a  $O(d)$ -invariant task with a smooth separator as in Assumption 1 and infinitely spherical data as in Assumption 3. Then, RSphereCs as in Definition 4 with absolutely continuous randomness source (cf. Assumption 2) are probabilistic  $G$ -invariant and universal for  $(\pmb{x},\pmb{y})$  (as in Theorem 2). Further, the number of parameters needed by RSphereCs in this task will depend only on the smallest finite absolute moments of the weight and bias distributions in the zero-error solution, i.e., the ones given by  $f_{\theta_f^*}(\pmb{u},\pmb{u}_i)$  and  $g_{\theta_g^*}(\pmb{u},\pmb{u}_b)$ .
Proof. The proof is exactly as the one from Proposition 2 but replacing de Finetti's theorem with Freedman's [13]. For a clean and more accessible material on Freedman's theorem, we refer the reader to the lectures by Kallenberg in https://mysite.science.uottawa.ca/givanoff/wskallenberg.pdf.
Finally, we note that tasks invariant to  $O(n)$  are not that common. However, tasks invariant to  $\mathrm{SO}(n)$  are extremely relevant in computer vision, see [6]. Therefore, we think this model is a first step towards probabilistic invariance for  $\mathrm{SO}(n)^6$ .
5This can be done by always adding a sufficiently small constant to the input.
Note that  $\mathrm{SO}(n)$  is the connected component of  $O(n)$ .
As mentioned in the main text, all models were trained for a maximum of 1000 epochs using a cosine annealing scheduler for the learning. We tuned all the hyperparameters on the validation set using a patience of 30. The Deep Sets models found a better learning rate of 0.001 and batch size of 250. The GNN model found a better learning rate of 0.01 and batch size 100. The RSetC model used a batch size of 250 and learning rate 0.5. The RGraphC model used a batch size of 100 and a learning rate of 0.5. Finally, we also note that RLCs can out put different answers given the same weights (due to the random input). Thus, we amplify each mini-batch with 1000 samples for each model—we noted that this reduces the variance and helps convergence. The amplification size used in test was  $m = 10$ . Finally, for all RLCs we used a single dimensional standard normal distribution in each of their noise sources.
Source code is available at: https://github.com/cottascience/invariant-rlcs