SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here we consider tasks that are invariant to permutations, i.e., our input is a set $^1$  of vectors. More formally, we let our input be supported on  $\mathcal{X} := (\mathbb{R}^k)^{d2}$  and  $G := \operatorname{Sym}([d])$  be the group of permutations of  $[d] := \{1, \ldots, d\}$ . Then, for an input  $x \in \mathcal{X}$  the action  $g \cdot x$  is given by permuting the  $d$  vectors of size  $k$  according to  $g$ .
The key insight to design RLCs for set data is leveraging the classic de Finetti's theorem [9]. In summary, de Finetti showed how any infinite sequence of exchangeable random variables can be expressed as an infinite sequence of i.i.d. random variables conditioned on a common latent measure. Now, when Assumption 3 is true, de Finetti tells us that we can sample our weights  $\mathbf{a}_{\theta}$  by first sampling a common noise and use it as input to the same function, which generates the linear weights together with an independent noise in an i.i.d. manner. Note that the bias does not have the exchangeability property. Thus, it needs to receive the same shared noise as the set but use a different function, since it is not necessarily sampled in the same way. Next, we formalize these notions by defining the class of Randomized Set Classifiers (RSetCs) and characterizing its universal approximation power together with resource consumption.
Definition 1 (Randomized Set Classifiers (RSetCs)). A Randomized Set Classifier (RSetC) uses two neural networks  $f_{\theta_f} : \mathbb{R}^2 \to \mathbb{R}^k$  and  $g_{\theta_g} : \mathbb{R}^2 \to \mathbb{R}$  together with  $d + 2$  sources of randomness:  $\pmb{u}$ ,  $(\pmb{u}_i)_{i=1}^d$ , and  $\pmb{u}_b$ . The random linear classifier coefficients are generated with
$$
\boldsymbol {a} _ {\theta_ {i}} ^ {(k)} \stackrel {{d}} {{=}} f _ {\theta_ {f}} (\boldsymbol {u}, \boldsymbol {u} _ {i}) a n d \boldsymbol {b} _ {\theta} \stackrel {{d}} {{=}} g _ {\theta_ {g}} (\boldsymbol {u}, \boldsymbol {u} _ {b}),
$$
where  $\pmb{a}_{\theta_i^{(k)}}$  is the  $i^{\text{th}}$  chunk of  $k$  random linear weights, i.e., the weights multiplying the  $i^{\text{th}}$  vector.
Proposition 2 (Universality and resource consumption of RSetCs). Let  $(\pmb{x},y)$  be a permutation-invariant task with a smooth separator as in Assumption 1 and infinitely permutation-invariant data as in Assumption 3. Then, RSetCs as in Definition 1 with absolutely continuous randomness source (cf. Assumption 2) are probabilistic  $G$ -invariant and universal for  $(\pmb{x},y)$  (as in Theorem 2). Further, the number of parameters needed by RSetCs in this task will depend only on the smallest finite absolute moments of the weight and bias distributions in the zero-error solution, i.e., the ones given by  $f_{\theta_f^*}(\pmb{u},\pmb{u}_i)$  and  $g_{\theta_g^*}(\pmb{u},\pmb{u}_b)$ .
The complete proof is in Appendix A. We highlight that Proposition 2 is combining de Finetti's theorem [9], Kallenberg's noise transfer theorem [16] and recent results on the capacity of neural networks to generate distributions from noise [34].
What is the practical impact of Assumption 3 in set tasks? Assumption 3 implies that an input set  $\mathbf{x}$  is generated by first sampling some latent variable  $\Theta \sim \nu$  which is then used to sample items in an i.i.d. manner  $\mathbf{x}_1, \mathbf{x}_2, \ldots \mid \sim_{\Theta \text{i.i.d.}} P_\Theta$ . We can make sense of this with an example in e-commerce systems. If the set represents items in a shopping cart, the latent factor  $\Theta$  can be thought of as a summary of the user's shopping behavior. Overall, this assumption is not appropriate if there could be mutually exclusive items in the set.
When are RSetCs better than Deep Sets? We contrast the results of RSetCs with Deep Sets [35] due to its universality and computational efficiency [32]. Overall, other deterministic set models
<sup>1</sup>If the support is discrete we shall consider multisetso instead.
2Note that this does not change the reference to  $d$  in previous results.
are either variations of Deep Sets that inherent the same properties [20, 24] or are computationally inefficient [23]. Apart from universality, it is important to note from Proposition 2 how the number of parameters used by the RSetC does not depend on the set size. Instead, its resource consumption is related to how smooth the distribution of linear classifiers weights and bias are. Since all the weights have the same distribution (conditioned on  $\mathbf{u}$ ), the set size is not relevant. This comes in contrast to Deep Sets, which as shown in [32] only achieves universality when the hidden-layer and the input set have the same size.
Moreover, Deep Sets is known to have poor set size generalization abilities, i.e., when the test set contains larger sets than the training set [31]. In RSetCs, although the number of parameters does not depend on the set size, the distribution of linear coefficients can use this information. Overall, the gain in robustness to set size might come from RSetCs performing simple linear transformations of the input. That is, for small changes in the set size, the output is not expected to change abruptly. This opposes to Deep Sets, where the representations of the items are aggregated and input to a neural network. Thus, any small addition to the set can arbitrarily change the output. Although we cannot always guarantee size generalization for RSetCs, we give empirical evidence in Section 4 that they do seem to be a lot more robust than Deep Sets.
3.2 RLCs for graph data
Here we consider tasks that take graphs as input. Given a (possibly weighted) graph  $G = (V, E)$  with  $d$  vertices and  $M$  edges, we take as input a vector  $x$  of size  $d^2$  representing the vectorized adjacency matrix of  $G$ . Our invariance is then to graph isomorphism. That is, we again have the permutation group  $G := \operatorname{Sym}([d])$ , but now with a different action. Here  $g \cdot x$  jointly permutes the rows and columns of the adjacency matrix corresponding to  $x$ . This way,  $x$  and  $g \cdot x$  represent isomorphic graphs. We will then assume that our graph tasks are  $G$ -invariant, i.e., two isomorphic graphs  $x, g \cdot x$  will always have the same label.
Now, we can leverage Aldous-Hoover's theorem, the analogous of de Finetti's theorem for joint exchangeability in 2-dimensional arrays. It gives us a model, invariance and universality results analogous to the ones for RSetCs.
Definition 2 (Randomized Graph Classifiers). A Randomized Graph Classifier (RGraphC) uses two neural networks  $f_{\theta_f} \colon \mathbb{R}^4 \to \mathbb{R}$  and  $g_{\theta_g} \colon \mathbb{R} \to \mathbb{R}$  together with  $d^2 + d + 1$  sources of randomness:  $\pmb{u}$ ,  $(\pmb{u}_i^{(n)})_{i=1}^d$ ,  $(\pmb{u}_{ij}^{(e)})_{i,j \in [d]}$ , and  $\pmb{u}_b$ . The random linear classifier coefficients are generated with
$$
\boldsymbol {a} _ {\theta i j} \stackrel {{d}} {{=}} f _ {\theta_ {f}} (\boldsymbol {u}, \boldsymbol {u} _ {i} ^ {(n)}, \boldsymbol {u} _ {j} ^ {(n)}, \boldsymbol {u} _ {i j} ^ {(e)}), a n d \boldsymbol {b} _ {\theta} \stackrel {{d}} {{=}} f _ {\theta_ {f}} (\boldsymbol {u}, \boldsymbol {u} _ {b}),
$$
where  $\mathbf{a}_{\theta ij}$  is the random linear weight multiplying the entry of  $x$  that corresponds to the entry in row  $i$  and column  $j$  of the graph's adjacency matrix.
Note that we can use the same proof of Proposition 2 and have an equivalent result for RGraphCs. However, since graphs are discrete objects the smooth separator assumption (cf. Assumption 1) might not be so easily satisfied. To address this, we define a sufficiently large class of graph models that RGraphCs can approximate.
Definition 3 (Inner-product decision graph problems). Let  $G = (V, E)$  be a graph over  $d$  vertices with vectorized adjacency matrix  $x \in \mathcal{X} := \operatorname{supp}(\boldsymbol{x}) \subseteq \mathbb{R}^{d^2}$ . We say that a graph property  $y: \mathcal{X} \to \{-1, 1\}$  is an inner product verifiable property if there exists a set of vectors  $S \subseteq \mathbb{R}^{d^2}$  and a constant  $b \in \mathbb{R}$  such that:
- For all graphs  $x^{+} \in \mathcal{X}$  satisfying the property  $y(x^{+}) = 1$ , there exists  $s \in S$ , such that  $\langle s, x^{+} \rangle \geq b$ ;
- For all graphs  $x^{-} \in \mathcal{X}$  not satisfying the property  $y(x^{-}) = -1$ , we have  $\langle x^{-}, s' \rangle < b, \forall s' \in S$ .
As an example of such property, consider connectivity. The set  $S$  above is defined to be the set of all binary vectors representing adjacencies of spanning trees. The threshold in this case would be  $d - 1$ . If  $G$  is connected, then there exists a spanning tree for  $G$ . When taking the inner product of  $x$  and the vector  $s \in S$  representing the spanning tree, the result would be the number of edges in the tree, which is  $d - 1$ . In contrast, if  $G$  is not connected, it does not have a spanning tree. This means that for all vectors  $s'$  in  $S$ , the inner product  $\langle s', x \rangle$  is less than  $d - 1$ . We can use the same logic to show that properties arising from NP-complete problems such as independent set, or simple ones that GNNs
cannot approximate, such as diameter, girth and connectivity are all encompassed by Definition 3. We are now ready to state the probabilistic invariance and universality result for RGraphCs.
Theorem 3. Let  $(\pmb{x},\pmb{y})$  be a graph isomorphism-invariant task that either i) has a smooth separator as in Assumption 1 or ii) is an inner-product decision graph problem as in Definition 3. Further, the task has infinitely graph isomorphism-invariant data as in Assumption 3. Then, there exists an RGraphC as in Definition 2 with absolutely continuous randomness source (cf. Assumption 2) that is probabilistic  $G$ -invariant and universal for  $(\pmb{x},\pmb{y})$  (as in Theorem 2). Further, the number of parameters of this RGraphC will depend only on the smallest finite absolute moments of its weight and bias distributions, i.e., the ones given by  $f_{\theta_f}(\pmb{u},\pmb{u}_i)$  and  $g_{\theta_g}(\pmb{u},\pmb{u}_b)$ .
The complete proof is in Appendix A The main insight behind Theorem 3 is using techniques from randomized algorithms to derive the universality of RLCs in the graph tasks from Definition 3. The other results follow the same line as Proposition 2 while replacing de Finetti's with Aldous-Hoover's theorem.
What is the practical impact of Assumption 3 in graph tasks? Janson and Diaconis [15] showed that Assumption 3 is equivalent to having the input graphs sampled from a graphon model. For the reader unfamiliar with graphons, this class encompasses from simple random graph models like  $G(n,p)$  [12], to modern matrix factorization methods in recommender systems [29]. Moreover, this assumption has been recently used to design representations that are invariant to graph size, see [3].
When are RGraphCs better than GNNs? We contrast the results of RGraphCs with GNNs[17, 22, 26] due its widespread use, empirical success and computational efficiency. A known issue with GNNs is its inability to approximate simple tasks such as graph connectivity [14]. This comes from the fact that GNNs cannot distinguish simple non-isomorphic graphs such as any pair of  $d$ -regular graphs. As such, GNNs cannot universally approximate tasks that assign different labels to any of such pairs. In contrast, RGraphCs can approximate any problem that either has a smooth boundary or can be tested with an inner product as in Definition 3.
Note that we can always define a task following Definition 3 that distinguishes between a specific graph and any other non-isomorphic input. Thus, unlike deterministic models, the set of tasks RGraphCs can solve is not attached to solving the graph isomorphism problem. It is simply attached to whether the decision problem can be tested with an inner product. Note that this does not imply that our model solves the graph isomorphism problem. To do so, we would need to test on an exponential number of tasks. The main implication of this observation is that the expressive power of probabilistic graph models is not attached to the ability to distinguish non-isomorphic graphs, but to the task's complexity. Thus, we are not doomed to fail at simple tasks like GNNs. Finally, note that GNNs aggregate messages using Deep Sets (or some variation of it). Thus, its approximation power, even for graphs that it can distinguish, requires a number of parameters that grows with the graphs' number of vertices.
Finally, we highlight that there exist more expressive, but still non-universal, models such as higher-order GNNs [22] and subgraph GNNs [2, 7]. However, they suffer from a heavy use of computational resources and are restricted to smaller, but existent, set of non-isomorphic graphs it cannot distinguish. As such, they suffer from the same aforementioned problems as GNNs.
Until now, our theory characterized the model space of invariant RLCs. It is natural to wonder whether such guarantees translate into practice. Concretely, we focus on investigating three questions related to our theoretical contributions.
Q1. Are RSetCs more parameter-efficient than Deep Sets in practice? (cf. Proposition 2)
Q2. Are RSetCs more robust than Deep Sets with out-of-distribution data? (cf. discussion in Section 3.1)
Q3. Can RGraphCs efficiently solve tasks that GNNs struggle in practice? (cf. Theorem 3)
Sorting task. To address Q1 and Q2, we consider the sorting task proposed in [32]. Our input is given by  $\mathbf{x}$ ,  $\mathrm{supp}(\mathbf{x}) = \mathbb{R}^d$ ,  $\mathbf{x}_i \sim \mathcal{N}(0,1)$  for some odd number of dimensions  $d$  and the labeling function by  $y(\mathbf{x}) = 1$  if for a vector  $w$ ,  $w_i := (-1)^{i+1}$  we have  $w^T \mathrm{sort}(\mathbf{x}) \geq 0$  or  $-1$  otherwise. We chose this task since it was shown in [32] that Deep Sets needs a hidden-layer at least as large as the input set  $(d)$  to approximate it with zero error. Thus, to showcase the parameter efficiency of
RSetCs in practice (cf. Proposition 2) we test both models in this task, while fixing the hidden layer at 5 on both. To make the results comparable, we train RSetCs with a single hidden layer (in both networks) and Deep Sets with a single hidden layer in each of its networks as well. Finally, to answer Q2 we test both models on sets twice as large as the ones used in training.
Sign task. We would also like to investigate Q1 in a setting where Deep Sets is not supposed to struggle. That is, we would like to know whether RSetCs can be more parameter-efficient in tasks that are (supposedly) easy for Deep Sets. For this, we consider the sign task  $y(\mathbf{x}) \coloneqq \operatorname{sgn}(\sum_{i=1}^{d} \operatorname{sgn}(\mathbf{x}_i))$  with  $\mathbf{x}_i \sim N(0,1)$ . Note that the networks in Deep Sets would need to simply learn the sign function and thus its parameters should not depend on the set size. To showcase the parameter efficiency of RSetCs, we consider Deep Sets with one and two hidden layers (in both networks) while keeping RSetCs with a single hidden layer. The rest of the experimental setup follows the sorting task verbatim.
Connectivity task. We chose the task of deciding whether a graph sampled from a  $G(n,p)$  model with  $p = 1.1 \cdot \log (n) / n$  is connected or not. Unlike RGraphCs, GNNs provably cannot approximate this simple task [14]. By comparing the performance of GNNs and RGraphCs in this task, we can empirically verify Theorem 3. Moreover, in contrast to GNNs, Theorem 3 tells us that the required number of parameters in RGraphCs is not attached to the graph size. Hence, we also test the performance of both models in this task while increasing the size of the input graphs and keeping hidden layer sizes fixed to 2 in both models. To make results comparable, we use a GNN with 3 layers and an RGraphC with 3 hidden layers of size in each network. Since in each GNN layer there are two network layers, the parameter sizes in the GNN and in the RGraphC are comparable.
Experimental setup. We used the Deep Sets architecture as proposed in the original paper [35]. For the GNN, we used the GIN [33] architecture, which completely captures the GNN properties mentioned in Section 3.2. Finally, to give perspective on how bad a model is performing, we also contrast the results with a constant classifier, i.e., how a classifier that always predicts the most common class performs. All models were trained with Pytorch [25] using Adagrad [10] for a maximum of 1000 epochs with early stopping. The reported results in Figures 1 and 2b are with respect to five runs. The training sets consisted of 1000 examples, while the validation and test sets contained 100 examples. We detail the hyperparameters and their search in Appendix C.
A1 (RSetC parameter efficiency). In Figure 1a we see the (test set) results for RSetCs and Deep Sets when varying the size of the input for a fixed hidden layer. We note that even in the first task,  $d = 5$ , when Deep Sets is supposed to approximate the task well it fails to generalize. Overall, for all input sizes we can see RSetCs consistently outperforming Deep Sets. On the other hand, in most of the tasks Deep Sets performs similarly or even worse than a constant classifier. Finally, in the sign task from Figure 2a we see that the performance of a single hidden-layer RSetC is closer to a two hidden-layer Depe Sets, showing its parameter efficiency even in tasks where Deep Sets is supposed to efficiently succeed. Thus, we can verify Theorem 2 and Proposition 2 in practice.
A2 (RSetC out-of-distribution robustness). We can see in Figure 1b how RSetCs provide consistently better results than Deep Sets when tested in sets twice the size as the ones used in training. This provides evidence to the size generalization discussion in Section 3.1. We can see that RSetCs perform even similarly to the in-distribution case, while Deep Sets is consistently worse than a constant classifier. Moreover, the high variance in Deep Sets' results confirm our observation in Section 3.1 about its sensitivity.
A3 (RGraphC universality and parameter efficiency). In Figure 2b we can see the results for RGraphCs vs. GNNs in the connectivity task. We note how the lack of expressive power coupled with the parameter inefficiency of GNNs is reflected in its poor performance, that decreases as the input size increases. In contrast, Theorem 3 is confirmed by RGraphCs' better performance even when the input size is increased (and the number of parameters is fixed). Finally, just as in Deep Sets we note a very high variance in the GNN results. This is often observed in tasks where there is no continuous features in the input, see e.g., [7].
Our work established the first class of models that leverages randomness to achieve universality and invariance in binary classification tasks. We combined tools from randomized algorithms and probability theory in our results. By leveraging this new principle, we were able to present resource-
(a) Results with in-distribution test set.
(b) Results using a test set containing sets twice the size of the ones used in training.
Figure 1: Results for the sorting task. We use a fixed hidden layer size of 5 in both models, while varying the training input size. The mean accuracy, together with standard deviation values, is reported using five runs.
(a) Results for the sign task. We use a fixed hidden layer size of 5 in both models, while varying the training input size. The mean accuracy, together with standard deviation values, is reported using five runs.
(b) Results for the connectivity task. We use a fixed hidden layer size of 2 in both models, while varying graph size. We report mean accuracy and standard deviation over five runs.
Figure 2: Results for the sign (set) and the connectivity (graph) tasks.
efficient models that, under mild assumptions, are universal and invariant in a probabilistic sense. This work can be extended in many different ways, e.g., designing architectures for other group invariances, such as  $\mathrm{SO}(n)$  or designing specific optimization procedures for RLCs. It is also interesting to understand the practical benefit of RLCs and their invariant counterparts in real-world tasks.
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), RGPIN-2021-03445 and of the Hasso Plattner Institute. L. Cotta is funded in part, by a postdoctoral fellowship provided by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. This work was done in part, when G. Yehuda was visiting the Simons Institute for the Theory of Computing.
[1] Aldous, D. J. (1981). Representations for partially exchangeable arrays of random variables. Journal of Multivariate Analysis, 11(4):581-598.
[27] Sieradzki, Y., Hodos, N., Yehuda, G., and Schuster, A. (2022). Coin flipping neural networks. In International Conference on Machine Learning, pages 20195-20214. PMLR.
[28] Simon, B. (1996). Representations of finite and compact groups. American Mathematical Soc.
[29] Srinivasan, B. and Ribeiro, B. (2020). On the equivalence between positional node embeddings and structural graph representations. In Iclr.
[30] Upfal, E. and Mitzenmacher, M. (2005). Probability and computing.
[31] Velickovic, P., Buesing, L., Overlan, M. C., Pascanu, R., Vinyals, O., and Blundell, C. (2020). Pointer graph networks. stat, 1050:11.
[32] Wagstaff, E., Fuchs, F. B., Engelcke, M., Osborne, M. A., and Posner, I. (2022). Universal approximation of functions on sets. Journal of Machine Learning Research, 23(151):1-56.
[33] Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2019). How powerful are graph neural networks? In International Conference on Learning Representations.
[34] Yang, Y., Li, Z., and Wang, Y. (2022). On the capacity of deep generative networks for approximating distributions. Neural Networks, 145:144-154.
[35] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. (2017). Deep sets. In Advances in neural information processing systems, pages 3391-3401.
Theorem 1 (Resource consumption of universal RLCs). Let  $(x,y)$  be a binary classification task that admits a smooth separator as in Assumption 1. Then, there exists an RLC with neural network  $f_{\theta^{\star}}$  and absolutely continuous randomness source  $\mathbf{u}$  (Assumption 2) that is universal in the limit, i.e.,
$$
P (\operatorname {m a j} \left(\left\{\operatorname {s g n} \left(\left\langle \boldsymbol {a} _ {\theta^ {*}} ^ {(j)}, x \right\rangle - \boldsymbol {b} _ {\theta^ {*}} ^ {(j)}\right) \right\} _ {j = 1} ^ {m}\right) = y (x)) > 1 - \exp \{- 2 \epsilon^ {2} m ^ {2} \},
$$
where  $\epsilon$  is the minimum bias of  $\mathcal{F}_{\theta^*}$ .
Proof of Theorem 1. Since Assumption 1 holds<sup>3</sup>, there exists a single hidden-layer neural network  $N$  that, like  $s$ , achieves zero-error in this task [8]. Further, since sgn is nonpolynomial, we can use it as the nonlinearity of this network [21]. Putting it all together, there exists a number of hidden units  $M$  and parameters  $b_{j}, o_{j} \in \mathbb{R}, w_{j} \in \mathbb{R}^{d}$  for  $j = 1, \ldots, M$  such that
$$
N (x) := \sum_ {j = 1} ^ {M} o _ {j} \operatorname {s g n} \left(\langle w _ {j}, x \rangle - b _ {j}\right),
$$
$$
N (x) = \operatorname {s g n} (s (x)), \forall x \in \mathcal {X}.
$$
Note that this means we can achieve zero-error in classification,  $N(x) = y(x),\forall x\in \mathcal{X}$ . Having this in mind, we will now show that for any network  $N$  constructed as such, we can build a limiting classifier  $\mathcal{F}_{\theta}$  equal to it. Let us first note that we can force positivity in the output weights of  $N$  by doing
$$
o _ {j} ^ {\prime} := o _ {j} \cdot \operatorname {s g n} (o _ {j})
$$
$$
w _ {j} ^ {\prime} := w _ {j} \cdot \operatorname {s g n} (o _ {j})
$$
without changing the classification