Title: Probabilistic Invariant Learning with Randomized Linear Classifiers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Invariant Learning

Geometric Deep Learning

Set Representations

Graph Representations

Expressive Power

Randomized Algorithms
Abstract: Designing models that are both expressive and preserve known invariances of tasks is an increasingly hard problem. Existing solutions tradeoff invariance for computational or memory resources. In this work, we show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, our key insight is that accepting probabilistic notions of universal approximation and invariance can reduce our resource requirements. More specifically, we propose a class of binary classification models called Randomized Linear Classifiers (RLCs). We give parameter and sample size conditions in which RLCs can, with high probability, approximate any (smooth) function while preserving invariance to compact group transformations. Leveraging this result, we design three RLCs that are provably probabilistic invariant for classification tasks over sets, graphs, and spherical data. We show how these models can achieve probabilistic invariance and universality using less resources than (deterministic) neural networks and their invariant counterparts. Finally, we empirically demonstrate the benefits of this new class of models on invariant tasks where deterministic invariant neural networks are known to struggle.

Probabilistic Invariant Learning with Randomized Linear Classifiers

Abstract

Designing models that are both expressive and preserve known invariances of tasks is an increasingly hard problem. Existing solutions tradeoff invariance for computational or memory resources. In this work, we show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, our key insight is that accepting probabilistic notions of universal approximation and invariance can reduce our resource requirements. More specifically, we propose a class of binary classification models called Randomized Linear Classifiers (RLCs). We give parameter and sample size conditions in which RLCs can, with high probability, approximate any (smooth) function while preserving invariance to compact group transformations. Leveraging this result, we design three RLCs that are provably probabilistic invariant for classification tasks over sets, graphs, and spherical data. We show how these models can achieve probabilistic invariance and universality using less resources than (deterministic) neural networks and their invariant counterparts. Finally, we empirically demonstrate the benefits of this new class of models on invariant tasks where deterministic invariant neural networks are known to struggle.

1 Introduction

A modern challenge in machine learning is designing model classes for invariant tasks that are both universal and resource-efficient. Consider designing an architecture for graph problems that is invariant under graph isomorphism. Ensuring that this class is universal is at least as hard as solving the graph isomorphism problem [5]. Similarly, permutation invariance universality comes at the cost of memory-the number of parameters scales with the sequence size [32].

Using randomness as a computational resource for algorithms is a fundamental idea in computer science. In their essence, randomized algorithms often provide simple solutions that use less memory or run in less time than their deterministic counterparts [30]. Since there is no free lunch, these gains come at the cost of accepting a probabilistic notion of correctness, i.e. , while deterministic algorithms always output correct answers, their randomized versions will be right only with high probability. Inspired by this, the key insight in our paper is using an external source of randomness to reduce time and space resources when learning an invariant task. To benefit from that, just as in randomized algorithms, we also need to introduce a probabilistic notion of invariance and expressive power. By guaranteeing invariance and universal approximation only with high probability, we show when it is possible to avoid the invariance-resource tradeoff.

We introduce a class of models for binary classification tasks named Randomized Linear Classifiers (RLCs). In an RLC the external source of randomness is input to a neural network that generates a random linear classifier -independently from the input- which is then used to make a prediction. To guarantee correctness with high probability, we need to repeat this process for a sufficient number of times, i.e. , take multiple samples, and predict the majority of their output- this is called amplification. The key features of RLCs are both i) randomness independent from input and ii) linear transformation of the input. Combined, these features provide both theoretical and practical simplicity. On the practical side, RLCs can offload computation and preserve privacy in resource-constrained devices. We highlight the benefits of RLCs in such applications below.

Online computation. Suppose that our goal is to answer classification queries as quickly as possible, or that the queries arrive in an 'online' fashion. By using RLCs, we can sample several linear classifiers 'offline' (before observing new inputs), and then in the online phase simply take the majority of a few (different) linear classifiers for each input. This way, the majority of computations can be done offline rather than online -as usual in existing online learning settings.

Private computation. Suppose that we wish to peform inference with an RLC, but we only have a low-resource computer ( e.g. , a smartwatch). A practical way of doing so is to perform the sampling on a remote server, then retrieve the answer. In our approach, the server can send the randomness, i.e. , the sampled linear classifier coefficients. There is no communication between the client and the server at all, and the client never sends the input -retaining its privacy. Moreover, since the client only needs to perform a few linear computations, it can be a low-resource computer.

The theoretical advantages of RLCs' features are explored in this work. We consider the required resources of an RLC to be universal and invariant for a given task as the number of parameters used by its neural network and the sufficient number of samples (amplification size). Then, we focus on establishing upper-bounds for these resources in different scenarios. More specifically, our contributions are three-fold:

We introduce the general class of Randomized Linear Classifiers (RLCs), presenting a universal approximation theorem coupled with a resource consumption characterization (Theorem 1). In specific, we show that any binary classification problem with a smooth boundary can be approximated with high probability by RLCs with at most the same number of parameters as a deterministic neural network. This is a result of general purpose and holds for any invariant or non-invariant task.

ii) We show how the problem of designing RLCs invariant to compact group transformations can be cast to designing invariant distributions of linear classifiers. As a result, we can leverage representation theorems in probability theory, such as de Finetti's [9], Aldous-Hoover 's[1] and Friedman's [13] to design invariant RLCs for tasks with set and graph data. Moreover, in Appendix B we also show a simple invariant model for spherical data using these ideas.

iii) Finally, we show both theoretically and empirically how invariant RLCs can succeed in tasks where deterministic architectures, such as Deep Sets and GNNs, cannot efficiently approximate. By introducing probabilistic notions of universality and invariance, our work establishes the first alternative to avoid the invariance-resource tradeoff with theoretical guarantees and empirical success.

1.1 Related Work

Here, we briefly review Coin-Flipping Neural Networks (CFNNs) and invariant representation learning -two core concepts that we build upon. Finally, we explain how Bayesian neural networks and RLCs belong to different learning paradigms.

Coin-flipping neural networks. CFNNs were presented by [27], where the authors propose several deep learning architectures using randomness as part of their inference. A few (constructive) examples were shown in which randomness can help by reducing the number of parameters or the depth of the network. In this paper we focus on the case of learning randomized linear classifiers and its use in invariant learning. The invariant models were not explored in [27],

and to the best of our knowledge this is the first work studying randomized models for invariant learning. One specific CFNN model proposed in [27] is the Hypernetwork CFNN. In this model, the weights of a model are first randomly generated. Then, the computation of a deterministic neural network using these random weights takes place. In this work, we focus on the 'extreme' case of this paradigm: we allow our model to use arbitrarily complex distributions to generate our final weights, but we insist that the deterministic part -using such weights- would be a linear function. That is, we push the computational complexity to the random part of the model. This gives rise to the class of models we study: Randomized Linear Classifiers (RLCs). The work of [27] considered a first version of RLCs, but did not provide any empirical results and their theoretical results hold only for single dimensional input ( i.e. , in R ). Hence, to the best of our knowledge our work also establishes the first (general purpose) universal approximation theorem of (non-invariant) RLCs ( cf. Section 2).

Invariant representation learning. Over the last years, there has been a growing interest in designing G -invariant architectures [4, 6, 18, 35]. In summary, we can often leverage an a priori knowledge that the function to be learned is invariant to the action of a group G by forcing our model to also be invariant to it. That is, G -invariant architectures incorporate a known propriety of the target. Such a reduced model space is known to both empirically and theoretically achieve better generalization [4, 6, 11, 19]. Existing works study the G -invariant function space and how to parameterize it with neural networks. Despite some empirical success, invariance guarantees often come at unreasonable prices. For instance, CNNs are guaranteed to be invariant to translations only with infinite grids (images) [19]. Regarding permutation invariance, Deep Sets is always invariant, but to remain universal it requires a hidden layer as large as the input set [32]. Finally, GNNs, as any other computationally efficient graph isomorphism-invariant architecture, cannot achieve universal approximation. More specifically, achieving graph isomorphism invariance and universal approximation is at least as hard as solving the graph isomorphism problem [5]. In contrast to these results, our work shows that such tradeoffs are usually restricted to the design of deterministic architectures. By accepting probabilistic notions of invariance and universality, we show that under mild data generation conditions we can circumvent these invariance-resource tradeoffs.

Bayesian neural networks. Bayesian Neural Networks (BNNs) incorporate Bayesian inference into both training and inference of neural networks. In short, BNNs explicitly assign a prior to the parameters and perform inference with a posterior. BNNs differ from RLCs in how randomness is used. BNNs explicitly model prior and posterior distributions over the weights. This is usually leveraged in the context of confidence estimation. On the other hand, RLCs do not model a prior or a posterior distribution over the parameters. Rather, RLCs take samples from a model that uses external randomness. In their essence, BNNs and RLCs are trying to achieve different goals. While BNNs are modeling uncertainty over decisions, RLCs are trying to output the correct target with the highest probability they can.

2 Randomized Linear Classifiers

We start by introducing the general class of Randomized Linear Classifiers. Due to our universal approximation result, this can be seen as the analogous of a multi-layer perceptron in the context of randomized linear models. In the next sections we will introduce the analogous of G -invariant models.

For notation simplicity, we denote the random variable of a value with bold letters, e.g. , a random variable x would have a realization x . We consider binary classification tasks where the input data x is supported on X := supp( x ) ⊆ R d and labeled according to a true labeling function y : X → {1 , 1 } . Therefore, a binary classification task can be summarized by the tuple ( x , y ) .

Given a source of randomness u supported on U := supp( u ) ⊆ R d u , and a neural network f θ : R d u → R d +1 parameterized by θ ∈ R d , an RLC predicts y ( x ) for x ∈ X according to

From above, one can see the coefficients ( a θ , b θ ) of the linear classifier given by the pushforward measure f θ # u . Finally, unless otherwise stated, we assume f θ to be a multi-layer perceptron with ReLU activations.

Analogously to randomized algorithms for decision problems, we can take multiple samples using Equation (1) and make a final prediction using their majority. To simplify notation, let us define the function of our final prediction taking m samples as

where each u ( j ) ∼ i.i.d P ( u ) . Note that as we take more samples, we approach the mode of the distribution of linear predictions for a given x . We define this value as the limiting classification of x , which implies the existence of the limiting classifier F θ : R d →{-1 , 1 } with

At this point, it is worth noting that although the random prediction (Equation (1)) is a linear function, its limiting classifier (Equation (3)) can induce an arbitrarily non-linear boundary. In fact, we leverage this notion to define the probabilistic version of universality. We say that an RLC is universal in a probabilistic sense if its limiting classifier (Equation (3)) is universal. It is easy to see that models making deterministic predictions collapse the notions of probabilistic and exact universality. On the other hand, for RLCs to be probabilistic universal it suffices to produce predictions that are biased towards the desired answer on every input, i.e. , P ( y ( m ) θ ( x ) = y ( x )) > 0 . 5 , ∀ x ∈ X .

In practice, we will make random predictions and therefore we need to capture a sufficient sample size m such that we are close to the limiting classifier with high probability. An important measure to understand how large m needs to be is what we call the minimum bias of the RLC. We denote it by ε and define it as the infimum of the total variation distances between the random predictions of every input x and a random variable r with Radamacher distribution. That is,

Put into words, ε captures a lower-bound on how close to a random prediction the RLC can be. And, naturally, the closer it is to a random prediction the larger m needs to be such that the majority converges to the limiting prediction. Therefore, our universal approximation result needs to capture how many parameters an RLC needs to converge to correct answers and how fast, i.e. , with respect to m , it does. Before we proceed with our results on m , p and the universality property, we need to define the general class of binary classification tasks we consider.

Assumption 1 (Tasks with smooth separators) . We say that a binary classification task ( x , y ) admits a smooth separator if there exists a neural network s : R d → R such that y ( x ) = sgn( s ( x )) , ∀ x ∈ supp( X ) .

Note that due to the universal approximation ability of neural networks, the set of tasks satisfying Assumption 1 is quite large. Moreover, Assumption 1 is a sufficient but not necessary condition . We will later prove universality for a wide class of graph problems and we believe many others exist. For now, let us finally state our result on the resource consumption and universal approximation of RLCs in tasks with smooth separators.

Assumption 2 (Absolutely continuous randomness source) . We say that the randomness source u is absolutely continuous if at least one of its coordinates u i , 1 ≤ i ≤ d u has an absolutely continuous marginal distribution.

Theorem 1 (Resource consumption of universal RLCs) . Let ( x , y ) be a binary classification task that admits a smooth separator as in Assumption 1. Then, there exists an RLC with neural network f θ ⋆ and absolutely continuous randomness source u (Assumption 2) that is universal in the limit, i.e.,

and makes random predictions that are correct with probability

where ϵ is the minimum bias of F θ ⋆ .

Further, if p † is the number of parameters used by a deterministic neural network with one hidden layer to achieve zero-error in the task, f θ has at most

The complete proof is in Appendix A Note that Theorem 1 does not assume or explore any invariance in the task. It is a general universality result that we will leverage in our study of invariant RLCs, but of independent interest. One way to interpret Theorem 1 is as a probabilistic version of the classical universal approximation theorem of multi-layer perceptrons [8]. The result on the number of parameters is important to make it clear that RLCs do not need to find a solution more complex than one given by a deterministic model in the supervised learning task.

Until now, we have established the expressive power of RLCs, i.e. , that, under mild assumptions, they can be as expressive and resource-efficient as deterministic neural networks. But, there is still the general question: When are RLCs more resource-efficient than deterministic neural networks? In [27] it was constructed an specific task where an RLC (although the authors do not name it as such) uses a constant number of parameters, while a deterministic neural network would need a number of parameters that grows with the input dimension. Here, we are interested in more general settings. In the next section we will show a large class of invariant tasks where invariant RLCs are more resource-efficient than their invariant deterministic neural network counterparts.

3 G -invariant Randomized Linear Classifiers

Now, we turn our focus to binary classification tasks that are invariant to compact group transformations. That is, let G be a compact group with an action of g ∈ G on x ∈ X denoted by g · x . A task ( x , y ) is G -invariant if y ( x ) = y ( g · x ) , ∀ x ∈ X , ∀ g ∈ G . It is easy to see that an RLC that perfectly classifies the task in the probabilistic notion, i.e. , F θ ⋆ ( x ) = y ( x ) , ∀ x ∈ X , will also be probabilistic G -invariant, i.e. , F θ ⋆ ( x ) = F θ ⋆ ( g · x ) , ∀ x ∈ X , ∀ g ∈ G . Thus, if we know that a task is G -invariant, can we restrict our model class to G -invariant RLCs? At first, the answer is not obvious. We need a design principle that guarantees a search space with only G -invariant solutions while at least one of them has zero error. In Theorem 2 we present the result that guides our design of invariant RLCs.

Theorem 2 ( G -invariant RLCs) . Let ( x , y ) be a G -invariant task with a smooth separator as in Assumption 1. Then, the set of RLCs with a G -invariant distribution in the classifier weights, i.e.,

and absolutely continuous randomness source (cf. Assumption 2) is both probabilistic G -invariant and universal in ( x , y ) . That is,

The complete proof is in Appendix A From above, we now know that designing universal RLCs for invariant tasks can be cast to the problem of designing universal G -invariant distributions -together with a separate (possibly dependent) bias distribution. This design principle differs fundamentally from deterministic invariant representation learning. Here, instead of designing architectures that are G -invariant, we wish to design architectures that induce a G -invariant distribution. That is, the RLC neural network f θ is not itself G -invariant.

Next, we will define a very useful assumption about the data generation process. It will allow us to leverage representation results in probability theory to both design universal and efficient G -invariant distributions and allow for variable-size set and graph data.

Assumption 3 (Infinitely G -invariant data) . Let G ∞ be a homomorphism of G into the composition of homormophisms of G in all finite dimensions, e.g., if G is the set of permutations of { 1 , . . . , d } , G ∞ is the set of permutations of N . We say that a task ( x , y ) has infinitely G -invariant data if there exists an infinite sequence of random variables ( x i ) ∞ i =1 d = g ∞ · ( x i ) ∞ i =1 , ∀ g ∞ ∈ G ∞ , where x d = ( x i ) i ∈ S for S ∈ ( N d ) , with ( N d ) being the set of all d -size subsets of N .

Although the above might seem unintuitive, the representation theorems also help us make sense of them. The data generation process will take the same form as our linear weights distribution. As we will see, for sets it means that items are sampled i.i.d. given a common latent factor of the set [9]. In graphs edges are also generated i.i.d. given latent factors of their endpoints and the graph [1].

and

Proposition 1 (Infinitely G -invariant RLCs) . Let ( x , y ) be a G -invariant task with infinitely G -invariant data (Assumption 3) with a smooth separator as in Assumption 1. Then, the set of RLCs with an infinitely G -invariant distribution in the linear classifier weights, i.e., as in Assumption 3 ( a θi ) ∞ i =1 d = g ∞ · ( a θi ) ∞ i =1 , ∀ g ∞ ∈ G ∞ , where a θ d = ( a θi ) i ∈ S , for S ∈ ( N d ) , and absolutely continuous randomness source (cf. Assumption 2) is probabilistic G -invariant and universal for ( x , y ) as in Theorem 2.

The complete proof is in Appendix A Next, we design G -invariant RLCs drawing from results in the representation theory of probability distributions. We consider tasks with set and graph data. In Appendix B we also consider spherical data. For each, we derive conditions for G -invariance and universality, highlighting their resource gain when compared to their deterministic counterpart.

3.1 RLCs for set data

Here we consider tasks that are invariant to permutations, i.e. , our input is a set 1 of vectors. More formally, we let our input be supported on X := ( R k ) d 2 and G := Sym([ d ]) be the group of permutations of [ d ] := { 1 , . . . , d } . Then, for an input x ∈ X the action g · x is given by permuting the d vectors of size k according to g .

The key insight to design RLCs for set data is leveraging the classic de Finetti's theorem [9]. In summary, de Finetti showed how any infinite sequence of exchangeable random variables can be expressed as an infinite sequence of i.i.d. random variables conditioned on a common latent measure. Now, when Assumption 3 is true, de Finetti tells us that we can sample our weights a θ by first sampling a common noise and use it as input to the same function, which generates the linear weights together with an independent noise in an i.i.d. manner. Note that the bias does not have the exchangeability property. Thus, it needs to receive the same shared noise as the set but use a different function, since it is not necessarily sampled in the same way. Next, we formalize these notions by defining the class of Randomized Set Classifiers (RSetCs) and characterizing its universal approximation power together with resource consumption.

Definition 1 (Randomized Set Classifiers (RSetCs)) . A Randomized Set Classifier (RSetC) uses two neural networks f θ f : R 2 → R k and g θ g : R 2 → R together with d +2 sources of randomness: u , ( u i ) d i =1 , and u b . The random linear classifier coefficients are generated with

where a θ ( k ) i is the i th chunk of k random linear weights, i.e., the weights multiplying the i th vector.

Proposition 2 (Universality and resource consumption of RSetCs) . Let ( x , y ) be a permutationinvariant task with a smooth separator as in Assumption 1 and infinitely permutation-invariant data as in Assumption 3. Then, RSetCs as in Definition 1 with absolutely continuous randomness source (cf. Assumption 2) are probabilistic G -invariant and universal for ( x , y ) (as in Theorem 2). Further, the number of parameters needed by RSetCs in this task will depend only on the smallest finite absolute moments of the weight and bias distributions in the zero-error solution, i.e., the ones given by f θ ⋆ f ( u , u i ) and g θ ⋆ g ( u , u b ) .

The complete proof is in Appendix A We highlight that Proposition 2 is combining de Finetti's theorem [9], Kallenberg's noise transfer theorem [16] and recent results on the capacity of neural networks to generate distributions from noise [34].

What is the practical impact of Assumption 3 in set tasks? Assumption 3 implies that an input set x is generated by first sampling some latent variable Θ ∼ ν which is then used to sample items in an i.i.d. manner x 1 , x 2 , . . . |∼ Θ i.i.d. P Θ . We can make sense of this with an example in e-commerce systems. If the set represents items in a shopping cart, the latent factor Θ can be thought of as a summary of the user's shopping behavior. Overall, this assumption is not appropriate if there could be mutually exclusive items in the set.

When are RSetCs better than Deep Sets? We contrast the results of RSetCs with Deep Sets [35] due to its universality and computational efficiency [32]. Overall, other deterministic set models

are either variations of Deep Sets that inherent the same properties [20, 24] or are computationally inefficient [23]. Apart from universality, it is important to note from Proposition 2 how the number of parameters used by the RSetC does not depend on the set size. Instead, its resource consumption is related to how smooth the distribution of linear classifiers weights and bias are. Since all the weights have the same distribution (conditioned on u ), the set size is not relevant. This comes in contrast to Deep Sets, which as shown in [32] only achieves universality when the hidden-layer and the input set have the same size.

Moreover, Deep Sets is known to have poor set size generalization abilities, i.e. , when the test set contains larger sets than the training set [31]. In RSetCs, although the number of parameters does not depend on the set size, the distribution of linear coefficients can use this information. Overall, the gain in robustness to set size might come from RSetCs performing simple linear transformations of the input. That is, for small changes in the set size, the output is not expected to change abruptly. This opposes to Deep Sets, where the representations of the items are aggregated and input to a neural network. Thus, any small addition to the set can arbitrarily change the output. Although we cannot always guarantee size generalization for RSetCs, we give empirical evidence in Section 4 that they do seem to be a lot more robust than Deep Sets.

3.2 RLCs for graph data

Here we consider tasks that take graphs as input. Given a (possibly weighted) graph G = ( V, E ) with d vertices and M edges, we take as input a vector x of size d 2 representing the vectorized adjacency matrix of G . Our invariance is then to graph isomorphism. That is, we again have the permutation group G := Sym([ d ]) , but now with a different action. Here g · x jointly permutes the rows and columns of the adjacency matrix corresponding to x . This way, x and g · x represent isomorphic graphs. We will then assume that our graph tasks are G -invariant, i.e. , two isomorphic graphs x, g · x will always have the same label.

Now, we can leverage Aldous-Hoover's theorem, the analogous of de Finetti's theorem for joint exchangeability in 2-dimensional arrays. It gives us a model, invariance and universality results analogous to the ones for RSetCs.

Definition 2 (Randomized Graph Classifiers) . A Randomized Graph Classifier (RGraphC) uses two neural networks f θ f : R 4 → R and g θ g : R → R together with d 2 + d +1 sources of randomness: u ,

where a θij is the random linear weight multiplying the entry of x that corresponds to the entry in row i and column j of the graph's adjacency matrix.

Note that we can use the same proof of Proposition 2 and have an equivalent result for RGraphCs. However, since graphs are discrete objects the smooth separator assumption ( cf. Assumption 1) might not be so easily satisfied. To address this, we define a sufficiently large class of graph models that RGraphCs can approximate.

Definition 3 (Inner-product decision graph problems) . Let G = ( V, E ) be a graph over d vertices with vectorized adjacency matrix x ∈ X := supp( x ) ⊆ R d 2 . We say that a graph property y : X → {1 , 1 } is an inner product verifiable property if there exists a set of vectors S ⊆ R d 2 and a constant b ∈ R such that:

For all graphs x + ∈ X satisfying the property y ( x + ) = 1 , there exists s ∈ S , such that ⟨ s, x + ⟩ ≥ b ;

For all graphs x -∈ X not satisfying the property y ( x -) = -1 , we have ⟨ x -, s ′ ⟩ < b, ∀ s ′ ∈ S .

As an example of such property, consider connectivity. The set S above is defined to be the set of all binary vectors representing adjacencies of spanning trees. The threshold in this case would be d -1 . If G is connected, then there exists a spanning tree for G . When taking the inner product of x and the vector s ∈ S representing the spanning tree, the result would be the number of edges in the tree, which is d -1 . In contrast, if G is not connected, it does not have a spanning tree. This means that for all vectors s ′ in S , the inner product ⟨ s ′ , x ⟩ is less than d -1 . We can use the same logic to show that properties arising from NP-complete problems such as independent set, or simple ones that GNNs

cannot approximate, such as diameter, girth and connectivity are all encompassed by Definition 3. We are now ready to state the probabilistic invariance and universality result for RGraphCs.

Theorem 3. Let ( x , y ) be a graph isomorphism-invariant task that either i) has a smooth separator as in Assumption 1 or ii) is an inner-product decision graph problem as in Definition 3. Further, the task has infinitely graph isomorphism-invariant data as in Assumption 3. Then, there exists an RGraphC as in Definition 2 with absolutely continuous randomness source (cf. Assumption 2) that is probabilistic G -invariant and universal for ( x , y ) (as in Theorem 2). Further, the number of parameters of this RGraphC will depend only on the smallest finite absolute moments of its weight and bias distributions, i.e., the ones given by f θ f ( u , u i ) and g θ g ( u , u b ) .

The complete proof is in Appendix A The main insight behind Theorem 3 is using techniques from randomized algorithms to derive the universality of RLCs in the graph tasks from Definition 3. The other results follow the same line as Proposition 2 while replacing de Finetti's with Aldous-Hoover's theorem.

What is the practical impact of Assumption 3 in graph tasks? Janson and Diaconis [15] showed that Assumption 3 is equivalent to having the input graphs sampled from a graphon model. For the reader unfamiliar with graphons, this class encompasses from simple random graph models like G ( n, p ) [12], to modern matrix factorization methods in recommender systems [29]. Moreover, this assumption has been recently used to design representations that are invariant to graph size, see [3].

When are RGraphCs better than GNNs? We contrast the results of RGraphCs with GNNs[17, 22, 26] due its widespread use, empirical success and computational efficiency. A known issue with GNNs is its inability to approximate simple tasks such as graph connectivity [14]. This comes from the fact that GNNs cannot distinguish simple non-isomorphic graphs such as any pair of d -regular graphs. As such, GNNs cannot universally approximate tasks that assign different labels to any of such pairs. In contrast, RGraphCs can approximate any problem that either has a smooth boundary or can be tested with an inner product as in Definition 3.

Note that we can always define a task following Definition 3 that distinguishes between a specific graph and any other non-isomorphic input. Thus, unlike deterministic models, the set of tasks RGraphCs can solve is not attached to solving the graph isomorphism problem. It is simply attached to whether the decision problem can be tested with an inner product. Note that this does not imply that our model solves the graph isomorphism problem. To do so, we would need to test on an exponential number of tasks. The main implication of this observation is that the expressive power of probabilistic graph models is not attached to the ability to distinguish non-isomorphic graphs, but to the task's complexity. Thus, we are not doomed to fail at simple tasks like GNNs. Finally, note that GNNs aggregate messages using Deep Sets (or some variation of it). Thus, its approximation power, even for graphs that it can distinguish, requires a number of parameters that grows with the graphs' number of vertices.

Finally, we highlight that there exist more expressive, but still non-universal, models such as higherorder GNNs [22] and subgraph GNNs [2, 7]. However, they suffer from a heavy use of computational resources and are restricted to smaller, but existent, set of non-isomorphic graphs it cannot distinguish. As such, they suffer from the same aforementioned problems as GNNs.

4 Experiments

Until now, our theory characterized the model space of invariant RLCs. It is natural to wonder whether such guarantees translate into practice. Concretely, we focus on investigating three questions related to our theoretical contributions.

Q1. Are RSetCs more parameter-efficient than Deep Sets in practice? ( cf. Proposition 2)

Q2. Are RSetCs more robust than Deep Sets with out-of-distribution data? ( cf. discussion in Section 3.1)

Q3. Can RGraphCs efficiently solve tasks that GNNs struggle in practice? ( cf. Theorem 3)

Sorting task. To address Q1 and Q2 , we consider the sorting task proposed in [32]. Our input is given by x , supp( x ) = R d , x i ∼ N (0 , 1) for some odd number of dimensions d and the labeling function by y ( x ) = 1 if for a vector w,w i := ( -1) i +1 we have w T sort ( x ) ≥ 0 or -1 otherwise. We chose this task since it was shown in [32] that Deep Sets needs a hidden-layer at least as large as the input set ( d ) to approximate it with zero error. Thus, to showcase the parameter efficiency of

RSetCs in practice ( cf. Proposition 2) we test both models in this task, while fixing the hidden layer at 5 on both. To make the results comparable, we train RSetCs with a single hidden layer (in both networks) and Deep Sets with a single hidden layer in each of its networks as well. Finally, to answer Q2 we test both models on sets twice as large as the ones used in training.

Sign task. We would also like to investigate Q1 in a setting where Deep Sets is not supposed to struggle. That is, we would like to know whether RSetCs can be more parameter-efficient in tasks that are (supposedly) easy for Deep Sets. For this, we consider the sign task y ( x ) := sgn( ∑ d i =1 sgn( x i )) with x i ∼ N (0 , 1) . Note that the networks in Deep Sets would need to simply learn the sign function and thus its parameters should not depend on the set size. To showcase the parameter efficiency of RSetCs, we consider Deep Sets with one and two hidden layers (in both networks) while keeping RSetCs with a single hidden layer. The rest of the experimental setup follows the sorting task verbatim.

Connectivity task. We chose the task of deciding whether a graph sampled from a G ( n, p ) model with p = 1 . 1 · log( n ) /n is connected or not. Unlike RGraphCs, GNNs provably cannot approximate this simple task [14]. By comparing the performance of GNNs and RGraphCs in this task, we can empirically verify Theorem 3. Moreover, in contrast to GNNs, Theorem 3 tells us that the required number of parameters in RGraphCs is not attached to the graph size. Hence, we also test the performance of both models in this task while increasing the size of the input graphs and keeping hidden layer sizes fixed to 2 in both models. To make results comparable, we use a GNN with 3 layers and an RGraphC with 3 hidden layers of size in each network. Since in each GNN layer there are two network layers, the parameter sizes in the GNN and in the RGraphC are comparable.

Experimental setup. We used the Deep Sets architecture as proposed in the original paper [35]. For the GNN, we used the GIN [33] architecture, which completely captures the GNN properties mentioned in Section 3.2. Finally, to give perspective on how bad a model is performing, we also contrast the results with a constant classifier, i.e. , how a classifier that always predicts the most common class performs. All models were trained with Pytorch [25] using Adagrad [10] for a maximum of 1000 epochs with early stopping. The reported results in Figures 1 and 2b are with respect to five runs. The training sets consisted of 1000 examples, while the the validation and test sets contained 100 examples. We detail the hyperparameters and their search in Appendix C.

A1 (RSetC parameter efficiency). In Figure 1a we see the (test set) results for RSetCs and Deep Sets when varying the size of the input for a fixed hidden layer. We note that even in the first task, d = 5 , when Deep Sets is supposed to approximate the task well it fails to generalize. Overall, for all input sizes we can see RSetCs consistently outperforming Deep Sets. On the other hand, in most of the tasks Deep Sets performs similarly or even worse than a constant classifier. Finally, in the sign task from Figure 2a we see that the performance of a single hidden-layer RSetC is closer to a two hidden-layer Depe Sets, showing its parameter efficiency even in tasks where Deep Sets is supposed to efficiently succeed. Thus, we can verify Theorem 2 and Proposition 2 in practice.

A2 (RSetC out-of-distribution robustness). We can see in Figure 1b how RSetCs provide consistently better results than Deep Sets when tested in sets twice the size as the ones used in training. This provides evidence to the size generalization discussion in Section 3.1. We can see that RSetCs perform even similarly to the in-distribution case, while Deep Sets is consistently worse than a constant classifier. Moreover, the high variance in Deep Sets' results confirm our observation in Section 3.1 about its sensitivity.

A3 (RGraphC universality and parameter efficiency). In Figure 2b we can see the results for RGraphCs vs. GNNs in the connectivity task. We note how the lack of expressive power coupled with the parameter inefficiency of GNNs is reflected in its poor performance, that decreases as the input size increases. In contrast, Theorem 3 is confirmed by RGraphCs' better performance even when the input size is increased (and the number of parameters is fixed). Finally, just as in Deep Sets we note a very high variance in the GNN results. This is often observed in tasks where there is no continuous features in the input, see e.g. , [7].

5 Conclusions

Our work established the first class of models that leverages randomness to achieve universality and invariance in binary classification tasks. We combined tools from randomized algorithms and probability theory in our results. By leveraging this new principle, we were able to present resource-

(a) Results with in-distribution test set.

(b) Results using a test set containing sets twice the size of the ones used in training.

(a) Results for the sign task. We use a fixed hidden layer size of 5 in both models, while varying the training input size. The mean accuracy, together with standard deviation values, is reported using five runs.

(b) Results for the connectivity task. We use a fixed hidden layer size of 2 in both models, while varying graph size. We report mean accuracy and standard deviation over five runs.

efficient models that, under mild assumptions, are universal and invariant in a probabilistic sense. This work can be extended in many different ways, e.g. , designing architectures for other group invariances, such as SO ( n ) or designing specific optimization procedures for RLCs. It is also interesting to understand the practical benefit of RLCs and their invariant counterparts in real-world tasks.

References

[1] Aldous, D. J. (1981). Representations for partially exchangeable arrays of random variables. Journal of Multivariate Analysis , 11(4):581-598.

Bevilacqua, B., Frasca, F., Lim, D., Srinivasan, B., Cai, C., Balamurugan, G., Bronstein, M. M., and Maron, H. (2022). Equivariant subgraph aggregation networks. In International Conference on Learning Representations .

Bevilacqua, B., Zhou, Y., and Ribeiro, B. (2021). Size-invariant graph representations for graph classification extrapolations. arXiv preprint arXiv:2103.05045 .

Bronstein, M. M., Bruna, J., Cohen, T., and Veliˇ ckovi´ c, P. (2021). Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 .

Chen, Z., Villar, S., Chen, L., and Bruna, J. (2019). On the equivalence between graph isomorphism testing and function approximation with GNNs. In Advances in Neural Information Processing Systems , pages 15868-15876.

Cohen, T. and Welling, M. (2016). Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. Pmlr.

Cotta, L., Morris, C., and Ribeiro, B. (2021). Reconstruction for powerful graph representations. Advances in Neural Information Processing Systems , 34.

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314.

De Finetti, B. (1929). Funzione caratteristica di un fenomeno aleatorio. In Atti del Congresso Internazionale dei Matematici: Bologna del 3 al 10 de settembre di 1928 , pages 179-190.

Duchi, J., Hazan, E., and Singer, Y . (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7).

Elesedy, B. (2022). Group symmetry in pac learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning .

Erdos, P., Renyi, A., et al. (1960). On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci , 5(1):17-60.

Freedman, D. A. (1962). Invariants under mixing which generalize de finetti's theorem. The Annals of Mathematical Statistics , 33(3):916-923.

Garg, V., Jegelka, S., and Jaakkola, T. (2020). Generalization and representational limits of graph neural networks. In International Conference on Machine Learning , pages 3419-3430. PMLR.

Janson, S. and Diaconis, P. (2008). Graph limits and exchangeable random graphs. Rendiconti di Matematica e delle sue Applicazioni. Serie VII , pages 33-61.

Kallenberg, O. (1997). Foundations of modern probability , volume 2. Springer.

Kipf, T. N. and Welling, M. (2016). Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning .

Kipf, T. N. and Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representation .

Kondor, R. and Trivedi, S. (2018). On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning , pages 2747-2755. PMLR.

Lee, J., Lee, Y ., Kim, J., Kosiorek, A., Choi, S., and Teh, Y . W. (2019). Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning , pages 3744-3753. PMLR.

Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks , 6(6):861-867.

Morris, C., Ritzert, M., Fey, M., Hamilton, W. L., Lenssen, J. E., Rattan, G., and Grohe, M. (2019). Weisfeiler and Leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence , pages 4602-4609.

Murphy, R. L., Srinivasan, B., Rao, V., and Ribeiro, B. (2019). Janossy pooling: Learning deep permutationinvariant functions for variable-size inputs. International Conference on Learning Representations .

Ong, E. and Veliˇ ckovi´ c, P. (2022). Learnable commutative monoids for graph neural networks. arXiv preprint arXiv:2212.08541 .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems , pages 8026-8037.

Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2009). The graph neural network model. IEEE Transactions on Neural Networks , 20(1):61-80.

Sieradzki, Y., Hodos, N., Yehuda, G., and Schuster, A. (2022). Coin flipping neural networks. In International Conference on Machine Learning , pages 20195-20214. PMLR.

Simon, B. (1996). Representations of finite and compact groups . American Mathematical Soc.

Srinivasan, B. and Ribeiro, B. (2020). On the equivalence between positional node embeddings and structural graph representations. In Iclr .

Upfal, E. and Mitzenmacher, M. (2005). Probability and computing.

Velickovic, P., Buesing, L., Overlan, M. C., Pascanu, R., Vinyals, O., and Blundell, C. (2020). Pointer graph networks. stat , 1050:11.

Wagstaff, E., Fuchs, F. B., Engelcke, M., Osborne, M. A., and Posner, I. (2022). Universal approximation of functions on sets. Journal of Machine Learning Research , 23(151):1-56.

Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2019). How powerful are graph neural networks? In International Conference on Learning Representations .

Yang, Y., Li, Z., and Wang, Y. (2022). On the capacity of deep generative networks for approximating distributions. Neural Networks , 145:144-154.

Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. (2017). Deep sets. In Advances in neural information processing systems , pages 3391-3401.

A Proofs

Theorem 1 (Resource consumption of universal RLCs) . Let ( x , y ) be a binary classification task that admits a smooth separator as in Assumption 1. Then, there exists an RLC with neural network f θ ⋆ and absolutely continuous randomness source u (Assumption 2) that is universal in the limit, i.e.,

and makes random predictions that are correct with probability

where ϵ is the minimum bias of F θ ⋆ .

Further, if p † is the number of parameters used by a deterministic neural network with one hidden layer to achieve zero-error in the task, f θ has at most

Proof of Theorem 1. Since Assumption 1 holds 3 , there exists a single hidden-layer neural network N that, like s , achieves zero-error in this task [8]. Further, since sgn is nonpolynomial, we can use it as the nonlinearity of this network [21]. Putting it all together, there exists a number of hidden units M and parameters b j , o j ∈ R , w j ∈ R d for j = 1 , . . . , M such that

Note that this means we can achieve zero-error in classification, N ( x ) = y ( x ) , ∀ x ∈ X . Having this in mind, we will now show that for any network N constructed as such, we can build a limiting classifier F θ equal to it. Let us first note that we can force positivity in the output weights of N by doing

and without changing the classification

Further, note that we can also multiply the output weights by the constant ∑ M j =1 o ′ j without changing the sign and thus the classification. Overall, we can define a network N ⋆ equivalent to N in classification with

Now, note that the limiting classifier can be written as F θ ( x ) = sgn( E a θ , b θ sgn( ⟨ a θ , x ⟩ = b θ )) . We can then define its distribution as one that samples from the weights and bias w ′ j , b j with probabilities according to 1 ∑ j M =1 o ′ j o ′ j . This implies that F θ ( x ) = N ⋆ ( x ) = N ( x ) , making our F θ achieve zero-error in the task. Note that if p † := M is the number of parameters in N , we can generate the linear coefficients in F θ ( x ) with at most p † + O (1) -simply take the external noise from a uniform distribution and map it to the probabilities in the output layer to pick the coefficients.

Finally, note that we have to also characterize the probability that y ( m ) θ outputs a different answer from F θ . The common tool for this is Hoeffding's inequality, which lets us upper-bound it for some x ∈ X with

̸

where ε x = P ( y ( m ) θ = F θ ( x ) ) -0 . 5 . Note that ε x depends on x and our definition of the minimum bias ϵ in the main text is precisely the lowest of all such ε x . Thus, as stated we finally have that for all x ∈ X

Note that the above is true for any F θ , including the universal and invariant ones later discussed in the main text.

Theorem 2 ( G -invariant RLCs) . Let ( x , y ) be a G -invariant task with a smooth separator as in Assumption 1. Then, the set of RLCs with a G -invariant distribution in the classifier weights, i.e.,

and absolutely continuous randomness source (cf. Assumption 2) is both probabilistic G -invariant and universal in ( x , y ) . That is, and

Proof of Theorem 2. Let us start with Proposition 3, a central observation needed in Theorem 2. Put into words Proposition 3 says that we can transfer the action of G to the weights of the linear classifier. For the reader more familiar with group representation theory, the result follows immediately from noting that compact groups admit unitary representations, see [28] for a good resource on the matter.

Proposition 3. If G is a compact group and g · x is an action of g ∈ G on x ∈ R d , there exists a bijective mapping α : G → G such that

Proof. Since we are dealing with actions on a finite-dimensional vector space, the action · of g can be associated with a linear representation ρ of G , i.e. , ρ ( g ) ∈ R d × d . That is, if [ x ] is the column vector of x , we have that ρ ( g )[ x ] = g · x . Now, since G is compact and R d is finite-dimensional, G also admits a unitary linear representation ρ u with associated action · u . Then, it is easy to see that

Thus, since inversion defines a bijection on G , Proposition 3 holds for the unitary representation action. Now, note that -as any other group action-· u defines a homomorphism between G and the symmetric group of R d . Then, for any other action · there exists a bijective function β : G → G such that g · x = β ( g ) · u x, ∀ x ∈ R d . Hence, we can write

Finally, since β and group inversion are bijective, we finish the proof by defining the bijective mapping α ( g ) := β -1 ( β ( g ) -1 ) for every g ∈ G .

Now, we can proceed to prove the universality part of Theorem 2. Since the task admits a smooth separator, there exists a universal limiting classifier F θ for it. That is, for every x ∈ X

Since the task is G -invariant, we know that for every g ∈ G

Note that since G is compact, it admits a unique normalized Haar measure λ 4 . Then, the insight comes from considering a random group action g ∼ λ . By Fubini's theorem and Proposition 3, we have

Now, we can define a limiting classifier F θ ⋆ that samples coefficients according to b θ ⋆ d = b θ and a θ ⋆ d = g · a θ . From above, we know that F θ ⋆ ( x ) = F θ ( x ) , ∀ x ∈ X . Since F θ is universal, F θ ⋆ is also universal. Finally, since g is sampled according to the Haar measure, we have that

At last, it is easy to see from Proposition 3 that the action on the weights can be transferred to the input as well, meaning that every model is probabilistic invariant, i.e. , F θ ( x ) = F θ ( g · x ) , ∀ x ∈ X , ∀ g ∈ G, ∀ θ ∈ R p , which finalizes the proof.

Proposition 1 (Infinitely G -invariant RLCs) . Let ( x , y ) be a G -invariant task with infinitely G -invariant data (Assumption 3) with a smooth separator as in Assumption 1. Then, the set of RLCs with an infinitely G -invariant distribution in the linear classifier weights, i.e., as in Assumption 3 ( a θi ) ∞ i =1 d = g ∞ · ( a θi ) ∞ i =1 , ∀ g ∞ ∈ G ∞ , where a θ d = ( a θi ) i ∈ S , for S ∈ ( N d ) , and absolutely continuous randomness source (cf. Assumption 2) is probabilistic G -invariant and universal for ( x , y ) as in Theorem 2.

Proof of Proposition 1. From Proposition 3 it also follows that the model is G -invariant. Let us proceed to prove it is universal. Since the task admits a smooth separator, from Theorem 1 there exists an RLC F θ such that

Since the data is infinitely G -invariant, we have that for the infinite G -invariant sequence ( x i ) ∞ i =1

with ( N d ) being the set of all d -size subsets of N . Let λ be the unique normalized Haar measure of G . As in the proof of Theorem 2, we can take a random group action g ∼ λ and have from Proposition 3 and Fubini's theorem that

Now, let us define a task ( x ′ , y ) on a higher dimension d ′ ≥ d, supp( x ) ⊂ supp( x ′ ) ⊆ R d ′ where y ( x ′ ) = y ( x ′ [1: d ] ) . Further, g ′ is a random group action (sampled from the Haar measure) from G ′ , the homomorphism of G into the dimension d ′ . From above, together with Proposition 3 we have that

Now, from above, since the lower dimensions { 1 , . . . , n } are G -invariant if all higher dimensions are G -invariant, it is easy to see that there is an infinitely G -invariant sequence of weights ( a θi ) ∞ i =1 that achieves zero error.

Proposition 2 (Universality and resource consumption of RSetCs) . Let ( x , y ) be a permutation-invariant task with a smooth separator as in Assumption 1 and infinitely permutation-invariant data as in Assumption 3. Then, RSetCs as in Definition 1 with absolutely continuous randomness source (cf. Assumption 2) are probabilistic G -invariant and universal for ( x , y ) (as in Theorem 2). Further, the number of parameters needed by RSetCs in this task will depend only on the smallest finite absolute moments of the weight and bias distributions in the zero-error solution, i.e., the ones given by f θ ⋆ f ( u , u i ) and g θ ⋆ g ( u , u b ) .

Proof of Proposition 2. The result follows directly from the combination of de Finetti's theorem [9], Kallenberg's noise transfer theorem [16] and the recent results on the capacity of neural networks to generate distributions from noise [34]. Let us outline it in more detail. From de Finetti's theorem [9] we know that any infinite sequence of exchangeable random variables can be expressed by i.i.d. random variables conditioned on a common latent measure. Combining this with Kallenberg's noise transfer theorem we have that the weights and biases can be written as f ( u , u i ) and g ( u , u b ) where f, g are measureable maps and the noises are sampled from a uniform distribution. Finally, the work of [34] showed that we can replace the uniform noise with absolutely continuous noise and f, g with a sufficiently expressive ReLU multi-layer perceptron. Moreover, Theorem 2.1 in [34] says that the number of parameters depends only on the output dimension and the smallest finite absolute moments of the distributions. Since the output dimension k is constant for us, it depends solely on the latter and we finalize the proof.

Theorem 3. Let ( x , y ) be a graph isomorphism-invariant task that either i) has a smooth separator as in Assumption 1 or ii) is an inner-product decision graph problem as in Definition 3. Further, the task has infinitely graph isomorphism-invariant data as in Assumption 3. Then, there exists an RGraphC as in Definition 2 with absolutely continuous randomness source (cf. Assumption 2) that is probabilistic G -invariant and universal for ( x , y ) (as in Theorem 2). Further, the number of parameters of this RGraphC will depend only on the smallest finite absolute moments of its weight and bias distributions, i.e., the ones given by f θ f ( u , u i ) and g θ g ( u , u b ) .

Proof of Theorem 3. Note that it suffices to show that there exists an RLC that takes as input graphs of size d and decides correctly tasks in Definition 3 with probability > 0 . 5 -this implies a universal limiting classifier

F θ . Then, we can replace the smoothness assumption in Theorem 2 and Proposition 1 with Definition 3 and we will have that there exists an infinitely jointly exchangeable sequence that is universal for tasks as in Definition 3. Finally, we follow Proposition 2's proof by simply replacing de Finetti's with Aldous-Hoover's theorem. Thus, let us now proceed to prove the main part of this theorem, i.e. , there exists an RLC that takes as input graphs of size d and decides correctly tasks in Definition 3 with probability > 0 . 5 .

Define an RLC that samples the linear coefficients as follows.

Let a ′ θ be a random variable such that supp( a ′ θ ) = S . It is important to note that we do not require that this random vector to have the same distribution as the input, or any other distribution on S . We simply need it to be supported on S .

Now, let t ∼ Bernoulli (0 . 5 + γ ) for some γ > 0 . We define a θ := t · a ′ θ and and b θ := t · b . Our random prediction can then be rewritten as sgn( t · ( ⟨ a ′ θ , x ⟩ -b )) .

Let us now calculate our probability of success. For convenience, let us define sgn(0) = 1 5 . Then, the probability of a positive graph x + , i.e. , y ( x + ) = 1 , being classified correctly is

while the probability of a negative graph x -, i.e. , y ( x -) = -1 , being classified correctly is

Then, since X is a finite set for any input distribution x there exists a constant γ where 0 < γ/ (0 . 5 + γ ) < P ( ⟨ a ′ θ , x ⟩ ≥ b ) for all x ∈ X . Given such gamma, every input has a probability of success greater than 0 . 5 , which implies that F θ ( x ) = y ( x ) , ∀ x ∈ X as we wanted to show.

B RLCs for spherical data

Here we want to show how the same ideas presented in Sections 3.1 and 3.2 can be applied to spherical data using Freedman's theorem [13]. More specifically, we let our input be d -dimensional vectors, i.e. , supp( x ) ⊆ R d and our task invariant to the orthogonal group, i.e. , y ( x ) = y ( g · x ) , ∀ x ∈ supp( x ) , ∀ g ∈ G , where G := O ( d ) . We refer to tasks like this as tasks over spherical data since it is invariant to rotations and reflections.

Just as de Finetti, Aldous and Hoover informed us about sets and graphs, Freedman did it for spherical data [13]. It follows from his work that an infinite sequence invariant to the action of the orthogonal group can be represented as a random scalar multiplying a sequence of i.i.d. standard Gaussian distributions. Leveraging this, we can define the following very simple invariant model for spherical data (assuming Assumption 3).

Definition 4 (Randomized Sphere Classifiers) . A Randomized Sphere Classifier (RSphereC) uses 2 neural networks f θ f : R 2 → R and g θ g : R 2 → R together with 3 sources of randomness: u , u a and u b . The random linear classifier coefficients are generated with

where [ N (0 , 1) , . . . N (0 , 1)] is a vector of d i.i.d. standard Gaussians.

Proposition 4 (Universality and resource consumption of RSphereCs) . Let ( x , y ) be a O ( d ) -invariant task with a smooth separator as in Assumption 1 and infinitely spherical data as in Assumption 3. Then, RSphereCs as in Definition 4 with absolutely continuous randomness source (cf. Assumption 2) are probabilistic G -invariant and universal for ( x , y ) (as in Theorem 2). Further, the number of parameters needed by RSphereCs in this task will depend only on the smallest finite absolute moments of the weight and bias distributions in the zero-error solution, i.e., the ones given by f θ ⋆ f ( u , u i ) and g θ ⋆ g ( u , u b ) .

Proof. The proof is exactly as the one from Proposition 2 but replacing de Finetti's theorem with Freedman's [13]. For a clean and more accessible material on Freedman's theorem, we refer the reader to the lectures by Kallenberg in https://mysite.science.uottawa.ca/givanoff/wskallenberg.pdf .

Finally, we note that tasks invariant to O ( n ) are not that common. However, tasks invariant to SO ( n ) are extremely relevant in computer vision, see [6]. Therefore, we think this model is a first step towards probabilistic invariance for SO ( n ) 6 .

C Implementation Details

As mentioned in the main text, all models were trained for a maximum of 1000 epochs using a cosine annealing scheduler for the learning. We tuned all the hyperparameters on the validation set using a patience of 30 . The Deep Sets models found a better learning rate of 0 . 001 and batch size of 250 . The GNN model found a better learning rate of 0 . 01 and batch size 100 . The RSetC model used a batch size of 250 and learning rate 0 . 5 . The RGraphC model used a batch size of 100 and a learning rate of 0 . 5 . Finally, we also note that RLCs can out put different answers given the same weights (due to the random input). Thus, we amplify each mini-batch with 1000 samples for each model -we noted that this reduces the variance and helps convergence. The amplification size used in test was m = 10 . Finally, for all RLCs we used a single dimensional standard normal distribution in each of their noise sources.

Source code is available at: https://github.com/cottascience/invariant-rlcs

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose probabilistic linear classifiers (RLCs) that are able to solve relatively general binary classification problems. They derive a trade-off between the probability that sampling of RLCs leads to an accurate prediction and the number of samples to obtain the majority vote of an RLC.\nFurthermore, there discuss more detailed results for three invariant learning problems related to inputs in form of sets, graphs, or spherical data.\n\nSTRENGTHS:\n- **Clarity:** The paper is well written and well understandable despite a strong theoretical component.\n\n- **Originality:** The authors propose to use probabilistic binary classifiers instead of deterministic ones to trade-off the number of required trainable parameters with prediction accuracy. A similar idea has been put forth by Sieradzki et al. (ICML22) for a number of interesting examples. This work seems to be the first to use randomized models for invariant learning (and relatively general binary classification problems).\n\n- **Significance:** The idea is significant because it discusses a novel problem solution perspective to overcome some computational barriers of the deterministic setting.\n\n- **Quality:** Mathematically rigorous analysis and exposition of the main idea. As a highlight, Theorem 1 gives a practically relevant estimate of the trade-off between required samples and the probability of accurate predictions.\nThe experiments could be designed in better support of the theory but still point out some potential benefits of RLCs for binary classification problems on sets and graphs.\n\n**Detailed points of strengths:**\n\n- The number of parameters used by RSetC does not depend on the set size, in contrast to DeepSets. Yet, the resource consumption depends on the smoothness of the distribution of linear classifier weights and bias. (Note that this might also imply disadvantages over DeepSets in some settings.)\n\n- RGraphCs can approximate any problem that either has a smooth boundary or can be tested with an inner product as in Definition 3. This includes the classification of graph connectivity (which cannot be solved by simple GNN architectures).\n\nWEAKNESSES:\n- The analysis is limited to binary classification problems.\n\n- The claimed computational resource savings are not obvious from the exposition and not directly covered by the theorems. According to the theory, the number of required parameters of RLCs could be higher than the ones of a deterministic solution (by a constant that could be very large). The theory does not establish that a smaller number of samples m (leading to potentially lower accuracy) would work in combination with fewer neural network parameters.\n(Some potential benefits for invariant learning are established based on reduced input dependence. But the changed criteria related to the smoothness of decision boundaries could also lead to worse results for RLCs in some situations.)\n\n- The experiments were not designed clearly to highlight computational advantages but to show accuracy improvements on small scale tasks, in which the baselines cannot perform reasonably well because of an unrealistic computational resource constraint.\nIt is likely that the baselines would actually achieve much higher accuracy with sufficient model capacity. Could RLCs actually perform at least on par if equipped with the same capacity that allow the baselines to work well?\n\n- The statement in Line 338: \"This comes from the fact that GNNs cannot distinguish simple non-isomorphic graphs such as any pair of d-regular graphs. As such, GNNs cannot universally approximate tasks that assign different labels to any of such pairs. In contrast, RGraphCs can approximate any problem that either has a smooth boundary or can be tested with an inner product...\"\nis not entirely correct. This limitation only applies to simple GNN architectures. Modern architectures that use attention and suitable node labels do not suffer from this problem. Experiments that compare with such stronger baselines would be more informative regarding the practical implications of the proposed work.\n\n- Conceptually, RLCs might require fewer trainable parameters in some cases, which could lead to advantageous memory requirements. Yet, they are not necessarily computationally more efficient at inference, which is not discussed at all in the paper. The fact that each inference needs to evaluate the trained neural network m times could increase the associated FLOPS significantly. \n\n- For the same reason, the experiments might not be fair, as the baselines are allowed fewer computational resources at inference than the RLCs.\n\n- **Reproducibility:** No code has been shared.\n\n\n**Points of minor critique:**\n\n- Line 297: $u_b$ is listed twice instead of once.\n\n- Figures: It would help to color also confidence intervals because it currently is impossible to distinguish them when they are overlapping.\n\n- The size of the validation set is not mentioned (neither in the main paper nor the Appendix). Only the size of the training and test set is given.\n\nQUESTIONS:\n- How many samples m did RsetC and RgraphC use in the experiments? This seems to be quite an important parameter to determine the maximum achievable accuracy and the required computational resources for the evaluation of RsetC and RgraphC.\n\n- Experimental set-up in support of story:\nI would like to understand the required computational resources in an experimental context that directly supports the theoretical claims. 1) Start from good baselines that have enough neural network capacity to actually solve the discussed problems. 2) Then design RLCs that achieve a similar accuracy and compare the used computational resources. In this context: How many FLOPS does the evaluation of RLCs cost? How many trainable parameters do they use?\n\nLIMITATIONS:\nThe limitations have been pointed out under weaknesses and were partially discussed by the authors. I do not foresee a major or immediate negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for achieving universality and invariance in binary classification tasks, while minimizing computational requirements. Instead of relying on deterministic neural networks such as DeepSet, which have parameterization complexity proportional to the set size, the paper proposes using randomised linear classifiers (RLCs). These RLCs can maintain invariance to compact group transformations and provide a universal approximation. Importantly, the parameterization complexity of RLCs remains independent of the set size. Building upon this finding, the paper extends the design of RLCs to ensure invariance in classification tasks involving sets, graphs, and spherical data. Experimental results demonstrate that the proposed RLCs outperform DeepSet and GNN approaches in certain invariant tasks.\n\nSTRENGTHS:\n•\tThe paper makes a notable theoretical contribution by presenting a novel approach that utilizes randomness to achieve universality and invariance in binary classification tasks. As far as my knowledge extends, this is the first approach of its kind to utilize randomness in this context.\n\n•\tThe incorporation of de Finetti's, Aldous-Hoover's, and Freedman's theorems is a significant contribution of the paper. These theorems play a crucial role in making RLCs applicable and practical for dealing with set, graph, and spherical data. By leveraging these theoretical foundations, the paper provides valuable insights into achieving invariance and universality across various data domains. This expands the scope of invariance and opens up new possibilities for addressing similar challenges in different contexts.\n\nWEAKNESSES:\nMy main concern pertains to the scalability of the proposed approach. While deterministic neural networks only require a single feed-forward process to produce an output, RLCs necessitate the sampling of multiple randomnesses to compute the linear weights. This introduces computational complexity during both training and testing phases. Consequently, I am particularly interested in understanding how RLCs perform on real-world tasks compared to deterministic models like DeepSet and GNN. It remains to be seen whether RLCs can deliver competitive results in practical scenarios, considering the additional computational demands imposed by their inherent stochastic nature.\n\nQUESTIONS:\n•\tWhat impact does the hyperparameter $m$ have on the model's performance? It appears that a larger value of $m$ is necessary to achieve highly confident predictions. This suggests that increasing $m$ allows the model to gather more evidence and make more precise decisions. It would be valuable to investigate how different values of $m$ influence the model's accuracy and confidence levels across various tasks and datasets.\n\n•\tThe impact of random distribution on model performance is a noteworthy aspect to consider. I am curious about whether different randomness would affect the model performance. What kind of randomness do you use in the experiment?\n\nLIMITATIONS:\nThe paper presents some notable limitations despite its promising theoretical results:\n\n•\tScalability: The applicability of the proposed RLCs appears to be challenging in large-scale scenarios compared to deterministic methods. Further investigations and enhancements are needed to improve scalability and address potential computational complexities.\n\n•\tUnclear impact of $m$ and randomness: The paper would benefit from additional discussions and experiments to explore the effects of the hyperparameter $m$ and the selection of randomness sources. This would provide better insights into the optimal choices and their influence on model performance.\n\n•\tLack of real-world tasks: The evaluation of the proposed method is limited to synthetic tasks, and it would be valuable to extend it to real-world datasets. This would allow for a more comprehensive understanding of the method's capabilities and potential applications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a very interesting method for training efficient invariant classifiers by leveraging randomness. More precisely, it proposes to train a neural network to sample linear classifier weights, by pushing forward some data-independent distribution, and using the majority vote over sampled classifiers to make predictions. It is rather a theoretical paper which proves universal approximation and G-invariance theorems for this class of classifiers (adapted to the probabilistic setting). The special cases of set and graph invariance are studied (with relaxed assumptions). Spherical data is also addressed in the appendix. A few toy experiments illustrate the theoretical results proved.\n\nSTRENGTHS:\n### Originality\n\nAlthough the paper borrows some ideas from CFNNs, it is largely original in its restriction to linear classifiers and focus on invariant representation learning.\n\n### Clarity\n\nThe paper is clearly written and well-structured.\n\n### Quality\n\nI read two out of three proofs from the supplementary material and they look fine to me. Toy experiments are well-designed.\n\n### Significance\n\nThis work addresses the important question of invariant representation learning with a very original method that has some benefits compared to existing methods.\n\nWEAKNESSES:\n### Originality\n\n1. The only remark I would have here is that there are a few references missing from the invariant representation learning paragraph of the related work section. Namely, there are methods which are quite different from the proposed one but which also make use of randomness to obtain invariant representations, such as Augerino (https://arxiv.org/abs/2010.11882) and AugNet (https://arxiv.org/abs/2202.02142).\n\n### Clarity\n\n2. The only remark I have is that the error bars on the figures are difficult to associate to each dot as they are not color-coded.\n\nTypos:\n\n- l.126: supp(x)\n- l. 297: “$u_b$, and $u_b$” ($u_b$ twice)\n\n### Quality\n\nI only have two remarks/questions regarding the method and experiments:\n\n3. I realize that the paper contribution is rather theoretical, but the experiments are very very toy. I realize the value of the experiments carried out, but I wonder whether it would be possible to showcase the method in a more realistic setting.\n4. As far as I understood, the model invariance relies on underlying theorems specific to each setting (de Finetti for sets, Aldous-Hoover for graphs and Freedman for spherical data). Is that right? It would be interesting if you could discuss to which extent your framework can be adapted to arbitrary symmetry groups and whether this is a limitation.\n\nQUESTIONS:\nSee questions above.\n\nLIMITATIONS:\nA few limitations are discussed throughout the paper, but not very thoroughly. A major limitation for now is that this is a very preliminary study showcasing the new method only on toy binary classification problems. Although one cannot expect it to address all research questions in a single paper, some directions on how the proposed approach could have practical use would be welcome.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors introduce Randomized Linear Classifiers, which are a way to randomly represent the weights of a linear classifier. Because there is a random reconstruction of the network, a sample majority of which is used to determine the actual inference of the network. The authors show a universal representation theorem for such a classifier, and show that the representation is quite efficient in some cases (for instance, sometimes the number of parameters needed does not scale with a relevant quantity, in which a deterministic network would scale).

STRENGTHS:
The notion that the number of parameters needed for $f_\theta$ can be much smaller than the number of parameters needed for a classifier that is a deterministic neural network is nice.

WEAKNESSES:
From what I can tell, the paper only considers the expressivity of RLCs, and the number of parameters needed. Is there a result about learning $f_\theta$?

QUESTIONS:
Typos in line 219, 287.

As a sanity check that I have understood this, you dont have access to the true gradients for $\theta$ until $m \to \infty$ right?

Is this similar to learning a one layer NN in which during training and inference some random subset of neurons is used each time (for forward and backward passes)?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a framework called Randomized Linear Classifiers (RLCs) that leverages external randomness to build models that are expressive and can encode invariance in the input space. The authors establish probabilistic versions of universal approximation theorem and invariance for several types of RLCs. The key insight from the theory is that by maintaining probabilistic universal approximation and invariance, RLCs can be more parameter-efficient than their deterministic counterparts. Numerical experiments verify the theoretic properties.\n\nSTRENGTHS:\n- The paper is generally well-written and explains the implications of the assumptions and theorems clearly.\n- RLCs extend the idea of Coin-Flipping Neural Network with stronger theoretic backgrounds and more general formulations and invariant extensions.\n- The probabilistic notions of universal approximation and invariance potentially open up a new direction.\n\nWEAKNESSES:\n- I am not so sure about the benefits of \"Online computation\" and \"Private computation\". What is the difference between standard online inference and RLCs? In the inference phase, the client downloads the model and computes the prediction without sending the inputs, so it is already private.\n- I am not sure how the external randomness would affect the generalization performance of RLCs. From the theorems, it seems that the only requirement for the randomness source is to be absolutely continuous, thus MLPs can universally approximates? In the experiments, the external randomness was fixed as standard normal. I assume that the distribution of the external randomness would greatly affect sample complexity? For example, RLCs may require more restarts to amplify the probability if a bad external randomness is used? I think this part needs some further discussion.\n- In the numerical experiments, the performance of DeepSets looks pretty bad, even worse than using a constant predictor. Is this normal? It would be more convincing if RLCs was examined on tasks that DeepSets performs good.\n\nQUESTIONS:\nIt would be interesting to see how the external randomness affects the generalization performance of RLCs.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}