SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Leonardo Cotta
leonardo.cotta@vectorinstitute.ai
Gal Yehuda
Technion, Haifa, Israel
ygal@cs.technion.ac.il
Assaf Schuster
assaf@technion.ac.il
Chris J. Maddison
cmaddis@cs.toronto.edu
Designing models that are both expressive and preserve known invariances of tasks is an increasingly hard problem. Existing solutions tradeoff invariance for computational or memory resources. In this work, we show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, our key insight is that accepting probabilistic notions of universal approximation and invariance can reduce our resource requirements. More specifically, we propose a class of binary classification models called Randomized Linear Classifiers (RLCs). We give parameter and sample size conditions in which RLCs can, with high probability, approximate any (smooth) function while preserving invariance to compact group transformations. Leveraging this result, we design three RLCs that are provably probabilistic invariant for classification tasks over sets, graphs, and spherical data. We show how these models can achieve probabilistic invariance and universality using less resources than (deterministic) neural networks and their invariant counterparts. Finally, we empirically demonstrate the benefits of this new class of models on invariant tasks where deterministic invariant neural networks are known to struggle.
A modern challenge in machine learning is designing model classes for invariant tasks that are both universal and resource-efficient. Consider designing an architecture for graph problems that is invariant under graph isomorphism. Ensuring that this class is universal is at least as hard as solving the graph isomorphism problem [5]. Similarly, permutation invariance universality comes at the cost of memory—the number of parameters scales with the sequence size [32].
Using randomness as a computational resource for algorithms is a fundamental idea in computer science. In their essence, randomized algorithms often provide simple solutions that use less memory or run in less time than their deterministic counterparts [30]. Since there is no free lunch, these gains come at the cost of accepting a probabilistic notion of correctness, i.e., while deterministic algorithms always output correct answers, their randomized versions will be right only with high probability. Inspired by this, the key insight in our paper is using an external source of randomness to reduce time and space resources when learning an invariant task. To benefit from that, just as in randomized algorithms, we also need to introduce a probabilistic notion of invariance and expressive power. By guaranteeing invariance and universal approximation only with high probability, we show when it is possible to avoid the invariance-resource tradeoff.
We introduce a class of models for binary classification tasks named Randomized Linear Classifiers (RLCs). In an RLC the external source of randomness is input to a neural network that generates a random linear classifier—independently from the input—which is then used to make a prediction. To guarantee correctness with high probability, we need to repeat this process for a sufficient number of times, i.e., take multiple samples, and predict the majority of their output—this is called amplification. The key features of RLCs are both i) randomness independent from input and ii) linear transformation of the input. Combined, these features provide both theoretical and practical simplicity. On the practical side, RLCs can offload computation and preserve privacy in resource-constrained devices. We highlight the benefits of RLCs in such applications below.
a. Online computation. Suppose that our goal is to answer classification queries as quickly as possible, or that the queries arrive in an "online" fashion. By using RLCs, we can sample several linear classifiers "offline" (before observing new inputs), and then in the online phase simply take the majority of a few (different) linear classifiers for each input. This way, the majority of computations can be done offline rather than online—as usual in existing online learning settings.
b. Private computation. Suppose that we wish to deform inference with an RLC, but we only have a low-resource computer (e.g., a smartwatch). A practical way of doing so is to perform the sampling on a remote server, then retrieve the answer. In our approach, the server can send the randomness, i.e., the sampled linear classifier coefficients. There is no communication between the client and the server at all, and the client never sends the input — retaining its privacy. Moreover, since the client only needs to perform a few linear computations, it can be a low-resource computer.
The theoretical advantages of RLCs' features are explored in this work. We consider the required resources of an RLC to be universal and invariant for a given task as the number of parameters used by its neural network and the sufficient number of samples (amplification size). Then, we focus on establishing upper-bounds for these resources in different scenarios. More specifically, our contributions are three-fold:
i) We introduce the general class of Randomized Linear Classifiers (RLCs), presenting a universal approximation theorem coupled with a resource consumption characterization (Theorem 1). In specific, we show that any binary classification problem with a smooth boundary can be approximated with high probability by RLCs with at most the same number of parameters as a deterministic neural network. This is a result of general purpose and holds for any invariant or non-invariant task.
ii) We show how the problem of designing RLCs invariant to compact group transformations can be cast to designing invariant distributions of linear classifiers. As a result, we can leverage representation theorems in probability theory, such as de Finetti's [9], Aldous-Hoover's [1] and Friedman's [13] to design invariant RLCs for tasks with set and graph data. Moreover, in Appendix B we also show a simple invariant model for spherical data using these ideas.
iii) Finally, we show both theoretically and empirically how invariant RLCs can succeed in tasks where deterministic architectures, such as Deep Sets and GNNs, cannot efficiently approximate. By introducing probabilistic notions of universality and invariance, our work establishes the first alternative to avoid the invariance-resource tradeoff with theoretical guarantees and empirical success.
Here, we briefly review Coin-Flipping Neural Networks (CFNNs) and invariant representation learning—two core concepts that we build upon. Finally, we explain how Bayesian neural networks and RLCs belong to different learning paradigms.
Coin-flipping neural networks. CFNNs were presented by [27], where the authors propose several deep learning architectures using randomness as part of their inference. A few (constructive) examples were shown in which randomness can help by reducing the number of parameters or the depth of the network. In this paper we focus on the case of learning randomized linear classifiers and its use in invariant learning. The invariant models were not explored in [27],
and to the best of our knowledge this is the first work studying randomized models for invariant learning. One specific CFNN model proposed in [27] is the Hypernetwork CFNN. In this model, the weights of a model are first randomly generated. Then, the computation of a deterministic neural network using these random weights takes place. In this work, we focus on the "extreme" case of this paradigm: we allow our model to use arbitrarily complex distributions to generate our final weights, but we insist that the deterministic part —using such weights—would be a linear function. That is, we push the computational complexity to the random part of the model. This gives rise to the class of models we study: Randomized Linear Classifiers (RLCs). The work of [27] considered a first version of RLCs, but did not provide any empirical results and their theoretical results hold only for single dimensional input (i.e., in  $\mathbb{R}$ ). Hence, to the best of our knowledge our work also establishes the first (general purpose) universal approximation theorem of (non-invariant) RLCs (cf. Section 2).
Invariant representation learning. Over the last years, there has been a growing interest in designing  $G$ -invariant architectures [4, 6, 18, 35]. In summary, we can often leverage an a priori knowledge that the function to be learned is invariant to the action of a group  $G$  by forcing our model to also be invariant to it. That is,  $G$ -invariant architectures incorporate a known propriety of the target. Such a reduced model space is known to both empirically and theoretically achieve better generalization [4, 6, 11, 19]. Existing works study the  $G$ -invariant function space and how to parameterize it with neural networks. Despite some empirical success, invariance guarantees often come at unreasonable prices. For instance, CNNs are guaranteed to be invariant to translations only with infinite grids (images) [19]. Regarding permutation invariance, Deep Sets is always invariant, but to remain universal it requires a hidden layer as large as the input set [32]. Finally, GNNs, as any other computationally efficient graph isomorphism-invariant architecture, cannot achieve universal approximation. More specifically, achieving graph isomorphism invariance and universal approximation is at least as hard as solving the graph isomorphism problem [5]. In contrast to these results, our work shows that such tradeoffs are usually restricted to the design of deterministic architectures. By accepting probabilistic notions of invariance and universality, we show that under mild data generation conditions we can circumvent these invariance-resource tradeoffs.
Bayesian neural networks. Bayesian Neural Networks (BNNs) incorporate Bayesian inference into both training and inference of neural networks. In short, BNNs explicitly assign a prior to the parameters and perform inference with a posterior. BNNs differ from RLCs in how randomness is used. BNNs explicitly model prior and posterior distributions over the weights. This is usually leveraged in the context of confidence estimation. On the other hand, RLCs do not model a prior or a posterior distribution over the parameters. Rather, RLCs take samples from a model that uses external randomness. In their essence, BNNs and RLCs are trying to achieve different goals. While BNNs are modeling uncertainty over decisions, RLCs are trying to output the correct target with the highest probability they can.
2 Randomized Linear Classifiers
We start by introducing the general class of Randomized Linear Classifiers. Due to our universal approximation result, this can be seen as the analogous of a multi-layer perceptron in the context of randomized linear models. In the next sections we will introduce the analogous of  $G$ -invariant models.
For notation simplicity, we denote the random variable of a value with bold letters, e.g., a random variable  $\mathbf{x}$  would have a realization  $x$ . We consider binary classification tasks where the input data  $\mathbf{x}$  is supported on  $\mathcal{X} := \operatorname{supp}(\mathbf{x}) \subseteq \mathbb{R}^d$  and labeled according to a true labeling function  $y \colon \mathcal{X} \to \{-1, 1\}$ . Therefore, a binary classification task can be summarized by the tuple  $(\mathbf{x}, y)$ .
Given a source of randomness  $\mathbf{u}$  supported on  $\mathcal{U} := \operatorname{supp}(\mathbf{u}) \subseteq \mathbb{R}^{d_u}$ , and a neural network  $f_{\theta} \colon \mathbb{R}^{d_u} \to \mathbb{R}^{d + 1}$  parameterized by  $\theta \in \mathbb{R}^d$ , an RLC predicts  $y(x)$  for  $x \in \mathcal{X}$  according to
$$
\operatorname {s g n} \left(\left\langle \mathbf {a} _ {\theta}, x \right\rangle - \mathbf {b} _ {\theta}\right) := \operatorname {s g n} \left(\left\langle f _ {\theta} (\mathbf {u}) _ {1: d}, x \right\rangle - f _ {\theta} (\mathbf {u}) _ {d + 1}\right). \tag {1}
$$
From above, one can see the coefficients  $(\mathbf{a}_{\theta},\mathbf{b}_{\theta})$  of the linear classifier given by the pushforward measure  $f_{\theta}\# \mathbf{u}$ . Finally, unless otherwise stated, we assume  $f_{\theta}$  to be a multi-layer perceptron with ReLU activations.
Analogously to randomized algorithms for decision problems, we can take multiple samples using Equation (1) and make a final prediction using their majority. To simplify notation, let us define the function of our final prediction taking  $m$  samples as
$$
\mathbf {y} _ {\theta} ^ {(m)} (x) := \operatorname {m a j} \left(\left\{\operatorname {s g n} \left(\left\langle \mathbf {a} _ {\theta} ^ {(j)}, x \right\rangle - \mathbf {b} _ {\theta} ^ {(j)}\right) \right\} _ {j = 1} ^ {m}\right) = \operatorname {m a j} \left(\left\{\operatorname {s g n} \left(\left\langle f _ {\theta} \left(\mathbf {u} ^ {(j)}\right) _ {1: d}, x \right\rangle - f _ {\theta} \left(\mathbf {u} ^ {(j)}\right) _ {d + 1}\right) \right\} _ {j = 1} ^ {m}\right), \tag {2}
$$
where each  $\mathbf{u}^{(j)}\sim_{\mathrm{i.i.d}}P(\mathbf{u})$ . Note that as we take more samples, we approach the mode of the distribution of linear predictions for a given  $x$ . We define this value as the limiting classification of  $x$  which implies the existence of the limiting classifier  $\mathcal{F}_{\theta}\colon \mathbb{R}^d\to \{-1,1\}$  with
$$
\mathcal {F} _ {\theta} (x) := \lim  _ {m \rightarrow \infty} \operatorname {m a j} \left(\left\{\operatorname {s g n} \left(\left\langle \mathbf {a} _ {\theta} ^ {(j)}, x \right\rangle - \mathbf {b} _ {\theta} ^ {(j)}\right)\right\} _ {j = 1} ^ {m}\right) = \underset {\hat {y} \in \{- 1, 1 \}} {\operatorname {a r g m a x}} P \left(\operatorname {s g n} \left(\left\langle \mathbf {a} _ {\theta}, x \right\rangle - \mathbf {b} _ {\theta}\right) = \hat {y}\right). \tag {3}
$$
At this point, it is worth noting that although the random prediction (Equation (1)) is a linear function, its limiting classifier (Equation (3)) can induce an arbitrarily non-linear boundary. In fact, we leverage this notion to define the probabilistic version of universality. We say that an RLC is universal in a probabilistic sense if its limiting classifier (Equation (3)) is universal. It is easy to see that models making deterministic predictions collapse the notions of probabilistic and exact universality. On the other hand, for RLCs to be probabilistic universal it suffices to produce predictions that are biased towards the desired answer on every input, i.e.,  $P(\mathbf{y}_{\theta}^{(m)}(x) = y(x)) > 0.5, \forall x \in \mathcal{X}$ .
In practice, we will make random predictions and therefore we need to capture a sufficient sample size  $m$  such that we are close to the limiting classifier with high probability. An important measure to understand how large  $m$  needs to be is what we call the minimum bias of the RLC. We denote it by  $\varepsilon$  and define it as the infimum of the total variation distances between the random predictions of every input  $x$  and a random variable  $\mathbf{r}$  with Radamacher distribution. That is,
$$
\varepsilon := \inf  \{d _ {\mathrm {V D}} (\mathbf {y} _ {\theta} ^ {(m)} (x), \mathbf {r}) \colon x \in \mathcal {X} \}.
$$
Put into words,  $\varepsilon$  captures a lower-bound on how close to a random prediction the RLC can be. And, naturally, the closer it is to a random prediction the larger  $m$  needs to be such that the majority converges to the limiting prediction. Therefore, our universal approximation result needs to capture how many parameters an RLC needs to converge to correct answers and how fast, i.e., with respect to  $m$ , it does. Before we proceed with our results on  $m$ ,  $p$  and the universality property, we need to define the general class of binary classification tasks we consider.
Assumption 1 (Tasks with smooth separators). We say that a binary classification task  $(\pmb{x},y)$  admits a smooth separator if there exists a neural network  $s\colon \mathbb{R}^d\to \mathbb{R}$  such that  $y(x) = \operatorname {sgn}(s(x)),\forall x\in$ $\mathrm{supp}(X)$
Note that due to the universal approximation ability of neural networks, the set of tasks satisfying Assumption 1 is quite large. Moreover, Assumption 1 is a sufficient but not necessary condition. We will later prove universality for a wide class of graph problems and we believe many others exist. For now, let us finally state our result on the resource consumption and universal approximation of RLCs in tasks with smooth separators.
Assumption 2 (Absolutely continuous randomness source). We say that the randomness source  $\mathbf{u}$  is absolutely continuous if at least one of its coordinates  $\mathbf{u}_i$ ,  $1 \leq i \leq d_u$  has an absolutely continuous marginal distribution.
Theorem 1 (Resource consumption of universal RLCs). Let  $(x, y)$  be a binary classification task that admits a smooth separator as in Assumption 1. Then, there exists an RLC with neural network  $f_{\theta^*}$  and absolutely continuous randomness source  $\mathbf{u}$  (Assumption 2) that is universal in the limit, i.e.,
$$
\mathcal {F} _ {\theta^ {*}} (x) = y (x), \forall x \in \mathcal {X},
$$
and makes random predictions that are correct with probability
$$
P (\mathrm {m a j} (\{\mathrm {s g n} (\langle \pmb {a} _ {\theta^ {\star}} ^ {(j)}, x \rangle - \pmb {b} _ {\theta^ {\star}} ^ {(j)} \} _ {j = 1} ^ {m}) = y (x)) > 1 - \exp \{- 2 \epsilon^ {2} m ^ {2} \},
$$
where  $\epsilon$  is the minimum bias of  $\mathcal{F}_{\theta^{\star}}$ .
Further, if  $p^\dagger$  is the number of parameters used by a deterministic neural network with one hidden layer to achieve zero-error in the task,  $f_{\theta}$  has at most
$$
p \leq p ^ {\dagger} + \mathcal {O} (1) \text {p a r a m e t e r s}.
$$
The complete proof is in Appendix A Note that Theorem 1 does not assume or explore any invariance in the task. It is a general universality result that we will leverage in our study of invariant RLCs, but of independent interest. One way to interpret Theorem 1 is as a probabilistic version of the classical universal approximation theorem of multi-layer perceptrons [8]. The result on the number of parameters is important to make it clear that RLCs do not need to find a solution more complex than one given by a deterministic model in the supervised learning task.
Until now, we have established the expressive power of RLCs, i.e., that, under mild assumptions, they can be as expressive and resource-efficient as deterministic neural networks. But, there is still the general question: When are RLCs more resource-efficient than deterministic neural networks? In [27] it was constructed an specific task where an RLC (although the authors do not name it as such) uses a constant number of parameters, while a deterministic neural network would need a number of parameters that grows with the input dimension. Here, we are interested in more general settings. In the next section we will show a large class of invariant tasks where invariant RLCs are more resource-efficient than their invariant deterministic neural network counterparts.
3  $G$ -invariant Randomized Linear Classifiers
Now, we turn our focus to binary classification tasks that are invariant to compact group transformations. That is, let  $G$  be a compact group with an action of  $g \in G$  on  $x \in \mathcal{X}$  denoted by  $g \cdot x$ . A task  $(\mathbf{x}, y)$  is  $G$ -invariant if  $y(x) = y(g \cdot x), \forall x \in \mathcal{X}, \forall g \in G$ . It is easy to see that an RLC that perfectly classifies the task in the probabilistic notion, i.e.,  $\mathcal{F}_{\theta^{\star}}(x) = y(x), \forall x \in \mathcal{X}$ , will also be probabilistic  $G$ -invariant, i.e.,  $\mathcal{F}_{\theta^{\star}}(x) = \mathcal{F}_{\theta^{\star}}(g \cdot x), \forall x \in \mathcal{X}, \forall g \in G$ . Thus, if we know that a task is  $G$ -invariant, can we restrict our model class to  $G$ -invariant RLCs? At first, the answer is not obvious. We need a design principle that guarantees a search space with only  $G$ -invariant solutions while at least one of them has zero error. In Theorem 2 we present the result that guides our design of invariant RLCs.
Theorem 2 (G-invariant RLCs). Let  $(x,y)$  be a  $G$ -invariant task with a smooth separator as in Assumption 1. Then, the set of RLCs with a  $G$ -invariant distribution in the classifier weights, i.e.,
$$
\boldsymbol {a} _ {\theta} \stackrel {{d}} {{=}} g \cdot \boldsymbol {a} _ {\theta}, \forall g \in G,
$$
and absolutely continuous randomness source (cf. Assumption 2) is both probabilistic  $G$ -invariant and universal in  $(x, y)$ . That is,
$$
\mathcal {F} _ {\theta} (x) = \mathcal {F} _ {\theta} (g \cdot x), \forall x \in \mathcal {X}, \forall g \in G, \forall \theta \in \mathbb {R} ^ {p},
$$
$$
\exists \theta^ {\star} \in \mathbb {R} ^ {p}: \mathcal {F} _ {\theta^ {\star}} (x) = y (x), \forall x \in \mathcal {X}, \forall g \in G.
$$
The complete proof is in Appendix A From above, we now know that designing universal RLCs for invariant tasks can be cast to the problem of designing universal  $G$ -invariant distributions—together with a separate (possibly dependent) bias distribution. This design principle differs fundamentally from deterministic invariant representation learning. Here, instead of designing architectures that are  $G$ -invariant, we wish to design architectures that induce a  $G$ -invariant distribution. That is, the RLC neural network  $f_{\theta}$  is not itself  $G$ -invariant.
Next, we will define a very useful assumption about the data generation process. It will allow us to leverage representation results in probability theory to both design universal and efficient  $G$ -invariant distributions and allow for variable-size set and graph data.
Assumption 3 (Infinitely  $G$ -invariant data). Let  $G_{\infty}$  be a homomorphism of  $G$  into the composition of homomorphisms of  $G$  in all finite dimensions, e.g., if  $G$  is the set of permutations of  $\{1,\ldots ,d\}$ ,  $G_{\infty}$  is the set of permutations of  $\mathbb{N}$ . We say that a task  $(x,y)$  has infinitely  $G$ -invariant data if there exists an infinite sequence of random variables  $(\pmb {x}_i)_{i = 1}^{\infty}\stackrel {d}{=}g_{\infty}\cdot (\pmb {x}_i)_{i = 1}^{\infty},\forall g_{\infty}\in G_{\infty}$ , where  $\pmb {x}\stackrel {d}{=}(\pmb {x}_i)_{i\in S}$  for  $S\in \binom{\mathbb{N}}{d}$ , with  $\binom{\mathbb{N}}{d}$  being the set of all  $d$ -size subsets of  $\mathbb{N}$ .
Although the above might seem unintuitive, the representation theorems also help us make sense of them. The data generation process will take the same form as our linear weights distribution. As we will see, for sets it means that items are sampled i.i.d. given a common latent factor of the set [9]. In graphs edges are also generated i.i.d. given latent factors of their endpoints and the graph [1].
Proposition 1 (Infinitely  $G$ -invariant RLCs). Let  $(x,y)$  be a  $G$ -invariant task with infinitely  $G$ -invariant data (Assumption 3) with a smooth separator as in Assumption 1. Then, the set of RLCs with an infinitely  $G$ -invariant distribution in the linear classifier weights, i.e., as in Assumption 3  $(\pmb{a}_{\theta i})_{i=1}^{\infty} \stackrel{d}{=} g_{\infty} \cdot (\pmb{a}_{\theta i})_{i=1}^{\infty}, \forall g_{\infty} \in G_{\infty}$ , where  $\pmb{a}_{\theta} \stackrel{d}{=} (\pmb{a}_{\theta i})_{i \in S}$ , for  $S \in \binom{\mathbb{N}}{d}$ , and absolutely continuous randomness source (cf. Assumption 2) is probabilistic  $G$ -invariant and universal for  $(x,y)$  as in Theorem 2.
The complete proof is in Appendix A Next, we design  $G$ -invariant RLCs drawing from results in the representation theory of probability distributions. We consider tasks with set and graph data. In Appendix B we also consider spherical data. For each, we derive conditions for  $G$ -invariance and universality, highlighting their resource gain when compared to their deterministic counterpart.
3.1 RLCs for set data