SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Erik Lien Bolager+
Iryna Burak+
Chinmay Datar
Qing Sun
Felix Dietrich\*\*
$^{+}$ School of Computation, Information and Technology;  $^{†}$ Institute for Advanced Study
We introduce a probability distribution, combined with an efficient sampling algorithm, for weights and biases of fully-connected neural networks. In a supervised learning context, no iterative optimization or gradient computations of internal network parameters are needed to obtain a trained network. The sampling is based on the idea of random feature models. However, instead of a data-agnostic distribution, e.g., a normal distribution, we use both the input and the output training data to sample shallow and deep networks. We prove that sampled networks are universal approximators. For Barron functions, we show that the  $L^2$ -approximation error of sampled shallow networks decreases with the square root of the number of neurons. Our sampling scheme is invariant to rigid body transformations and scaling of the input data, which implies many popular pre-processing techniques are not required. In numerical experiments, we demonstrate that sampled networks achieve accuracy comparable to iteratively trained ones, but can be constructed orders of magnitude faster. Our test cases involve a classification benchmark from OpenML, sampling of neural operators to represent maps in function spaces, and transfer learning using well-known architectures.
Training deep neural networks involves finding all weights and biases. Typically, iterative, gradient-based methods are employed to solve this high-dimensional optimization problem. Randomly sampling all weights and biases before the last, linear layer circumvents this optimization and results in much shorter training time. However, the drawback of this approach is that the probability distribution of the parameters must be chosen. Random projection networks [54] or extreme learning machines [30] involve weight distributions that are completely problem- and data-agnostic, e.g., a normal distribution. In this work, we introduce a data-driven sampling scheme to construct weights and biases close to gradients of the target function (cf. Figure 1). This idea provides a solution to three main challenges that have prevented randomly sampled networks to compete successfully against iterative training in the setting of supervised learning: deep networks, accuracy, and interpretability.
Figure 1: Random feature models choose weights in a data-agnostic way, compared to sampling them where it matters: at large gradients. The arrows illustrate where the network weights are placed.
*Corresponding author, felix.dietrich@tum.de.
Deep neural networks. Random feature models and extreme learning machines are typically defined for networks with a single hidden layer. Our sampling scheme accounts for the high-dimensional ambient space that is introduced after this layer, and thus deep networks can be constructed efficiently.
Approximation accuracy. Gradient-based, iterative approximation can find accurate solutions with a relatively small number of neurons. Randomly sampling weights using a data-agnostic distribution often requires thousands of neurons to compete. Our sampling scheme takes into account the given training data points and function values, leading to accurate and width-efficient approximations. The distribution also leads to invariance to orthogonal transformations and scaling of the input data, which makes many common pre-processing techniques redundant.
Interpretability. Sampling weights and biases completely randomly, i.e., without taking into account the given data, leads to networks that do not incorporate any information about the given problem. We analyze and extend a recently introduced weight construction technique [27] that uses the direction between pairs of data points to construct individual weights and biases. In addition, we propose a sampling distribution over these data pairs that leads to efficient use of weights; cf. Figure 1.
Regarding random Fourier features, Li et al. [41] and Liu et al. [43] review and unify theory and algorithms of this approach. Random features have been used to approximate input-output maps in Banach spaces [50] and solve partial differential equations [16, 48, 10]. Gallicchio and Scardapane [28] provide a review of deep random feature models, and discuss autoencoders and reservoir computing (resp. echo-state networks [34]). The latter are randomly sampled, recurrent networks to model dynamical systems [6]. Regarding construction of features, Monte Carlo approximation of data-dependent parameter distributions is used towards faster kernel approximation [1, 59, 47]. Our work differs in that we do not start with a kernel and decompose it into random features, but we start with a practical and interpretable construction of random features and then discuss their approximation properties. This may also help to construct activation functions similar to collocation [62]. Fiedler et al. [23] and Fornasier et al. [24] prove that for given, comparatively small networks with one hidden layer, all weights and biases can be recovered exactly by evaluating the network at specific points in the input space. The work of Spek et al. [60] showed a certain duality between weight spaces and data spaces, albeit in a purely theoretical setting. Recent work from Bollt [7] analyzes individual weights in networks by visualizing the placement of ReLU activation functions in space.
Regarding approximation errors and convergence rates of networks, Barron spaces are very useful [2, 20], also to study regularization techniques, esp. Tikhonov and Total Variation [40]. A lot of work [54, 19, 17, 57, 65] surrounds the approximation rate of  $\mathcal{O}(m^{-1/2})$  for neural networks with one hidden layer of width  $m$ , originally proved by Barron [2]. The rate, but not the constant, is independent of the input space dimension. This implies that neural networks can mitigate the curse of dimensionality, as opposed to many approximation methods with fixed, non-trained basis functions [14], including random feature models with data-agnostic probability distributions. The convergence rates of over-parameterized networks with one hidden layer is considered in [19], with a comparison to the Monte Carlo approximation. In our work, we prove the same convergence rate for our networks. Regarding deep networks, E and Wojtowitsch [17, 18] discuss simple examples that are not Barron functions, i.e., cannot be represented by shallow networks. Shallow [15] and deep random feature networks [29] have also been analyzed regarding classification accuracy. Regarding different sampling techniques, Bayesian neural networks are prominent examples [49, 5, 26, 61]. The goal is to learn a good posterior distribution and ultimately express uncertainty around both weights and the output of the network. These methods are computationally often on par with or worse than iterative optimization. In this work, we directly relate data points and weights, while Bayesian neural networks mostly employ distributions only over the weights. Generative modeling has been proposed as a way to sample weights from existing, trained networks [56, 51]. It may be interesting to consider our sampled weights as training set in this context. In the lottery ticket hypothesis [25, 8], "winning" subnetworks are often not trained, but selected from a randomly initialized starting network, which is similar to our approach. Still, the score computation during selection requires gradient updates. Most relevant to our work is the weight construction method by Galaris et al. [27], who proposed to use pairs of data points to construct weights. Their primary goal was to randomly sample weights that capture low-dimensional structures. No further analysis was provided, and only a uniform distribution over the data pairs was used. We expand and analyze their setting here.
3 Mathematical framework
We introduce sampled networks, which are neural networks where each pair of weight and bias of all hidden layers is completely determined by two points from the input space. This duality between weights and data has been shown theoretically [60], here, we provide an explicit relation. The weights are constructed using the difference between the two points, and the bias is the inner product between the weight and one of the two points. After all hidden layers are constructed, we must only solve an optimization problem for the coefficients of a linear layer, mapping the output from the last hidden layer to the final output. We start to formalize this construction by introducing some notation.
Let  $\mathcal{X} \subseteq \mathbb{R}^D$  be the input space with  $\|\cdot\|$  being the Euclidean norm with inner product  $\langle \cdot, \cdot \rangle$ . Further, let  $\Phi$  be a neural network with  $L$  hidden layers, parameters  $\{W_l, b_l\}_{l=1}^{L+1}$ , and activation function  $\phi: \mathbb{R} \to \mathbb{R}$ . For  $x \in \mathcal{X}$ , we write  $\Phi^{(l)}(x) = \phi(W_l \Phi^{(l-1)}(x) - b_l)$  as the output of the  $l$ th layer, with  $\Phi^{(0)}(x) = x$ . The two activation functions we focus on are the rectified linear unit (ReLU),  $\phi(x) = \max\{x, 0\}$ , and the hyperbolic tangent (tanh). We set  $N_l$  to be the number of neurons in the  $l$ th layer, with  $N_0 = D$  and  $N_{L+1}$  as the output dimension. We write  $w_{l,i}$  for the  $i$ th row of  $W_l$  and  $b_{l,i}$  for the  $i$ th entry of  $b_l$ . Building upon work of Galaris et al. [27], we now introduce sampled networks. The probability distribution to sample pairs of data points is arbitrary here, but we will refine it in Definition 2. We use  $\mathcal{L}$  to denote the loss of our network we would like to minimize.
Definition 1. Let  $\Phi$  be a neural network with  $L$  hidden layers. For  $l = 1, \ldots, L$ , let  $(x_{0,i}^{(1)}, x_{0,i}^{(2)})_{i=1}^{N_l}$  be pairs of points sampled over  $\mathcal{X} \times \mathcal{X}$ . We say  $\Phi$  is a sampled network if the weights and biases of every layer  $l = 1, 2, \ldots, L$  and neurons  $i = 1, 2, \ldots, N_l$ , are of the form
$$
w _ {l, i} = s _ {1} \frac {x _ {l - 1 , i} ^ {(2)} - x _ {l - 1 , i} ^ {(1)}}{\| x _ {l - 1 , i} ^ {(2)} - x _ {l - 1 , i} ^ {(1)} \| ^ {2}}, \quad b _ {l, i} = \langle w _ {l, i}, x _ {l - 1, i} ^ {(1)} \rangle + s _ {2}, \tag {1}
$$
where  $s_1, s_2 \in \mathbb{R}$  are constants,  $x_{l-1,i}^{(j)} = \Phi^{(l-1)}(x_{0,i}^{(j)})$  for  $j = 1,2$ , and  $x_{l-1,i}^{(1)} \neq x_{l-1,i}^{(2)}$ . The last set of weights and biases are  $W_{L+1}, b_{L+1} = \arg \min \mathcal{L}(W_{L+1} \Phi^{(L)}(\cdot) - b_{L+1})$ .
The constants  $s_1, s_2$  are used to fix what values the activation function takes on when it is applied to the points  $x^{(1)}, x^{(2)}$ ; cf. Figure 2. For ReLU, we set  $s_1 = 1$  and  $s_2 = 0$ , so that  $\phi(x^{(1)}) = 0$  and  $\phi(x^{(2)}) = 1$ . For tanh, we set  $s_1 = 2s_2$  and  $s_2 = \ln(3)/2$ , which implies  $\phi(x^{(1)}) = 1/2$  and  $\phi(x^{(2)}) = -1/2$ , respectively, and  $\phi\left(1/2\left(x^{(1)} + x^{(2)}\right)\right) = 0$ . This means that in a regression problem with ReLU, we linearly interpolate values between the two points. For classification, the tanh construction introduces a boundary if  $x^{(1)}$  belongs to a different class than  $x^{(2)}$ . We will use this idea later to define a useful distribution over pairs of points (cf. Definition 2).
Figure 2: Placement of the point pairs  $x^{(1)}, x^{(2)}$  for activation functions ReLU (left) and tanh (right). Two data pairs are chosen in each subfigure, resulting in two activation functions on each data domain.
The space of functions that sampled networks can approximate is not immediately clear. First, we are only using points in the input space to construct both the weights and the biases, instead of letting them take on any value. Second, there is a dependence between the bias and the direction of the weight. Third, for deep networks, the sampling space changes after each layer. These apparent restrictions require investigation into which functions we can approximate. We assume that the input space in Theorem 1 and Theorem 2 extends into its ambient space  $\mathbb{R}^D$  as follows. Let  $\mathcal{X}'$  be any compact subset of  $\mathbb{R}^D$  with finite reach  $\tau > 0$ . Informally, such a set has a boundary that does not change too quickly [12]. We then set the input space  $\mathcal{X}$  to be the space of points including  $\mathcal{X}'$
and those that are at most  $\epsilon_I$  away from  $\mathcal{X}'$ , given the canonical distance function in  $\mathbb{R}^D$ , where  $0 < \epsilon_I < \tau$ . In Theorem 1, we also consider  $L$  layers to show that the construction of weights in deep networks does not destroy the space of functions that networks with one hidden layer can approximate, even though we alter the space of weights we can construct when  $L > 1$ .
Theorem 1. For any number of layers  $L \in \mathbb{N}_{>0}$ , the space of sampled networks with  $L$  hidden layers,  $\Phi \colon \mathcal{X} \to \mathbb{R}^{N_{L+1}}$ , with activation function  $ReLU$ , is dense in  $C(\mathcal{X}, \mathbb{R}^{N_{L+1}})$ .
Sketch of proof: We show that the possible biases that sampled networks can produce are all we need inside a neuron, and the rest can be added in the last linear part and with additional neurons. We then show that we can approximate any neural network with one hidden layer with at most  $6 \cdot N_{1}$  neurons — which is not much, considering the cost of sampling versus backpropagation. We then show that we can construct weights so that we preserve the information of  $\mathcal{X}$  through the first  $L - 1$  layers, and then we use the arbitrary width result applied to the last hidden layer. The full proof can be found in Appendix A.1. We also prove a similar theorem for networks with tanh activation function and one hidden layer. The proof differs fundamentally, because tanh is not positive homogeneous.
We now show existence of sampled networks for which the  $L^2$  approximation error of Barron functions is bounded (cf. Theorem 2). We later demonstrate that we can actually construct such networks (cf. Section 4.1). The Barron space [2, 20] is defined as
$$
\mathcal {B} = \{f \colon f (x) = \int_ {\Omega} w _ {2} \phi (\langle w _ {1}, x \rangle - b) d \mu (b, w _ {1}, w _ {2}) \text {a n d} \| f \| _ {\mathcal {B}} <   \infty \}
$$
with  $\phi$  being the ReLU function,  $\Omega = \mathbb{R} \times \mathbb{R}^D \times \mathbb{R}$ , and  $\mu$  being a probability measure over  $\Omega$ . The Barron norm is defined as  $\|f\|_{\mathcal{B}} = \inf_{\mu} \max_{(b, w_1, w_2) \in \operatorname{supp}(\mu)} \{|w_2|(|w_1|_1 + |b|)\}$ .
Theorem 2. Let  $f \in \mathcal{B}$  and  $\mathcal{X} = [0,1]^D$ . For any  $N_1 \in \mathbb{N}_{>0}$ ,  $\epsilon > 0$ , and an arbitrary probability measure  $\pi$ , there exist sampled networks  $\Phi$  with one hidden layer,  $N_1$  neurons, and ReLU activation function, such that
$$
\| f - \Phi \| _ {2} ^ {2} = \int_ {\mathcal {X}} | f (x) - \Phi (x) | ^ {2} d \pi (x) <   \frac {(3 + \epsilon) \| f \| _ {\mathcal {B}} ^ {2}}{N _ {1}}.
$$
Sketch of proof: It quickly follows from the results of E et al. [20], which showed it for regular neural networks, and Theorem 1. The full proof can be found in Appendix A.2.
Up until now, we have been concerned with the space of sampling networks, but not with the distribution of the parameters. We found that putting emphasis on points that are close and differ a lot with respect to the output of the true function works well. As we want to sample layers sequentially, and neurons in each layer independently from each other, we define a layer-wise conditional definition underneath. The norms  $\| \cdot \|_{\mathcal{X}_{l - 1}}$  and  $\| \cdot \| _y$ , that defines the following densities, are arbitrary over their respective space, denoted by the subscript.
Definition 2. Let  $f\colon \mathbb{R}^D\to \mathbb{R}^{N_{L + 1}}$  be Lipschitz-continuous and set  $\mathcal{Y} = f(\mathcal{X})$ . For any  $l\in \{1,2,\ldots ,L\}$ , setting  $\epsilon = 0$  when  $l = 1$  and otherwise  $\epsilon >0$ , we define
$$
q _ {l} ^ {\epsilon} \left(x _ {0} ^ {(1)}, x _ {0} ^ {(2)} \mid \{W _ {j}, b _ {j} \} _ {j = 1} ^ {l - 1}\right) = \left\{ \begin{array}{l l} \frac {\| f \left(x _ {0} ^ {(2)}\right) - f \left(x _ {0} ^ {(1)}\right) \| _ {\mathcal {Y}}}{\max  \{\| x _ {l - 1} ^ {(2)} - x _ {l - 1} ^ {(1)} \| _ {\mathcal {X} _ {l - 1}}, \epsilon \}}, & x _ {l - 1} ^ {(1)} \neq x _ {l - 1} ^ {(2)} \\ 0, & o t h e r w i s e, \end{array} \right. \tag {2}
$$
where  $x_0^{(1)}, x_0^{(2)} \in \mathcal{X}$ ,  $x_{l-1}^{(1)} = \Phi^{(l-1)}(x_0^{(1)})$ , and  $x_{l-1}^{(2)} = \Phi^{(l-1)}(x_0^{(2)})$ , with the network  $\Phi^{(l-1)}$  parameterized by sampled  $\{W_j, b_j\}_{j=1}^{l-1}$ . Then, using  $\lambda$  as the Lebesgue measure, we define the integration constant  $C_l = \int_{\mathcal{X} \times \mathcal{X}} q_l^\epsilon d\lambda$ . The density  $p_l^\epsilon$  to sample pairs of points for weights and biases in layer  $l$  is equal to  $q_l^\epsilon / C_l$  if  $C_l > 0$ , and uniform over  $\mathcal{X} \times \mathcal{X}$  otherwise.
Note that here, a distribution over pair of points is equivalent to a distribution over weights and biases, and the additional  $\epsilon$  is a regularization term. Now we can sample for each layer sequentially, starting with  $l = 1$ , using the conditional density  $p_l^\epsilon$ . This induces a probability distribution  $P$  over the full parameter space, which, with the given regularity conditions on  $\mathcal{X}$  and  $f$ , is a valid probability distribution. For a complete definition of  $P$  and proof of validity, see Appendix A.3.
Using this distribution also comes with the benefit that sampling and training are not affected by rigid body transformations (affine transformation with orthogonal matrix) and scaling, as long as the true function  $f$  is equivariant w.r.t. to the transformation. That is, if  $H$  is such a transformation, we say  $f$  is equivariant with respect to  $H$ , if there exists a scalar and rigid body transformation  $H'$  such that  $H'(f(x)) = f(H(x))$  for all  $x \in \mathcal{X}$ , and invariant if  $H'$  is the identity function. We also assume that norms  $\| \cdot \|_{\mathcal{Y}}$  and  $\| \cdot \|_{\mathcal{X}_0}$  in Equation (2) are chosen such that orthogonal matrix multiplication is an isometry.
Theorem 3. Let  $f$  be the target function and equivariant w.r.t. to a scalar and rigid body transformation  $H$ . If we have two sampled networks,  $\Phi, \hat{\Phi}$ , with the same number of hidden layers  $L$  and neurons  $N_{1}, \ldots, N_{L}$ , where  $\Phi: \mathcal{X} \to \mathbb{R}^{N_{L+1}}$  and  $\hat{\Phi}: H(\mathcal{X}) \to \mathbb{R}^{N_{L+1}}$ , then the following statements hold for all  $x \in \mathcal{X}$ :
(1) If  $\hat{x}_{0,i}^{(1)} = H(x_{0,i}^{(1)})$  and  $\hat{x}_{0,i}^{(2)} = H(x_{0,i}^{(2)})$  for all  $i = 1,2,\ldots ,N_1$ , then  $\Phi^{(1)}(x) = \hat{\Phi}^{(1)}(H(x))$ .
(2) If  $f$  is invariant w.r.t.  $H$ , then for any parameters of  $\Phi$ , there exist parameters of  $\hat{\Phi}$  such that  $\Phi(x) = \hat{\Phi}(H(x))$ , and vice versa.
(3) The probability measure  $P$  over the parameters is invariant under  $H$ .
Sketch of proof: Any neuron in the sampled network can be written as  $\phi (\langle s_1\frac{w}{\|w\|^2},x - x^{(1)}\rangle -s_2)$ . As we divide by the square of the norm of  $w$ , the scalar in  $H$  cancels. There is a difference between two points in both inputs of  $\langle \cdot ,\cdot \rangle$ , which means the translation cancels. Orthogonal matrices cancel due to isometry. When  $f$  is invariant with respect to  $H$ , the loss function is also unchanged and lead to the same output. Similar argument is made for  $P$ , and the theorem follows (cf. Appendix A.3).
If the input is embedded in a higher-dimensional ambient space  $\mathbb{R}^{D'}$ , with  $D < D'$ , we sample from a subspace with dimension  $\tilde{D} = \dim(\overline{\operatorname{span}\{\mathcal{X}\}}) \leq D'$ . All the results presented in this section still hold due to orthogonal decomposition. However, the standard approach of backpropagation and initialization allows the weights to take on any value in  $\mathbb{R}^{D'}$ , which implies a lot of redundancy when  $\tilde{D} \ll D'$ . The biases are also more relevant to the input space than when initialized to zero — potentially avoiding the issues highlighted by Holzmüller and Steinwart [32]. For these reasons, we have named the proposed method Sampling Where It Matters (SWIM), which is summarized in Algorithm 1. For computational reasons, we consider a random subset of all possible pairs of training points when sampling weights and biases.
We end this section with a time and memory complexity analysis of Algorithm 1. In Table 1, we list runtime and memory usage for three increasingly strict assumptions. The main assumption is that the dimension of the output is less than or equal to the largest dimension of the hidden layers. This is true for the problems we consider, and the difference in runtime without this assumption is only reflected in the linear optimization part. The term  $\lceil N / M \rceil$ , i.e., integer ceiling of  $N / M$ , is required because only a subset of points are considered when sampling. For the full analysis, see Appendix F.
We now demonstrate the performance of Algorithm 1 on numerical examples. Our implementation is based on the numpy and scipy Python libraries, and we run all experiments on a machine with 32GB system RAM (256GB in Section 4.3 and Section 4.4) and a GeForce 4x RTX 3080 Turbo GPU with 10GB RAM. The Appendix contains detailed information on all experiments. In Section 4.1 we compare sampling to random Fourier feature models regarding the approximation of a Barron function. In Section 4.2 we compare classification accuracy of sampled networks to iterative, gradient-based optimization in a classification benchmark with real datasets. In Section 4.3 we demonstrate that more specialized architectures can be sampled, by constructing deep neural architectures as PDE solution operators. In Section 4.4 we show how to use sampling of fully-connected layers for transfer learning. For the probability distribution over the pairs in Algorithm 1, we always choose the  $L^{\infty}$  norm for  $\| \cdot \|_y$  and for  $l = 1, 2, \ldots, L$ , we choose the  $L^2$  norm for  $\| \cdot \|_{\mathcal{X}_{l-1}}$ . The code to reproduce the experiments from the paper, and an up-to-date code base, can be found at
https://gitlab.com/felix.dietrich/swimnetworks-paper,
https://gitlab.com/felix.dietrich/swimnetworks.