SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: The SWIM algorithm, for activation function  $\phi$ , and norms on input, output of the hidden layers, and output space,  $\| \cdot \|_{\mathcal{X}_0}, \| \cdot \|_{\mathcal{X}_l}$ , and  $\| \cdot \|_{\mathcal{Y}}$  respectively. Also,  $\mathcal{L}$  is a loss function, which in our case is always  $L^2$  loss, and arg  $\min \mathcal{L}(\cdot, \cdot)$  becomes a linear optimization problem.
Constant:  $\epsilon \in \mathbb{R}_{>0},\varsigma \in \mathbb{N}_{>0},L\in \mathbb{N}_{>0},\{N_l\in \mathbb{N}_{>0}\}_{l = 1}^{L + 1}$  , and  $s_1,s_2\in \mathbb{R}$    
Data:  $X = \{x_{i}\colon x_{i}\in \mathbb{R}^{D},i = 1,2,\ldots ,M\}$ $Y = \{y_{i}\colon f(x_{i}) = y_{i}\in \mathbb{R}^{N_{L + 1}},i = 1,2,\ldots ,M\}$ $\Phi^{(0)}(x) = x$  .   
for  $l = 1,2,\dots ,L$  do   
 $\tilde{M}\gets \varsigma \cdot \left[\frac{N_l}{M}\right]\cdot M$  ..   
 $P^{(l)}\in \mathbb{R}^{\tilde{M}};P_{i}^{(l)}\gets 0\quad \forall i;$ $\tilde{X} = \{(x_i^{(1)},x_i^{(2)})\colon \Phi^{(l - 1)}(x_i^{(1)})\neq \Phi^{(l - 1)}(x_i^{(2)})\}_{i = 1}^{\tilde{M}}\sim \mathrm{Uniform}(X\times X);$    
for  $i = 1,2,\ldots ,\tilde{M}$  do   
 $\tilde{x}_i^{(1)},\tilde{x}_i^{(2)}\gets \Phi^{(l - 1)}(x_i^{(1)}),\Phi^{(l - 1)}(x_i^{(2)})$ $\tilde{y}_i^{(1)},\tilde{y}_i^{(2)} = f(x_i^{(1)}),f(x_i^{(2)})$ $P_{i}^{(l)}\gets \frac{\|\tilde{y}_{i}^{(2)} - \tilde{y}_{i}^{(1)}\|_{\mathcal{V}}}{\max\{\|\tilde{x}_{i}^{(2)} - \tilde{x}_{i}^{(1)}\|_{X_{l - 1}},\epsilon\}}$    
end   
 $W_{l}\in \mathbb{R}^{N_{l},N_{l - 1}},b_{l}\in \mathbb{R}^{N_{l}}$  .   
for  $k = 1,2,\ldots ,N_l$  do Sample  $(x^{(1)},x^{(2)})$  from  $\tilde{X}$  , with replacement and with probability proportional to  $P^{(l)}$  .  $\tilde{x}^{(1)},\tilde{x}^{(2)}\gets \Phi^{(l - 1)}(x^{(1)}),\Phi^{(l - 1)}(x^{(2)})$ $W_{l}^{(k,:)}\gets s_{1}\frac{\tilde{x}^{(2)} - \tilde{x}^{(1)}}{\|\tilde{x}^{(2)} - \tilde{x}^{(1)}\|^2};b_l^{(k)}\gets \langle W_l^{(k,:)},\tilde{x}^{(1)}\rangle +s_2;$    
end   
 $\Phi^{(l)}(\cdot)\gets \phi (W_l\Phi^{(l - 1)}(\cdot) - b_l)$    
end   
 $W_{L + 1},b_{L + 1}\gets \arg \min L(\Phi^{(L)}(X),Y);$    
return  $\{W_l,b_l\}_{l = 1}^{L + 1}$
Table 1: Runtime and memory requirements for training sampled neural networks with the SWIM algorithm, where  $N = \max \{N_0,N_1,N_2,\dots ,N_L\}$ . Assumption (I) is that the output dimension is less than or equal to  $N$ . Assumption (II) adds that  $N < M^2$ , i.e., number of neurons and input dimension is less than the size of dataset squared. Assumption (III) requires a fixed architecture.
Runtime	Memory
Assumption (I)	O(L·M(min{[N/M], M} + N2))	O(M·min{[N/M], M} + LN2)
Assumption (II)	O(L·M([N/M] + N2))	O(M·[N/M] + LN2)
Assumption (III)	O(M)	O(M)
4.1 Illustrative example: approximating a Barron function
We compare random Fourier features and our sampling procedure on a test function for neural networks [64]:  $f(x) = \sqrt{3/2}(\|x - a\| - \|x + a\|)$ , with  $x \in \mathbb{R}^D$  and the vector  $a \in \mathbb{R}^D$  defined by  $a_j = 2j/D - 1, j = 1, 2, \ldots, D$ . The Barron norm of  $f$  is equal to one for all input dimensions, and it can be represented exactly with a network with one infinitely wide hidden layer, ReLU activation, and weights uniformly distributed on a sphere of radius  $D^{1/2}$ . We approximate  $f$  using networks  $\hat{f}$  of up to three hidden layers. The error is defined by  $e_{\mathrm{rel}}^2 = \sum_{x \in \mathcal{X}} (f(x) - \hat{f}(x))^2 / \sum_{x \in \mathcal{X}} f(x)^2$ . We compare this error over the domain  $\mathcal{X} = [-1, 1]^2$ , with 10,000 points sampled uniformly, separately for training and test sets. For random features, we use  $w \sim N(0, 1)$ ,  $b \sim U(-\pi, \pi)$ , as proposed in [54], and  $\phi = \sin \theta$ . For sampling, we also use  $\phi = \sin \theta$  to obtain a fair comparison. We also observed similar accuracy results when repeating the experiment with the tanh function. The number of neurons  $m$  is the same in each hidden layer and ranges from  $m = 64$  up to  $m = 4096$ . Figure 3 shows results for  $D = 5, 10$  (results are similar for  $D = 2, 3, 4$ , and sampled networks can be constructed as fast as the random feature method, cf. Appendix B). Random features here have
Figure 3: Relative  $L^2$  approximation error of a Barron function (test set), using random features and sampling, both with sine activation. Left: input dimension  $D = 5$ . Right: input dimension  $D = 10$ .
comparable accuracy for networks with one hidden layer, but very poor performance for deeper networks. This may be explained by the much larger ambient space dimension of the data after it is processed through the first hidden layer. With our sampling method, we obtain accurate results even with more layers. The convergence rate for  $D < 10$  seems to be faster than the theoretical rate.
4.2 Classification benchmark from OpenML
We use the "OpenGL-CC18 Curated Classification benchmark" [4] with all its 72 tasks to compare our sampling method to the Adam optimizer [36]. With both methods, we separately perform neural architecture search, changing the number of hidden layers from 1 to 5. All layers always have 500 neurons. Details of the training are in Appendix C. Figure 4 shows the benchmark results. On all tasks, sampling networks is faster than training them iteratively (on average, 30 times faster). The classification accuracy is comparable (cf. Figure 4, second and third plot). The best number of layers for each problem is slightly higher for the Adam optimizer (cf. Figure 4, fourth plot).
Figure 4: Fitting time, accuracy, and number of layers using weight sampling, compared to training with the Adam optimizer. The best architecture is chosen separately for each method and each problem, by evaluating 10-fold cross-validation error over 1-5 layers with 500 neurons each.
4.3 Deep neural operators
We sample deep neural operators and compare their speed and accuracy to iterative gradient-based training of the same architectures. As a test problem, we consider Burgers' equation,  $\frac{\partial u}{\partial t} + u\frac{\partial u}{\partial x} = \nu \frac{\partial^2 u}{\partial^2 x}, x \in (0, 2\pi), t \in (0, 1]$ , with periodic boundary conditions and viscosity  $\nu = 0.1$ . The goal is to predict the solution at  $t = 1$  from the initial condition at  $t = 0$ . Thus, we construct neural operators that represent the map  $\mathcal{G}: u(x, 0) \to u(x, 1)$ . We generate initial conditions by sampling five Fourier coefficients of lowest frequency and restoring the function values from these coefficients. Using a classical numerical solver, we generate 15000 pairs of  $(u(x, 0), u(x, 1))$ , and split them into the train  $(60\%)$ , validation  $(20\%)$ , and test sets  $(20\%)$ . Figure 5 shows samples from the generated dataset.
4.3.1 Fully-connected network in signal space
The first baseline for the task is a fully-connected network (FCN) trained with tanh activation to predict the discretized solution from the discretized initial condition. We trained the classical version using the Adam optimizer and the mean squared error as a loss function. We also performed early stopping based on the mean relative  $L^2$ -error on the validation set. For sampling, we use Algorithm 1 to construct a fully-connected network with tanh as the activation function.
4.3.2 Fully-connected network in Fourier space
Similarly to Poli et al. [53], we train a fully-connected network in Fourier space. For training, we perform a Fourier transform on the initial condition and the solution, keeping only the lowest frequencies. We always split complex coefficients into real and imaginary parts, and train a standard FCN on the transformed data. The reported metrics are in signal space, i.e., after inverse Fourier transform. For sampling, we perform exactly the same pre-processing steps.
4.3.3 POD-DeepONet
The third architecture considered here is a variation of a deep operator network (DeepONet) architecture [44]. The original DeepONet consists of two trainable components: the trunk net, which transforms the coordinates of an evaluation point, and the branch net, which transforms the function values on some grid. The outputs of these nets are then combined into the predictions of the whole network  $\mathcal{G}(u)(y) = \sum_{k=1}^{p} b_k(u)t_k(y) + b_0$ , where  $u$  is a discretized input function;  $y$  is an evaluation point;  $[t_1,\dots,t_p]^T \in \mathbb{R}^p$  are the  $p$  outputs of the trunk net;  $[b_1,\dots,b_p]^T \in \mathbb{R}^p$  are the  $p$  outputs of the branch net; and  $b_0$  is a bias. DeepONet sets no restrictions on the architecture of the two nets, but often fully-connected networks are used for one-dimensional input. POD-DeepONet proposed by Lu et al. [46] first assumes that evaluation points lie on the input grid. It performs proper orthogonal decomposition (POD) of discretized solutions in the train data and uses its components instead of the trunk net to compute the outputs  $\mathcal{G}(u)(\xi_j) = \sum_{k=1}^{p} b_k(u)\psi_k(\xi_j) + \psi_0(\xi_j)$ . Here  $[\psi_1(\xi_j),\dots,\psi_p(\xi_j)]$  are  $p$  precomputed POD components for a point  $\xi_j$ , and  $\psi_0(\xi_j)$  is the mean of discretized solutions evaluated at  $\xi_j$ . Hence, only the branch net is trained in POD-DeepONet. We followed Lu et al. [46] and applied scaling of  $1/p$  to the network output. For sampling, we employ orthogonality of the components and turn POD-DeepONet into a fully-connected network. Let  $\xi = [\xi_1,\dots,\xi_n]$  be the grid used to discretize the input function  $u$  and evaluate the output function  $\mathcal{G}(u)$ . Then the POD components of the training data are  $\Psi(\xi) = [\psi_1(\xi),\dots,\psi_p(\xi)] \in \mathbb{R}^{n\times p}$ . If  $b(u) \in \mathbb{R}^p$  is the output vector of the trunk net, the POD-DeepONet transformation can be written  $\mathcal{G}(u)(\xi) = \Psi b(u) + \psi_0(\xi)$ . As  $\Psi^T\Psi = I_p$ , we can express the output of the trunk net as  $b(u) = \Psi^T(\mathcal{G}(u)(\xi) - \psi_0(\xi))$ . Using this equation, we can transform the training data to sample a fully-connected network for  $b(u)$ . We again use tanh as the activation function for sampling.
4.3.4 Fourier Neural Operator
The concept of a Fourier Neural Operator (FNO) was introduced by Li et al. [42] to represent maps in function spaces. An FNO consists of Fourier blocks, combining a linear operator in the Fourier space and a skip connection in signal space. As a first step, FNO lifts an input signal to a higher dimensional channel space. Let  $v_{t} \in \mathbb{R}^{n \times d_{v}}$  be an input to a Fourier block having  $d_{v}$  channels and discretized with  $n$  points. Then, the output  $v_{t+1}$  of the Fourier block is computed as  $v_{t+1} = \phi(F_{k}^{-1}(R \cdot F_{k}(v_{t})) + W \circ v_{t}) \in \mathbb{R}^{n \times d_{v}}$ . Here,  $F_{k}$  is a discrete Fourier transform keeping only
Figure 5: Left: Samples of initial conditions  $u_{0}$  and corresponding solutions  $u_{1}$  for Burgers' equation. Right: Parameters of the best model for each architecture, the mean relative  $L^2$  error on the test set, and the training time. We average the metrics across three runs with different random seeds.
Model	width	layers	mean rel. \( L^2 \) error	Time
Adam	FCN; signal space	1024	2	\( 4.48 \times 10^{-3} \)	644s	Op
FCN; Fourier space	1024	1	\( 3.29 \times 10^{-3} \)	1725s
POD-DeepONet	2048	4	\( 1.62 \times 10^{-3} \)	4217s
FNO	n/a	4	\( 0.38 \times 10^{-3} \)	3119s
Sampling	FCN; signal space	4096	1	\( 0.92 \times 10^{-3} \)	20s	Op
FCN; Fourier space	4096	1	\( 1.08 \times 10^{-3} \)	16s
POD-DeepONet	4096	1	\( 0.85 \times 10^{-3} \)	21s
FNO	4096	1	\( 0.94 \times 10^{-3} \)	387s
the  $k$  lowest frequencies and  $F_{k}^{-1}$  is the corresponding inverse transform;  $\phi$  is an activation function;  $\cdot$  is a spectral convolution;  $\circ$  is a  $1\times 1$  convolution with bias; and  $R\in \mathbb{C}^{k\times d_v\times d_v}$ ,  $W\in \mathbb{R}^{d_v\times d_v}$  are learnable parameters. FNO stacks several Fourier blocks and then projects the output signal to the target dimension. The projection and the lifting operators are parameterized with neural networks. For sampling, the construction of convolution kernels is not possible yet, so we cannot sample FNO directly. Instead, we use the idea of FNO to construct a neural operator with comparable accuracy. Similar to the original FNO, we normalize the input data and append grid coordinates to it before lifting. Then, we draw the weights from a uniform distribution on  $[-1,1]$  to compute the  $1\times 1$  lifting convolution. We first apply the Fourier transform to both input and target data, and then train a fully-connected network for each channel in Fourier space. We use skip connections, as in the original FNO, by removing the input data from the lifted target function during training, and then add it before moving to the output of the block. After sampling and transforming the input data with the sampled networks, we apply the inverse Fourier transform. After the Fourier block(s), we sample a fully-connected network that maps the signal to the solution.
The results of the experiments in Figure 5 show that sampled models are comparable to the Adam-trained ones. The sampled FNO model does not directly follow the original FNO architecture, as we are able to only sample fully-connected layers. This shows the advantage of gradient-based methods: as of now, they are applicable to much broader use cases. These experiments showcase one of the main advantages of sampled networks: speed of training. We run sampling on the CPU; nevertheless, we see a significant speed-up compared to Adam training performed on GPU.
4.4 Transfer learning
Training deep neural networks from scratch involves finding a suitable neural network architecture [21] and hyper-parameters [3, 66]. Transfer learning aims to improve performance on the target task by leveraging learned feature representations from the source task. This has been successful in image classification [35], multi-language text classification, and sentiment analysis [63, 9]. Here, we compare the performance of sampling with iterative training on an image classification transfer learning task. We choose the CIFAR-10 dataset [39], with 50000 training and 10000 test images. Each image has dimension  $32 \times 32 \times 3$  and must be classified into one of the ten classes. We consider ResNet50 [31], VGG19 [58], and Xception [11], all pre-trained on the ImageNet dataset [55]. We freeze the weights of all convolutional layers and append one fully connected hidden layer and one output layer. We refer to these two layers as the classification head, which we sample with Algorithm 1 and compare the classification accuracy against iterative training with the Adam optimizer.
Figure 6 (left) shows that for a pre-trained ResNet50, the test accuracy using the sampling approach is higher than the Adam training approach for a width greater than 1024. We observe similar qualitative behavior for VGG19 and Xception (figures in Appendix E). Figure 6 (middle) shows that the sampling approach results in a higher test accuracy with all three pre-trained models. Furthermore, the deviation in test accuracy obtained with the sampling algorithm is very low, demonstrating that sampling is more robust to changing random seeds than iterative training. After fine-tuning the whole neural network with the Adam optimizer with a learning rate of  $10^{-5}$ , the test accuracies of sampled networks are close to the iterative approach. Thus, sampling provides a good starting point for fine-tuning the entire model. A comparison for the three models before and after fine-tuning is contained in Appendix E. Figure 6 (right) shows that sampling is up to two orders of magnitude faster than iterative training for
Figure 6: Left: Train and test accuracies with different widths for ResNet50 (averaged over 5 random seeds). Middle: Test accuracy with different models with and without fine-tuning (width = 2048). Right: Adam training and sampling times of the classification head (averaged over 5 experiments).
smaller widths, and around ten times faster for a width of 2048. In summary, Algorithm 1 is much faster than iterative training, yields a higher test accuracy for certain widths before fine-tuning, and is more robust with respect to changing random seeds. The sampled weights also provide a good starting point for fine-tuning of the entire model.
Sampling weights through data pairs at large gradients of the target function offers improvements over random feature models. In terms of accuracy, networks with relatively large widths can even be competitive to iterative, gradient-based optimization. Constructing the weights through pairs of points also allows to sample deep architectures efficiently. Sampling networks offers a straightforward interpretation of the internal weights and biases, namely, which data pairs are important. Given the recent critical discussions around fast advancement in artificial intelligence, and calls to slow it down, publishing work that potentially speeds up the development (concretely, training speed) in this area by orders of magnitude may seem irresponsible. The solid mathematical underpinning of random feature models and, now, sampled networks, combined with much greater interpretability of the individual steps during network construction, should mitigate some of these concerns.
We present a data-driven sampling method for fully-connected neural networks that outperforms random feature models in terms of accuracy, and in many cases is competitive to gradient-based optimization. The time to obtain a trained network is orders of magnitude faster compared to gradient-based optimization. In addition, much fewer hyperparameters need to be optimized, as opposed to learning rate, number of training epochs, and type of optimizer.
Several open issues remain, we list the most pressing here. Many architectures like convolutional or transformer networks cannot be sampled with our method yet, and thus must still be trained with iterative methods. Implicit problems, such as the solution to PDE without any training data, are a challenge, as our distribution over the data pairs relies on known function values from a supervised learning setting. Iteratively refining a random initial guess may prove useful here. On the theory side, convergence rates for Algorithm 1 beyond the default Monte-Carlo estimate are not available yet, but are important for robust applications in engineering.
In the future, hyperparameter optimization, including neural architecture search, could benefit from the fast training time of sampled networks. We already demonstrate benefits for transfer learning here, which may be exploited for other pre-trained models and tasks. Analyzing which data pairs are sampled during training may help to understand the datasets better. We did not show that our sampling distribution results in optimal weights, so there is a possibility of even more efficient heuristics. Applications in scientific computing may benefit most from sampling networks, as accuracy and speed requirements are much higher than for many tasks in machine learning.
We are grateful for discussions with Erik Bollt, Constantinos Siettos, Edmilson Roque Dos Santos, Anna Veselovska, and Massimo Fornasier. We also thank the anonymous reviewers at NeurIPS for their constructive feedback. E.B., I.B., and F.D. are funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - project no. 468830823, and also acknowledge association to DFG-SPP-229. C.D. is partially funded by the Institute for Advanced Study (IAS) at the Technical University of Munich. F.D. and Q.S. are supported by the TUM Georg Nemetschek Institute - Artificial Intelligence for the Built World.
[53] Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Ré, and Stefano Ermon. Transform once: Efficient operator learning in frequency domain. In Advances in Neural Information Processing Systems, volume 35, pages 7947-7959, 2022.
[54] Ali Rahimi and Benjamin Recht. Uniform approximation of functions with random bases. In 46th Annual Allerton Conference on Communication, Control, and Computing, pages 555-561. IEEE, 2008. ISBN 978-1-4244-2925-7. doi: 10.1109/ALLERTON.2008.4797607.
[55] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
[56] Konstantin Schürholt, Boris Knyazev, Xavier Giro-i Nieto, and Damian Borth. Hyper-representations as generative models: Sampling unseen neural network weights. In Advances in Neural Information Processing Systems, volume 35, pages 27906-27920, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b2c4b7d34b3d96b9dc12f7bce424b7ae-Paper-Conference.pdf.
[57] Jonathan W. Siegel and Jinchao Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313-321, 2020. ISSN 08936080. doi: 10.1016/j.neunet.2020.05.019.
[58] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. (arXiv:1409.1556), 2014.
[59] Sho Sonoda. Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models. (arXiv:1902.00648), 2020.
[60] Len Spek, Tjeerd Jan Heeringa, and Christoph Brune. Duality for neural networks through Reproducing Kernel Banach Spaces. (arXiv:2211.05020), 2022.
[61] Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. In International Conference on Learning Representations, 2019.
[62] Michael Unser. A representative theorem for deep neural networks. Journal of Machine Learning Research, 20(110):1-30, 2019. URL http://jmlr.org/papers/v20/18-418.html.
[63] Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 3(1):9, 2016. ISSN 2196-1115. doi: 10.1186/s40537-016-0043-6.
[64] Stephan Wojtowytsch and Weinan E. Can Shallow Neural Networks Beat the Curse of Dimensionality? A Mean Field Training Perspective. IEEE Transactions on Artificial Intelligence, 1(2):121-129, 2020. ISSN 2691-4581. doi: 10.1109/TAI.2021.3051357.
[65] Lei Wu and Jihao Long. A Spectral-Based Analysis of the Separation between Two-Layer Neural Networks and Linear Methods. Journal of Machine Learning Research, 23(1), 2022. ISSN 1532-4435. doi: 10.5555/3586589.3586708.
[66] Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295-316, 2020.
Here we include full proofs of all theorems in the paper, and then details on all numerical experiments. In the last section, we briefly explain the code repository (URL in section Appendix F) and provide the analysis of the algorithm runtime and memory complexity.
A Sampled networks theory 1
A.1 Universality of sampled networks 3
A.1.1 Input space 4
A.1.2 Networks with a single hidden layer 5
A.1.3 Deep networks 9
A.1.4 Networks with a single hidden layer, tanh activation 11
A.2 Barron spaces 14
A.3 Distribution of sampled networks 15
B Illustrative example: approximating a Barron function 18
C Classification benchmark from OpenGL 18
D Deep neural operators 18
D.1 Dataset 18
D.2 Experiment setup 21
D.3 Results 22
E Transfer learning 23
E.1 Sampling Vs Adam training 24
E.2 Sampling with tanh Vs ReLU activation functions 24
E.3 Fine-tuning 24
E.4 One vs two hidden layers in the classification head 25
F Code repository and algorithm complexity 26
A Sampled networks theory
In this section, we provide complete proofs of the results stated in the main paper. In Appendix A.1 we consider sampled networks and the universal approximation property, followed by the result bounding  $L_{2}$  approximation error between sampled networks and Barron functions in Appendix A.2. We end with the proof regarding equivariance and invariance in Appendix A.3.