SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Adam training was done on the GeForce 4x RTX 3080 Turbo GPU with 10 GB RAM, while sampling was performed on the Intel Core i7-7700 CPU @ 3.60GHz × 8 with 32GB RAM.
We use all 72 datasets in the "OpenML-CC18 Curated Classification benchmark" from the "OpenML Benchmarking Suites" (CC-BY) [4], which is available on openml.org: https://openml.org/search?type=benchmark&sort=tasks_included&study_type=task&id=99.
We use the OpenML Python API (BSD-3 Clause) [22] to download the benchmark datasets. The hyperparameters used in the study are listed in Table 3. From all datasets, we use at most 5000 points. This was done for all datasets before applying any sampled or gradient-based approach, because we wanted to reduce the training time when using the Adam optimizer. We pre-process the input features using one-hot encoding for categorical variables, and robust whitening (removal of mean, division by standard deviation using the RobustScaler class from scikit-learn). Missing features are imputed with the median of the feature. All layers of all networks always have 500 neurons. For Adam, we employ early stopping with patience  $= 3$ , monitoring the loss value. The least squares regularization constant for sampling is  $10^{-10}$ . We split the data sets for 10-fold cross-validation using stratified k-fold (StratifiedKFold class from scikit-learn), and report the average of the accuracy scores and fit times over the ten folds (Figure 4 in the main paper).
D Deep neural operators
To generate an initial condition  $u_{0}$ , we first sample five Fourier coefficients for the lowest frequencies. We sample both real and imaginary parts from a normal distribution with zero mean and standard
Figure 8: Relative  $L^2$  approximation error for random features (using sine activation) and sampling (left: using tanh activation, right: using sine activation), with input dimensions  $d = 1, 2, 3, 4, 5$  and 10. Results are averaged over five runs with different random seeds.
deviation equal to five. We then scale the  $k$ -th coefficient by  $1 / (k + 1)^2$  to create a smoother initial condition. For the first coefficient corresponding to the zero frequency, we also set the imaginary part to zero as our initial condition should be real-valued. Then, we use real inverse Fourier transform with orthonormal normalization (np. fft. irrft) to generate an initial condition with the
Figure 9: Fit times in seconds for random features (using sine activation) and sampling (left: using tanh activation, right: using sine activation), with input dimensions  $d = 1,2,3,4,5$  and 10. Results are averaged over five runs with different random seeds.
discretization over 256 grid points. To solve the Burgers' equation, we use a classical numerical solver (scipy.integrate.solve_ivp) and obtain a solution  $u_{1}$  at  $t = 1$ . This way, we generate 15000 pairs of  $(u_0,u_1)$  and split them between train (9000 pairs), validation (3000 pairs), and test sets (3000 pairs).
Table 3: Network hyperparameters used in the OpenML benchmark study.
Parameter	Value
Shared	Number of layers	[1,2,3,4,5]
Layer width	500
Activation	tanh
Sampling	L2-regularization	10-5
Adam	Learning rate	10-3
Max. epochs	100
Early stopping patience	3
Batch size	64
Loss	mean-squared error
Table 4: Network hyperparameters used to train deep neural operators.
Parameter	Values	Note
Shared	Number of layers	[1,2,4]
Layer width	[256,512,1024,2048,4096]
Number of modes	8,16,32
Number of channels	16,32
Activation	tanh
Sampling	L2-regularization	10-10,10-8,10-6
Adam	Learning rate	10-5,5·10-5,10-4,5·10-4	FCNs and POD-DeepONet
10-4,5·10-4,10-3,5·10-3	FNO
Weight decay	0
Max. epochs	2000	FNO, FCN in signal space
5000	FCN in Fourier space
90000	POD-DeepONet
Patience	100	FNO, FCN in signal space
200	FCN in Fourier space
4500	POD-DeepONet
Batch size	64	FCNs and FNO
full data	POD-DeepONet
Loss	mean-squared error	FCNs and POD-DeepONet
mean relative H1-loss	FNO
D.2 Experiment setup
The Adam training was done on the GeForce 4x RTX 3080 Turbo GPU with 10 GB RAM, while sampling was performed on the 1x AMD EPYC 7402 @ 2.80GHz × 8 with 256GB RAM. We use the sacred package (https://github.com/IDSIA/sacred, MIT license, [37]) to conduct the hyperparameter search.
We perform a grid search over several hyperparameters (where applicable): the number of layers/Fourier blocks, the width of layers, the number of hidden channels, the regularization scale, the learning rate, and the number of frequencies/modes to keep. The full search grid is listed in Table 4.
Adam With the exception of FNO, we use the mean squared error as the loss function when training with Adam. For FNO, we train with the default option, that is, with the mean relative  $H^{1}$ -loss. This loss is based on the  $H^{1}$ -norm for a one-dimensional function  $f$  defined as
$$
\left\| f \right\| _ {H ^ {1}} = \left\| f \right\| _ {L ^ {2}} + \left\| f ^ {\prime} \right\| _ {L ^ {2}}.
$$
For all the experiments, we use the mean relative  $L^2$  error as the validation metric. We perform an early stopping on the validation set and restore the model with the best metric.
DeepONet We use the deepxde package (https://github.com/lululxvi/deepxde, LGPL-2.1 License, [45]) to define and train POD-DeepONet for our dataset. After defining the model, we apply an output transform by centering and scaling the output with  $1 / p$ , where  $p$  is the number of modes we
Figure 10: The architecture of the sampled FNO.
keep in the POD. To compute the POD, we first center the data and then use np.linalg.eigh to obtain the modes. We add two callbacks to the standard training: the early stopping and the model checkpoint. We use the default parameters for the optimization and only set the learning rate. The default training loop uses the number of iterations instead of the number of epochs, and we train for 90000 iterations with patience set to 4500. By default, POD-DeepONet uses all the data available as a batch. We ran several experiments with the batch size set to 64, but the results were significantly worse compared to the default setting. Hence, we stick to using the full data as one batch.
FNO We use the neuralop (https://github.com/neuraloperator/neuraloperator, MIT license, [42, 38]) package with slight modifications. We changed the trainer.py to add the early stopping and to store the results of experiments. Apart from these changes, we use the functionality provided by the library to define and run experiments. We use batch size 64 and train for 2000 epochs with patience 100. In addition to the hyperparameters considered for other architecture, we also did a search over the number of hidden channels (32 or 64). We use the same number of channels for lifting and projection operators.
FCN For both fully-connected networks in our experiments, we use PyTorch framework to define the models. For training a network in Fourier space, we prepend a forward Fourier transform before starting the training. During the inference, we applied the inverse Fourier transform to compute the validation metric. We considered using the full data as a batch, but several experiments indicated that this change worsens the final metric. Hence, we use batch size 64 for both architectures.
Sampling FNO Here, we want to give more details about sampled FNO. First, we start with input  $u_{0}$  and append the coordinate information to it as an additional channel. Then, we lift this input to a higher dimensional channel space by drawing  $1 \times 1$  convolution filters from a normal distribution. Then, we apply a Fourier block which consists of fully-connected networks in Fourier space for each channel. After applying the inverse Fourier transform, we add the input of the block to the restored signal as a skip connection. After applying several Fourier blocks, we learn another FCN in signal space to obtain the final prediction. We can see the schematic depiction of the model in Figure 10. We note that to train fully-connected networks in the Fourier space, we apply Fourier transform to labels  $u_{1}$  and use it during sampling as target functions. Similarly to the Adam-trained FNO, we search over the number of hidden channels we use for lifting.
We can see the results of the hyperparameter search in Figure 11. We see that sampled FNO and FCN in Fourier space perform well even with a smaller number of frequencies available, while DeepONet requires more modes to achieve comparable accuracy. We also see that adding more layers is not beneficial for sampled networks. A more important factor is the width of the layers: all of the sampled networks achieved the best results with the largest number of neurons.
For the Adam-trained models, we note that we could not significantly improve the results by increasing the widths of the layers. Possibly, this is due to a limited range of learning rates considered in the experiments.
Figure 11: Comparison of different deep neural operators trained with Adam and with SWIM algorithm. FNO with Adam has no dependency on a layer width. Thus, we show results for FNO trained with Adam as a straight line in all rows. For the hyperparameters not present in the plot, we chose the values that produce the lowest error on the validation set. We repeat each experiment three times and average the resulting metrics for the plotting. The error is computed on the validation set.
E Transfer learning
This section contains details about the results shown in the main paper and the supporting hyperparameter search concerning the transfer learning task. The source and target tasks of transfer learning, train-test split, and pre-trained models are the same as in the main paper. We describe the software, hardware, data pre-processing, and details of the sampling and the Adam training approach. Note that the results shown here are not averaged over five runs, as was done for the plots in the main paper.
Software and hardware We performed all the experiments concerning the transfer learning task in Python, using the TensorFlow framework. We used a GeForce 4x RTX 3080 Turbo GPU with 10 GB RAM for the Adam training and a 1x AMD EPYC 7402 socket with 24 cores @ 2.80GHz for sampling. We use the pre-trained models from Keras Applications which are licensed under the Apache 2.0 License which can be found at https://github.com/keras-team/keras/blob/v2.12.0/License.
Pre-processing of data The size of images in the ImageNet data set is larger than the ones in CIFAR-10, so we use a bicubic interpolation to upscale the CIFAR-10 images to dimensions  $224 \times 224 \times 3$ . We pre-process the input data for each model the same way it was pre-processed during the pretraining on ImageNet. Moreover, we apply some standard data augmentation techniques before training, such as vertical and horizontal shifts, rotation, and horizontal flips. After pre-processing, the images are first passed through the pre-trained classification layers and a global average pooling layer. The output of the global average pooling layer and classification labels serve as the input and output data for the classification head respectively.
Details of the Adam training and sampling approaches We find the weights and biases of the hidden layer of the classification head using the proposed sampling algorithm and the usual gradient-based training algorithm.
First, in the Adam-training approach, we find the parameters of the classification head by iterative training with the Adam optimizer. We use a learning rate of  $10^{-3}$ , batch size of 32, and train for 20 epochs with early-stopping patience of 10 epochs. We store the parameters that yield the lowest loss on test data. We use the tanh activation function for the hidden layer, the softmax activation function for the output layer, and the categorical cross-entropy loss function with no regularization.
Second, in the sampling approach, we use the proposed sampling algorithm to sample parameters of the hidden layer of the classification head. We use the tanh activation function for the hidden layer.
Once the parameters of the hidden layers are sampled, an optimization problem for the coefficients of a linear output layer is solved. For this, the mean squared error loss function without regularization is used.
Unless mentioned otherwise, the above-mentioned setup is used for all the experiments in this section.
E.1 Sampling Vs Adam training
Figure 12 compares the train and test accuracy for different widths (number of neurons in the hidden layer of the classification head) using three pre-trained neural network architectures for the Adam training and sampling approaches. As in the main paper, we observe that for all the models, the sampling approach results in a higher test accuracy than the Adam training approach for sufficiently higher widths.
The sampling algorithm tends to over-fit for very high widths. The loss on the train data for the iterative training approach decreases with width, particularly for the Xception network. This suggests that an extensive hyper-parameter search for the iterative training approach could yield higher classification accuracy on the train data. However, as shown in the main paper, the iterative training approach can be orders of magnitude slower than the sampling approach.
Figure 12: Left: ResNet50. Middle: VGG19. Right: Xception.
E.2 Sampling with tanh Vs ReLU activation functions
This sub-section compares the performance of the sampling algorithm used to sample the weights of the hidden layer of the classification head for tanh and ReLu activation functions.
Figure 13 shows that for ResNet50, the test accuracy with tanh is similar to that obtained with ReLU for a width smaller than 2048. As the width increases beyond 4096, test accuracy with ReLU is slightly better than with tanh. However, for VGG19 and Xception, test accuracy with tanh is higher than or equal to that with ReLU for all widths. Thus, we find the sampling algorithm with the tanh activation function yields better results for classification tasks than with ReLU.
On the training dataset, tanh and ReLU yield similar accuracies for all widths for ResNet50 and Xception. For VGG19, using the ReLU activation function yields much lower accuracies on the train and test data sets, especially as the width of the fully connected layer is increased. Thus, we observe that the tanh activation function is more suitable for classification tasks.
E.3 Fine-tuning
There are two typical approaches to transfer learning: feature extraction and fine-tuning. In feature extraction, there is no need to retrain the pre-trained model. The pre-trained models capture the
Figure 13: Left: ResNet50. Middle: VGG19. Right:Xception
essential representation/features from a similar task. One only needs to add a new classifier (one or more fully connected layers) and find the parameters of the new classifier.
In fine-tuning, after the feature extraction, some or all the parameters of the pre-trained model (a few of the last layers or the entire model) are re-trained with a much smaller learning rate using typical gradient-based optimization algorithms. Fine-tuning is not the focus of this work. Nevertheless, it is essential to verify that the weights of the last layer sampled by the proposed sampling algorithm can also be trained effectively in the fine-tuning phase. The results are shown in Figure 14.
In Figure 14, we observe that for certain widths (512, 1024, and 4096), sampling the last layer followed by fine-tuning the entire model yields slightly higher test accuracies than training the last layer followed by fine-tuning. For widths 2048, 6144, and 8192, for some models, sampling the last layer, followed by fine-tuning, is better; for others, training the last layer, followed by fine-tuning, is better.
Nevertheless, these experiments validate that the parameters of the last layer sampled with the proposed algorithm serve as a good starting point for fine-tuning. Moreover, the test accuracy after fine-tuning is comparable irrespective of whether the last layer was sampled or trained. However, as we show in the main paper, sampling the weights in the feature extraction phase takes much less time and gives better accuracy than training with the Adam optimizer for appropriate widths.
E.4 One vs two hidden layers in the classification head
The goal of this sub-section is to explore whether adding an additional hidden layer in the classification head leads to an improvement in classification accuracy. We keep the same width for the extra layer in this experiment.
Figure 15 compares the train and test accuracies obtained with one and two hidden layers in the classification head for different widths.
Figure 15 (left) shows that unless one chooses a very high width with the sampling approach  $\geq 6148$ , adding an extra hidden layer yields a lower test accuracy. On the train data, the sampling approach with 2 hidden layers results in lower accuracy for all widths in consideration.
Figure 15 (right) shows that with the Adam training approach, adding an extra hidden layer yields a lower test accuracy for all the widths in consideration. For lower widths, adding more layers results in over-fitting. We believe that the accuracy on the train data for higher widths could be improved with an extensive hyper-parameter search. However, adding an extra layer increases the training time too.
Figure 14: Comparison of test accuracies obtained by the sampling approach followed by fine-tuning and Adam training approach followed by fine-tuning for different widths
F Code repository and algorithm complexity
The code to reproduce the experiments from the paper can be found at
https://gitlab.com/felix.dietrich/swimnetworks-paper.
An up-to-date code base is maintained at
Python 3.8 is required to run the computational experiments and install the software. Installation works using "pip install -e ." in the main code repository. The python packages required can be installed using the file requirements.txt file, using pip. The code base is contained in the folder swimnetworks, the experiments and code for hyperparameter studies in the folder experiments.
We now include a short complexity analysis of the SWIM algorithm (Algorithm 1 in the main paper), which is implemented in the code used in the experiments. The main points of the analysis can be found in Table 5.
There are two separate parts that contribute to the runtime. First, we consider sampling the parameters of the hidden layers. Letting  $N = \max \{N_0, N_1, N_2, \ldots, N_L\}$ , i.e., the maximum of the number of neurons in any given layer and the input dimension, and  $M$  being the size of the training set. The size of the training set also limits the number of possible weights / biases that we can sample, namely  $M^2$ . The time complexity to sample the parameters of the hidden layers is  $\mathcal{O}(L \cdot M(\min \{\lceil N / M \rceil, M \} + N^2))$ . We can further refine the expression with  $\mathcal{O}(\sum_{l=1}^{L} M \cdot (\min \{\lceil N_l / M \rceil, M \} + N_l \cdot N_{l-1}))$ . Several factors contribute to this runtime. (1) The
1-Layer: Train data 1-Layer: Test data 2-Layers: Train data 2-Layers: Test data
Figure 15: Left: Sampling, Right: Adam training
Table 5: Runtime and memory requirements for training a sampled neural networks with the SWIM algorithm, where  $N = \max \{N_0,N_1,N_2,\dots ,N_L\}$ . Assumption (I) assume the output dimension is less than or equal to  $N$ . Assumption (II) assumes in addition that  $N < M^2$ , i.e., number of neurons and input dimension is less than the size of dataset squared. Assumption (III) assumes a fixed architecture.
Runtime	Memory
Assumption (I)	O(L·M(min{[N/M], M} + N2))	O(M·min{[N/M], M} + LN2)
Assumption (II)	O(L·M([N/M] + N2))	O(M·[N/M] + LN2)
Assumption (III)	O(M)	O(M)
number of layers increases the number of samples. (2) Computing the output after  $l - 1$  layers for the entire dataset to compute the space from which we construct the weights, hence the term  $N_{l} \cdot N_{l-1} \cdot M$ . (3) Computing  $M \cdot \tilde{M}_{l}$  probabilities, where  $\tilde{M}_{l} = \min \{\lceil N_{l} / M \rceil, M\}$ . When the size of the hidden layer is less than the number of training points — which is often the case — we compute  $2 \cdot M$  probabilities — depending on a scaling constant. On the other hand, when the size of the layer is greater than or equal to  $M^2$ , we compute in worst case all possible probabilities, that is,  $M^2$  probabilities. The last contributing factor is to sample  $N_{l}$  pair of points, that in the expression is dominated by the term  $N_{l} \cdot N_{l-1} \cdot M$ . We are often interested in a fixed architecture, or at least to bound the number of neurons in each hidden layer to be less than the square of the number of training points, i.e.,  $N < M^2$ . Adding the latter as an assumption, we end up with the runtime  $O(L \cdot M(\lceil N / M \rceil + N^2))$ .
For the second part, we optimize the weights/biases  $W_{L + 1}, b_{L + 1}$  to map from the last hidden layer to the output. Assume that we use the SVD decomposition to compute the pseudo-inverse and subsequently solve the least squares problem. The time complexity in general is then  $\mathcal{O}(M \cdot N_L \cdot \min\{M, N_L\} + N_L \cdot M \cdot N_{L + 1})$ . Again, if we make the reasonable assumption that the number of training points is larger than the number of hidden layers, and that the output dimension is smaller than the dimension of the last hidden layer, we find  $\mathcal{O}(MN_L^2)$ , which can be rewritten to  $\mathcal{O}(MN^2)$ . With these assumptions in mind, the run time for the full training procedure, from start to finish, is  $\mathcal{O}(L \cdot M(\lceil N / M \rceil + N^2))$ , and when the architecture is fixed, we have  $\mathcal{O}(M)$  runtime for the full training procedure.
In terms of memory, at every layer  $l$ , we need to store the probability matrix, the output of the training set when mapped through the previous  $l - 1$  layers, and the number of points we sample. This means the memory required is  $\mathcal{O}(M \cdot \lceil N_l / M \rceil + N_{L+1} \cdot N_L)$ . In the last layer, we only need the image of the data passed through all the hidden layers, as well as the weights/biases, which leads to  $\mathcal{O}(M + N_{L+1} \cdot N_L)$ . We end up with the required memory for the SWIM algorithm is  $\mathcal{O}(M \cdot \lceil N / M \rceil + LN^2)$ .