SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also investigated the computational scaling behavior of our implementation with respect to the number of sin components  $K$ . To this end, we ran a benchmark experiment, in which we compared forward and backward latencies of RELU, METASIN Native, and METASIN CUDA functions on the same input tensors using different  $K$  values. The results can be found in Figure 11.
Figure 12: METASIN activation shape evolution over time and final distribution of frequency parameters. In this figure, we report the behavior of three activated neurons (of the same layer) over the training process.
G Evolution of METASIN Activations during Training
Figure 12 shows illustrative examples of the evolution of METASIN activations during the course of training in a toy example. In this experiment the weight of the RELU component is set to 1, whereas the amplitudes of all sin components are set to 0. Thus a METASIN activation starts out as RELU, but then evolves into various shapes depending on the training of the shape parameters via backpropagation. It is worth noting that the shape of a METASIN activation tends to not change after the initial phase of training, which agrees with the observation that weights change very slowly in overparameterized networks. Moreover, the different rows in the figure show three different METASIN neurons of the same layer that interestingly converge to a complementary frequency response that can capture different frequency components of the input signal.
To shed some light on the shapes that METASIN activations take in a more practically relevant setting with and without KD-Bootstrapping, we additionally produced a visualization of the METASIN shapes from the resampling network described in Section 4.1. The METASIN shapes are obtained from the
first and last blocks of models trained from scratch and with KD Bootstrapping. This visualization in Figure 13 shows that, while there is significant local variation between individual METASIN activations, globally the rough RELU shape is still discernible even without any involvement from the RELU teacher.
Figure 13: Visualization of METASIN shapes with and without KD bootstrapping.
H Comparison with Ensemble Activations
We ran additional experiments using ensemble activations as described in a recent survey [3], namely Adaptive Blending Units (abu), Variable AFs (vaf), Adaptive AFs (aaf) in the same experimental setting that we used to generate the first row of Figure 1 in the main paper. We present the best results we obtained from multiple runs with different initializations in Table 10, also including RELU and METASIN as references:
abu	vaf	AAF	RELU	METASIN
PSNR[db]	29.79	28.98	29.06	29.81	35.04
Table 10: Comparison of various ensemble activations in an image overfitting setting.
We also ran two experiments in image resampling setting using abu and aaf versions of the model from Section 4.1, which we present in Table 11:
METASIN
Figure 14: Additional upsampling results comparing the state-of-the-art model with RELU activations [5] and the corresponding METASIN network.
Factor	Activation
abu	AAF	RELU	METASIN w/ KD-B
P×2	33.02	32.97	33.03	33.26
P×3	29.31	29.28	29.36	29.58
P×4	27.09	27.10	27.09	27.29
Table 11: Comparison of various ensemble activations for image resampling
I Effect of Setting  $K$  to a high number
We note that further improvements in terms of model accuracy can be made by pushing  $K$  beyond what we present in the paper. We did not go beyond 10-12 as throughout our experiments we observed that doing so often resulted in diminishing returns. As a more concrete example we provide results from a resampling experiment in Table 12 (Note that these results are from a preliminary run with slightly inferior hyperparameters compared to the corresponding experiments we report in the main paper, and the training was stopped prematurely after 900K iterations instead of the full 2M - hence the lower PSNRs compared to Table 2). These results suggest that choosing  $K > 10$  would indeed make sense in cases where the absolute best quality is desired and additional latencies are tolerable, but overall  $K \approx 10$  seems to be the happy medium.
Activation/Upsample Factor	P×2	P×3	P×4
METASIN-10	33.09	29.42	27.17
METASIN-20	33.14	29.45	27.20
J Classification Experiment with METASIN Model Trained from Scratch
To investigate the effect of KD-Bootstrapping in the classification setting, we trained various WideResNets with original RELU activations and with METASIN activations entirely from scratch and without using KD-Bootstrapping. Table 13 presents test accuracy on CIFAR-100.
Table 12: The effect of setting  $K$  to 10 and 20 on resampling quality
Model/Activation	WRN-16-2	WRN-28-2	WRN-40-2	WRN-40-4
RELU	72.85	74.82	75.95	78.99
METASIN w/o KD-B	71.82	73.88	75.44	78.44
In accordance with the above discussion these results underline the role of KD Bootstrapping for achieving the best results with METASIN networks. To give a concrete example: as we show above, when training from scratch WRN-16-2-METASIN at 71.82 accuracy is behind its RELU counterpart (WRN-16-2-ReLU) with accuracy 72.85. On the other hand, the last column of Table 6 shows that by distilling from a WRN-40-2 teacher, the accuracy of WRN-16-2-METASIN can be brought up to 74.10, whereas WRN-16-2-RELU achieves 73.65 accuracy using the same procedure.
K Resampler Architecture and Additional Results
Our network architecture is based on the design proposed in [5]. We utilize a ProSR feature extractor [45], consisting of 3 residual blocks with 4 dense blocks within each residual block and a growth rate of 40 and 160 channels. The subsequent prediction layer is implemented as an MLP with 4 hidden layers and 256 neurons. In our experiments, this prediction layer remains unchanged. We apply METASIN activations only to the convolutional feature extractor. We present additional results in Figure 14 for the resampler experiment presented in Section 4.1.
L DPCN Denoiser Architecture and Additional Results
The design of the U-Net architecture we use in Section 4.2 is detailed in Table 14. We also present
Table 13: Comparison of METASIN and RELU models in image classification with training without KD-Bootstrapping. In this case METASIN models only become advantageous when train using KD-Bootstrapping, underlining the importance of the proposed training methodology.
Scale	Encoder ResBlocks	Encoder Channels	Decoder ResBlocks	Decoder Channels
0	2	64	2	64
1	2	128	2	128
2	2	256	2	256
3	2	256	2	256
4	2	256	2	256
Table 14: The Configuration of the Standard DPCN U-Net. Scale 0 is the full-resolution scale. From one scale to the next one, the image resolution is halved in both dimensions.
additional results in Figure 15 to the results we presented in Section 4.2 in the main paper.
Reference (Full)
DPCN
METASIN-5 w/ KD-B
Reference
Figure 15: Additional results to Section 4.2
M KPAL Denoiser Results
In order to factor out the potential effects of kernel prediction to our denoising experiments, we utilized a direct prediction version of the state-of-the-art Monte-Carlo denoiser presented in [38]. In this section we present results of training a METASIN version of the original kernel prediction network. The improvement over the baseline RELU network remains significant in this experiment across all metrics. An interesting observation from this study was the significantly reduced effect of KD-Bootstrapping, which we eventually decided not to employ to produce the results in Table 15. We hypothesize that the lesser effect of KD-Bootstrapping in this case can be attributed to kernel prediction, which however requires further investigation.
Metric	KPAL	METASIN-10 (Glorot)
SMAPE	3.098	2.966 (-4.26%)
flip_loss	0.934	0.893 (-4.39%)
1-MS_SSIM	2.893	2.622 (-9.37%)
1-SSIM	6.227	5.934 (-4.71%)
Table 15: Comparison of the state-of-the-art kernel predicting denoiser [38] and its significantly improved METASIN version.
N Failure Cases
While our preliminary experiments on using METASIN activations with MLPs generally resulted in improvements over corresponding baselines, we also observed some failure cases. A particular
experiment where the tested METASIN network did not perform well was on signed distance field (SDF) overfitting. In this case, the SIREN reproduction of the ground-truth SDF is better at fine scale details than the corresponding METASIN reproduction. We hypothesize that this discrepancy in performance is due to the gradient loss that SIREN employs and the METASIN network lacks due to not being continuously differentiable. We defer further investigation of this limitation to future work.
(b) SIREN
Figure 16: SDF overfitting on Thai Statue [1]. Here the SIREN network produces a better representation of the underlying 3D shape than a comparable METASIN activated MLP.
(c)  $f_{j} = \mathcal{U}(1,35)$