SlowGuess/NeurIPS_2023_OCR huggingface dataset:
KD-Bootstrapping to MRELU which made a big improvement in all metrics, still, however, scoring far behind the corresponding METASIN network.
Overall, METASIN activations ended up consistently yielding notably better results than the activations we tested in all our experiments. In Appendix H we present further comparisons with ensemble activations.
5.2 The Effect of Hyperparameters
The METASIN activation as formulated in Equation 4 has a single hyperparameter  $K$  that determines the number of constituent sine waves. While in our experiments we often set  $K = 10$ , we also run ablation experiments to measure the effect of different  $K$  values. In these experiments we used the exact same settings as in Section 4.2, except varying the  $K$  parameter. In Table 5 we show the effect of changing  $K$  to 1, 5, and 12, compared to the baseline where  $K = 10$ . Not surprisingly, lowering  $K$  results in consistently worse results according to all four metrics we tested. These findings also show that the results that we presented in Section 4.2 can be further improved simply by setting  $K = 12$  (See Appendix I for further discussion). Aside from  $K$ , another decision to be made when using METASIN activations is whether to train with KD-Bootstrapping or not. While in toy examples we haven't noticed any difference between the two alternatives, the last two columns in Table 4 show a comparison with and without KD-Bootstrapping, which clearly demonstrates the benefit of employing KD-Bootstrapping in a real-world scenario.
Metric	METASIN-10 KD-B	METASIN-1 KD-B	METASIN-5 KD-B	METASIN-12 KD-B
FLIP	0.972	0.997 (+2.57%)	0.979 (+0.72%)	0.967 (-0.51%)
1-MS_SSIM	2.784	2.825 (+1.47%)	2.805 (+0.75%)	2.767 (-0.61%)
Table 5: Comparison of DPCN networks (discussed in Section 4.2) with various  $K$  parameters trained with KD-Bootstrapping and Glorot frequency initialization.
5.3 Image Classification using Convolutional METASIN Networks
We ran several preliminary experiments for exploring the use of METASIN networks outside our target domain of image prediction tasks. Here we report the results of various experiments in a classification setting, where we trained various student Wide-ResNets using supervision from comparably larger Wide-ResNets that we used as teacher networks. As the teacher network in this case is a larger network with higher accuracy, replicating its predictions becomes a reasonable objective. Therefore, we employ feature knowledge distillation throughout the entire training procedure. The Wide-ResNet architecture [47] follows the same architectural design as ResNet [15], except that the number of channels in each residual block is expanded.
The METASIN activated students are identical to their RELU activated counterparts, except the activations are replaced by METASIN with  $K = 8$ . We ran our experiments on the CIFAR100 [23] dataset, which consists of 50K training and 10K validation images with resolution  $32 \times 32$ . The student networks were trained for 300 epochs using standard cross-entropy loss following the knowledge distillation methodology discussed in [36]. While not setting a new state-of-the-art, Table 6 shows that the METASIN student networks consistently outperform their RELU counterparts, and in some cases also the corresponding SWISH student networks, motivating further investigation of using METASIN activations in classification tasks. However, we note that the accuracy of METASIN models are less competitive compared to alternatives when models are trained from scratch without knowledge distillation (Appendix J).
5.4 METASIN with Multi-Layer Perceptrons
While in this work we focused on applications of METASIN with convolutional neural networks, we also ran several exploratory experiments where we used METASIN activations with MLPs to perform implicit neural representation tasks as in [39]. We refer the reader to Appendix A.1 for a discussion on video overfitting, where the METASIN network achieves 1-4 dB PSNR improvement over the SIREN
Teacher	WRN-40-2	WRN-40-4	WRN-28-2	WRN-40-2
Student	WRN-40-1	WRN-40-1	WRN-16-2	WRN-16-2
Teacher Acc.	76.08	78.58	74.82	76.08
Activ.	RELU	METASIN	SWISH	RELU	METASIN	SWISH	RELU	METASIN	SWISH	RELU	METASIN	SWISH
Val. Acc.	73.39	73.74	73.28	70.53	72.55	72.54	71.76	72.33	72.98	73.65	74.10	72.60
Table 6: Comparing RELU student networks with METASIN activated networks with otherwise identical architecture, both trained through knowledge distillation from the same RELU activated teacher.
baseline with noticeably less visual artifacts. In Appendix A.3 we discuss a voxel grid overfitting experiment where the METASIN network exhibits significantly improved detail reconstruction over the RELU baseline. We also present a preliminary novel view synthesis experiment where we compare a baseline Neural Radiance Field (NeRF) method [29] with a corresponding METASIN model with  $K = 10$ .
We present an example result from this experiment in Figure 4, and refer the reader to Appendix A.2 for more details. Despite not utilizing positional encoding, the METASIN network surpasses vanilla NeRF in performance. It has been observed that Fourier features[42], which bear similarities to positional encoding, aid in learning high-frequency functions. This experiment implies that such feature engineering may not be crucial, as our METASIN network achieves comparable or even superior performance. While this result is intriguing, we acknowledge recent techniques that leverage latent codes in the form of memory, combined with specialized data structures [31, 41], which can significantly enhance vanilla NeRF in terms of performance and training time. While not conclusive, we hope the initial results we share in this section motivate further investigation of METASIN activations in the context of other architectures and application areas.
Figure 4: Zoomed qualitative results on scene flower [28]. Comparison between ground-truth, NeRF [29] and METASIN-10 model, with corresponding PSNR scores in parenthesis.
NeRF-PyTorch (20.78 dB)
METASIN-10 (21.46 dB)
We presented a novel activation function with trainable parameters that improves upon state-of-the-art convolutional image prediction models. Our work has several limitations. While METASIN networks can reliably be trained without it, training with KD-Bootstrapping helps the network to eventually converge to better minima. While it is not uncommon when training a METASIN network to have access to a comparable trained RELU network, nevertheless this requirement might lead to some extra effort when a pre-trained base model is not available. Additionally, in our current formulation (Equation 4) the number of sin components  $K$  is a hyperparameter that we set manually at design time. We performed preliminary experiments with determining  $K$  automatically in an adaptive fashion, which would be an interesting direction to study more in the future.
The METASIN function is only  $C^0$  in contrast to the infinitely differentiable SIREN. The  $C^\infty$  property allows SIREN networks to be trained using loss functions on the gradient or Hessian of the network. This distinct characteristic of the SIREN network is used later in [24] to automatically integrate the function that appears in neural volume rendering. The lack of a gradient loss in METASIN networks seems to limit the performance of METASIN activations for applications such as implicit 3D shape representations (Appendix N). It would be interesting to tackle this inherent limitation of METASIN networks in future work.
Fundamentally, neural network blocks comprise a linear transformation and a non-linear activation. Our work introduces a new activation function that enables significant prediction accuracy improvements over previous alternatives. Being at such a fundamental level, our proposed contribution thus can be applied to a wide array of neural network types that perform a variety of tasks and facilitate improvements upon existing state-of-the-art results. The resulting improved models may be leveraged for both positive and negative ends.
We thank Xianyao Zhang, Marios Pappas, Michael Bernasconi, and Aziz Djelouah for their help in Monte-Carlo denoising and resampling experiments, Katarina Tothova for proof-reading our manuscript. We also thank Wig42 for the "RoomChair" and "LivingRoom" images, Xianyao Zhang for the "furball" image in Figure 3, nacimus for "Can" image, SlykDrako for the "BedRoom" image in Figure 15. The dataset for the Nerf experiments were provided by Matthew Tancik and Ben Mildenhall.
[1] The stanford 3d scanning repository. URL http://graphics.stanford.edu/data/3Dscanrep/.
[2] Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D Fairchild. Flip: A difference evaluator for alternating images. Proc. ACM Comput. Graph. Interact. Tech., 3(2):15-1, 2020.
[3] Andrea Apicella, Francesco Donnarumma, Francesco Isgro, and Roberto Prevete. A survey on modern trainable activation functions. Neural Networks, 138:14-32, 2021.
[4] Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. Kernel-predicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph., 36(4):97-1, 2017.
[5] Michael Bernasconi, Abdelaziz Djelouah, Farnood Salehi, Markus Gross, and Christopher Schroers. Kernel aware resampler. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[6] Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makovi-ichuk, S. Tulyakov, and Jian Ren. Real-time neural light field on mobile devices. ArXiv, abs/2212.08057, 2022.
[7] Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning, 2020.
[8] Ka-Hou Chan, Sio-Kei Im, Wei Ke, and Ngan-Lin Lei. Sinp[n]: A fast convergence activation function for convolutional neural networks. In 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion), pages 365–369, 2018. doi: 10.1109/UCC-Companion.2018.00082.
[9] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628-8638, 2021.
[10] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
[11] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus), 2016.
[48] Xianyao Zhang, Marco Manzi, Thijs Vogels, Henrik Dahlberg, Markus Gross, and Marios Papas. Deep compositional denoising for high-quality monte carlo rendering. In Computer Graphics Forum, volume 40, pages 1-13. Wiley Online Library, 2021.
[49] Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 33:1583-1594, 2020.
A Use of METASIN Beyond Image Prediction
In addition to our main results presented in Section 4 of the paper, we also performed various exploratory experiments to investigate further application cases of the METASIN activations. The experiments cover image classification where we show favorable results of using convolutional METASIN networks over baseline RELU networks, as well as various overfitting experiments to explore the use of METASIN activations with MLPs. We present our preliminary findings here with the hope of motivating further investigation in these directions.
In all MLP experiments, we use METASIN with  $K = 10$  sine components, and distribute the frequencies evenly across the range [1, 35]. The initialization of the remaining parameters follows the description provided in Section 3.
Figure 5: Visualization of selected reconstructed frames of the video. To fully appreciate the details and visual cues presented in the figure, we recommend visualizing the figures in color and zooming in for a more comprehensive analysis.
Video Dataset	Activation	Model	Channels	Depth	MSE ↓	PSNR (dB) ↑
Cat	SIREN	MLP	1024	3	2.60 · 10-3	32.26
METASIN	MLP	1024	3	1.96 · 10-3	33.32
Bikes	SIREN	MLP	1024	3	1.69 · 10-3	34.50
METASIN	MLP	1024	3	5.58 · 10-4	39.32
Table 7: Results on implicit video representation.
A.1 Video Overfitting using METASIN MLP
In this section, we present the overfitting experiments on videos using an MLP network with 3 hidden layers and 1024 neurons to overfit a video. The input is a 3 dimensional vector consisting of a frame ID and pixel coordinate, and the output is the RGB value of the corresponding pixel. The METASIN network improves the PNSR by  $1 - 4\mathrm{dB}$  compared to the SIREN baseline as shown in Table 7.
Overall the Cat dataset has more fine details, making it more challenging to accurately represent these intricate details. Consequently, the PSNR for the Cat dataset is comparatively lower in comparison to the Bikes dataset. The results are depicted in Figure 5. While RELU networks struggle to represent the signal, both SIREN network and METASIN can reconstruct the frames. Notably, the METASIN network reconstructs more details, see Figure 5 (Bicycle) and has less noise, as we see in Figure 5 (Wall and Car). It is worth mentioning that various techniques, such as positional encoding or Fourier
NeRF
METASIN-10
Figure 6: Zoomed qualitative results on static forward scenes [28]. Comparison between ground-truth, NeRF and METASIN-10 model.
features [43, 42], as well as utilizing latent codes [31], have shown significant improvements in RELU networks. However, exploring the impact of these techniques when applied in conjunction with METASIN activation is a direction we defer to future work.
Method	Leaves	Orchids	Flower	T-Rex	Horns
NeRF [29]	20.92	20.36	27.40	26.80	27.45
METASIN-10	21.46	20.52	28.20	26.81	27.98
Table 8: Per-scene PSNR† comparison. In this table, we report the results obtained by running NeRF PyTorch implementation, based on [46] and previously used in [21, 6], and the results of our proposed METASIN. Our method achieves higher PSNR in all tested scenes.
A.2 Novel View Synthesis using METASIN MLP
Neural Radiance Fields [29] (NeRF) enable synthesizing novel views with only a sparse set of input views. NeRF methods exploit positional encoding to map the input into high-frequency space, which is the key to high-quality scene rendering, which we drop in our experiments and replace the standard RELU with a 10 components METASIN-10 activation. Frequencies and phases are initialized uniformly over [1, 35] and  $[0, 2\pi]$ . The experiment setting is the same as [29], except that we extend the training process to  $250\mathrm{K}$  iterations instead of  $200\mathrm{K}$ , in order to make sure that METASIN models converge well. Our implementation is based on [46], The results are reported in Table 8. We also provide a comparison of qualitative results in Figure 6.
Figure 7: False-color visualization of the reconstructed density grid where METASIN is configured with  $K = 10$
A.3 Density Overfitting using METASIN MLP
In this section we present an overfitting experiment on a 3D voxel grid. The smoke scene we used in this experiment contains  $175 \times 232 \times 175$  density values within range [0, 1]. Following the MLP architecture used in [39], we map the 3D coordinates to the corresponding density value using RELU and METASIN activated networks. Comparisons between the METASIN activated network and the baseline RELU model are presented in Figure 7.
B Details of Overfitting Experiment with Convolutional Network
Table 9 shows further architectural details of the overfitting experiment we discuss in Section 3 of the main paper. We perform experiments with two base convolutional architectures, a shallow and a deep network. M-SIREN refers to the amplitude modulated SIREN [27], which has significantly more parameters due to the introduction of an additional amplitude modulation network.
Figure 8: Peak Signal-to-Noise Ratio (PSNR) comparison of different activations employed on CNNs for image up-sampling (with  $16\mathrm{x}$  of upsampling factor).
Figure 8 shows a visual comparison of reconstructed images by the different versions of both base networks. Both the shallow and deep versions of the METASIN network can be trained reliably (unlike their sin activated counterparts that diverge in the case of the deep architecture). The METASIN networks come with relatively little parameter overhead (the relative overhead becomes even smaller with larger networks that have more channels, as discussed in Appendix E), and produce visually sharper results that also are closer to the reference in terms of PSNR.
CNN Size	Activation	#Param.	Depth	PSNR (dB) ↑
Shallow	RELU	75 K (1x)	4	29.81
SIREN [39]	76 K (1x)	4	32.55
M-SIREN [27]	281 K (3.7x)	4	30.88
METASIN (ours)	82 K (1.1x)	4	35.04
Deep	RELU	408 K (1x)	13	30.36
SIREN [39]	408 K (1x)	13	12.65
M-SIREN [27]	651 K (1.6x)	13	35.70
METASIN (ours)	432 K (1.1x)	13	40.01
Table 9: Comparison of different activations employed on CNNs trained for image up-sampling (16x).
Figure 9: Representing  $y(t) = \sin (10t + 5\sin (10t)) \times \sin (10\sin (10t))$  with 3 layer FC networks
C Sensitivity to Frequency Initialization
Although the sin activation function has promising properties as outlined in the main paper, it is known that they can be extremely difficult to train [39, 49].
We illustrate this in a toy experiment where we try to reconstruct the signal  $y(t) = \sin (10t + 5\sin (10t)) \times \sin (10\sin (10t))$  when the input is  $x(t) = t$ . Most of the energy of  $y(t)$  is distributed between frequencies between 10 and 170. We sample  $t$  from the normal distribution and train fully connected networks with 1 input neuron, 3 layers having 20 neurons each, and 1 output neuron. In one of the networks, we use RELU activation in between fully connected layers except for the last
one. In the remaining two networks, we use  $\sin (fx)$ . In one we initialize  $f = 1$ , in the other one we initialize  $f = 5$ , and  $f = 10$ . We use  $L1$  loss and Adam as the optimizer. The results are shown in Figure 9. As it is seen, RELU-network can reconstruct part of the signal. The sin-network with the initialization  $f = 1$  does not learn at all, while the network initialized with  $f = 5$  is able to match  $y(t)$ , and with  $f = 10$  the model fits the original signal reasonably well. This suggests the importance of initialization and the difficulty of training of the sin activation function.
On the other hand, a METASIN network with the frequency parameters from Equation 4 initialized as  $f_{j}^{[l]} = j$  and  $K = 10$  effectively covers a wide range of frequencies including the one that is important to represent the signal. Consequently, using a METASIN we are able to lower the loss to 0.013 and eliminate the trial and error process for initializing frequencies.
In Figure 10 we additionally plot the results of an experiment where we sweep through a wide range of  $w_{0}$  values for a fully-connected SIREN network that was trained to overfit to a single image. The plot illustrates that only a narrow range among possible initial  $w_{0}$  values results in a plausible training error, confirming the importance of initialization in SIREN networks.
Figure 10: Sensitivity of SIREN w.r.t. base frequency  $w_{0}$ . In this figure, we reported the PSNR and MSE Loss values across a wide range of SIREN's base frequency  $w_{0}$  values. For each  $w_{0}$ , we trained a SIREN network and showed that only a restricted set of frequencies (around  $10^{2}$ ) converges to sufficiently good local minima. Specifically, in this experiment, we train the networks to up-sample the image reported in Figure 8, with an up-scale of  $\times 16$  for 100,000 iterations using the Adam optimizer with a learning rate of  $10^{-5}$ .
D Connection to Fourier Series
Well-behaved functions, such as continuous functions can be expressed as a Fourier series with finitely many terms. Let us define a function  $\phi$  as a Fourier series:
$$
\phi (\mathbf {x}) = a _ {0} + \sum_ {j = 1} ^ {K} \left(a _ {j} \cos \left(\frac {2 \pi j}{P} \mathbf {x}\right) + b _ {j} \sin \left(\frac {2 \pi j}{P} \mathbf {x}\right)\right), \tag {5}
$$
where  $K \in \mathbb{N}^{+}$  is finite, and  $P$  denotes length of one period. Using the harmonic addition rule, i.e.
$$
a \cos \alpha + b \sin \alpha = c \cos (\alpha + r), \text {w h e r e}
$$
$$
c = \operatorname {s g n} (a) \sqrt {a ^ {2} + b ^ {2}} \tag {6}
$$
$$
r = \arctan \left(- \frac {b}{a}\right),
$$
we can rewrite Equation 5 as:
$$
\phi (\mathbf {x}) = a _ {0} + \sum_ {j = 1} ^ {K} c _ {n} \cos \left(\frac {2 \pi j}{P} \mathbf {x} + r\right). \tag {7}
$$
Substituting in the identity  $\cos \alpha = \sin \left(\alpha +\frac{\pi}{2}\right)$  we get:
$$
\phi (\mathbf {x}) = a _ {0} + \sum_ {j = 1} ^ {K} c _ {n} \sin \left(\frac {2 \pi j}{P} \mathbf {x} + \frac {\pi + 2 r}{2}\right). \tag {8}
$$
Dropping the constant term we obtain the following:
$$
\phi (\mathbf {x}) = \sum_ {j = 1} ^ {K} c _ {j} \sin \left(f _ {j} \mathbf {x} + p _ {j}\right), \text {w h e r e}
$$
$$
f _ {j} = \frac {2 \pi j}{P}, \text {a n d} \tag {9}
$$
$$
p _ {j} = \arctan \left(- \frac {b _ {j}}{a _ {j}}\right) + \frac {\pi}{2}.
$$
From Equation 9 we can see that any Fourier Series can be expressed as a METASIN activation
$$
\phi (\mathbf {x}) = c _ {0} \operatorname {R E L U} (\mathbf {x}) + \sum_ {j = 1} ^ {K} c _ {j} \sin \left(f _ {j} \mathbf {x} + p _ {j}\right), \tag {10}
$$
by setting  $c_0 = 0$  and the rest of its parameters  $\{c_j, f_j, p_j; j \in [1, K]\}$  accordingly.
E METASIN Overhead
METASIN parameter overhead formula for a convolutional layer compared to RELU baseline is as follows:
$$
\text {o v e r h e a d} = \frac {3 \times K + 1}{\text {k e r n e l - s i z e} ^ {2} \times \text {i n p u t - c h a n n e l s}}. \tag {11}
$$
For example if we have a layer with kernel-size  $= 3$  with 128 input-channels, then overhead for  $K = 10$  is 0.026. Which means we have approximately  $2.6\%$  more parameters. If input-channels is 32 the overhead becomes  $10.08\%$ . On the other hand for kernel-size  $= 5$  and 512 input-channels the overhead is  $0.24\%$ .
F Details on C++/CUDA Implementation
To account for the inefficiency of a native Python API implementation of METAsin we implemented custom-optimized fused CUDA kernels for both forward and backward functions of the MetaSin activation in C++. Our implementation can be integrated into the PyTorch and TensorFlow Python APIs. Throughout the development process we also tested native automatic compilation functionalities provided by both frameworks (specifically: JIT, torch.compile for PyTorch, and JIT and XLA in TensorFlow). While notable improvements over the baseline Python API implementation can be made through the use of these out-of-the-box facilities, we obtained the best performances in terms of speed and memory consumption using our custom-designed CUDA functions.
Some of the techniques we utilized in our code are as follows: in order to optimize the memory footprint and the inference speed of the METASIN, we remove the intermediate quantities that the autograd engine computes and instead compute the output and gradient tensors directly from the input and the METASIN parameters. Moreover, we further optimized the computation speed with improved caching and reduction strategy on warp and block levels. Meanwhile, we have exploited a 2-level reduction strategy based on pairwise summation in our backward kernel. In this way, we could avoid numerical errors in gradient tensors and achieve comparable accuracy to autograd in float32 precision.
The reduction in computational overhead using the aforementioned optimizations enabled running the compute-intensive experiments we present throughout the paper in a feasible manner, and we believe demonstrates the viability of METASIN activations for most practical tasks. That being said, other optimizations that we were not able to explore due to time constraints could help reduce the overhead even further, which is an interesting direction for future research.
Figure 11: Effect of METASIN components to latency.