Empowering Convolutional Neural Nets with MetaSin Activation
Authors: Farnood Salehi
Tunc Ozan Aydin
André Gaillard
Guglielmo Camporese
Yuxuan Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: sin activation
image prediction
image resampling
monte-carlo denoising
knowledge distillation
TL;DR: A novel sin-based activation function that improves upon state-of-the-art convolutional image prediction models.

Abstract: ReLU networks have remained the default choice for models in the area of image prediction despite their well-established spectral bias towards learning low frequencies faster, and consequently their difficulty of reproducing high frequency visual details. As an alternative, sin networks showed promising results in learning implicit representations of visual data. However training these networks in practically relevant settings proved to be difficult, requiring careful initialization, dealing with issues due to inconsistent gradients, and a degeneracy in local minima. In this work, we instead propose replacing a baseline network’s existing activations with a novel ensemble function with trainable parameters. The proposed MetaSin activation can be trained reliably without requiring intricate initialization schemes, and results in consistently lower test loss compared to alternatives. We demonstrate our method in the areas of Monte-Carlo denoising and image resampling where we set new state-of-the-art through a knowledge distillation based training procedure. We present ablations on hyper-parameter settings, comparisons with alternative activation function formulations, and discuss the use of our method in other domains, such as image classification.

Empowering Convolutional Neural Networks with MetaSin Activation
Farnood Salehi'
Tunç Ozan Aydın 1.3
André Gaillard?
Guglielmo Camporese
Yuxuan Wang
Disney Research Studios
University of Padova
*ETH Zürich
Abstract
RELU networks have remained the default choice for models in the area of image prediction despite their well-established spectral bias towards learning low fre- quencies faster, and consequently their difficulty of reproducing high frequency visual details. As an alternative, sin networks showed promising results learning implicit representations of visual data. However training these networks in practi- cally relevant settings proved to be difficult, requiring careful initialization, dealing with issues due inconsistent gradients, and degeneracy in local minima. In this work, we instead propose replacing a baseline network's existing activations with novel ensemble function with trainable parameters. The proposed METASIN activation can be trained reliably without requiring intricate initialization schemes, and results consistently lower test loss compared alternatives. We demonstrate our method in the areas of Monte-Carlo denoising and image resampling where we set new state-of-the-arr through knowledge distillation based training procedure. We present ablations on hyper-paramette settings, comparisons with alternative ac- tivation function formulations, and discuss the use of our method in other domains, such as image classification.
Introduction
Deep convolutional neural networks are highly proficient in a wide array of tasks including image prediction applications. While there no scareity of inventive methods when it comes to designing convolutional blocks (e.g. (25)), another crucial design element, namely the activation function, has seen little change over the years. The RELU activation, despite (and arguably because of) its simplicity has remained the default choice for majority of the models in the area of image prediction.
is often desirable for models that produce visual results to produce sharp imagery with abundant high frequency details. The widespread use of RELU networks for image prediction applications somewhat surprising given their spectral bias towards learning low frequencies first [33], and consequently their difficulty in reproducing high frequency visual details. While sin activations have been shown be superior in simple overfitting experiments on image reconstruction [39]. empirical evidence showed that training sin networks in practically relevant settings is challenging even when employing meticulous initialization schemes. While some exploration on using sin activations with convolutional networks exists (8), their use has been mostly limited to fully connected architectures.
In this work make the following contributions:
"This work was done while the author was an intern Disney Research Studios
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We shed light on challenges associated with training networks with sin-based activations through an extensive set of experiments, and address those by formulating the METASIN activation coupled with stable training procedure.
We show that METASIN networks consistently enable improvements over alternatives when tested on comparable architectures in variety of image prediction applications, including Monte-Carlo denoising and image resampling where we set new state-of-the-an
We present wide range of ablations on hyperparamter settings, comparisons with various alternative activations, and discuss the use of METASIN networks in other domains such as classification and neural signal representations.
2 Related Work
Activation Functions In addition to standard non-polynomial functions such as Sigmoid, tanh, sin, RELU, and exp, numerous alternative activation functions have been explored in the literature. Previous work explored RELU variants such as Leaky RELU [26), Parametric rectified linear unit (PReLU) [14), and Gaussian error linear unit (GELU) [17]. Other authors presented activations with exponential components, such as the Exponential linear unit (ELU) [[]] Scaled exponential linear unit (SELU) 22 as well as Sigmoid variants such as Softplus 11]. and SWISH [34). A comparative study of these activations re-affirmed RELU as a strong baseline, while suggesting S SWISH might be better alternative for image classification (4], Another study [25] reported no accuracy change when switching from RELU to GELU in image classification. The aforementioned body of work has mostly focused on classification experiments, whereas their use in image prediction models was left mostly unexplored.
Previous investigations explored in the form of sin activations for implicit neural representations of continuous signals [39].
3 pointed out periodic activations as a promising direction, which was later
More recent work proposed modulating the amplitudes of sin functions using a secondary fully
connected network (27). Another sin variant is Snake [49]. which takes the form x + sin?(x) and focuses on improving extrapolation of periodic functions. Finally, several papers explored combining multiple primitive functions, such as Mish (z tanh (softplus(z)) x>) [30], SinL.U (sin and Sigmoid) (32)
and activation ensembles [3]. Preliminary experiments [39 suggest that activations alternative to RELU, with the exception of sin, tend to perform sub-pu par on image reconstruction. The popularity of RELU networks in image prediction tasks contradicts the theoretical analysis presented in previous work
33 that established the spectral bias of RELU networks towards learning low frequencies faster. Later work re-affirmed this finding through a neural tangent kernel
20 analysis.
Image Resampling Tasks such as correcting lens distortion, retargeting, and upsampling are some image resampling examples that are considered as key operations in real-world computer vision and image processing pipelines. Recent neural methods [5] 40 9 have been shown to outperform traditional techniques, such as upsampling with a fixed kernel. These neural networks often comprise convolutional blocks [16 [9| with RELU activations. Typically utilizing deep networks, current resampling methods can learn complex representations of input images and produce high-quality resampled outputs. In this work, we employ METASIN to improve upon recent state-of saute-of-the-ar of-thh-art resampler 5
Monte-Carlo Denoising Generating high quality rendered images may require prohibitive amount of compute resources. This can be alleviated by stopping the rendering process prematurely and employing a specialized denoiser to remove the residual noise. Neural denoisers have demonstrated superior performance compared to classical denoising techniques often leverage U-Net with RELU activation [37]. which allows them to process noisy images across multiple scales. We likewise demonstrate the use of METASIN in such U-Nets and achieve significant improvements in denoising quality.
4
These denoisers
3 METASIN Activation
Let us define a generic neural network g(x) of depth L that predicts target image y Rd given input x E d as:
y Rd g(x) = (61 T(, o |!t--1) T!---1] o .... wl') T(1) (x),
(1)
2

---

where 7 denotes a linear transformation that results from either convolutional or fully connected layer. We would like to find set of activations .ell....l such that they minimize the average reconstruction error Σ C (g (x(+)) - y()) over a dataset fx(") y()) E [1,n]]-
In hypothetical limit cases where the network 9 has infinite width or depth, few constraints on activation functions are necessary for the resulting architecture have the universal approximation capability [88]. However, in practice the selection of the activation functions can play significant role in the training of neural networks, specifically the resulting model's prediction accuracy and, in case of image prediction models, the visual characteristics of the predictions. While it is unclear how principled method can be developed for rank ordering all possible selections of activations, empirical evidence suggests that multi-layer perceptron (MLP) networks with sin activations exhibit desirable characteristics both in expressiveness and their ability to reproduce rich high frequency details [3]] especially in comparison to RELU networks that suffer from spectral bias towards reproducing low frequencies (33.
Input
Target
RELU
SIREN
METASIN
Shallow CNN
29.81 dB
32.55 dB
35.04 dB
Deep CNN
Diverged
30.36 dB
12.65 dB
40.01 dB
Figure I: Peak Signal-to-Noise Ratio (PSNR) comparison of different activations employed on CNNs to overfit image from its downsampled version (with downsampling factor 16x). Notably, while the shallow sin network performs better than its RELU counterpart, the deep sin network fails to converge, demonstrating the fragility of training deep convolutional networks with sin activations. See Appendix B for details.
However, practical experience also revealed the difficulty of training deep convolutional sin networks in real-world scenarios. Figure[]shoo an illustrative example that on one hand demonstrates the clear advantage of sin activations over RELU when training shallow networks, but the other hand illustrates the fragility of training deeper sin networks that are closer representatives of real-world models. To examine further we express intermediate segment from sin activated network
a M sin (T') (al-1)).
(2)
where a! the output of layer after applying the corresponding linear transformation and acti- vation function. We can also express the usual parameterization of sine waves as f(z; e, J.P) = esin (fz + p). where c, f. and p are the amplitude, frequency, and phase of the sine wave that we collectively refer the shape parameters.
In overparameterized networks the initial parameter values tend to change very slowly 10 (also known as lazy training). Therefore ensuring the selection of plausible initial values for the shape parameters in sin networks is crucial for training. Some exploration has been done in this direction: For instance, SIREN 39 introduces scaling constant Wp that is applied to the weight matrix W of the first layer the underlying fully connected network, i.e. sin (wo Wxx + b), whereas recent work explored introducing an explicit amplitude parameter whose value is predicted through a secondary neural network (27]. Yet initialization remains a challenge as in practice only a very narrow range of values leads to plausible average loss (Refer to Figure 10 in Appendix C.

---

In this work. we instead introduce explicit shape parameters for directly controlling amplitude, frequency, and phase. These parameters are completely disentangled from other network parameters, and can be initialized intuitively by reasoning about the shape of resulting sine wave. Moreover, in order to provide better initial coverage of plausible ranges of the shape parameters, we construct a composite periodic function by linearly combining K sine waves with individual shape parameters:
K 3 sin ( + + where z = =T() (a!)) (a()
The shape parameters are shared among individual channels of layers in convolutional neural networks, and are optimized through backpropagation along with other network parameter In our experiments
we observed that initializing the frequency parameters f/!) of the K constituent sine waves of the composite function to cover a large range of frequencies consistently yields improved results, without necessitating lengthy trial and error procedures that are often not feasible when training large models in practice (Refer to Appendix forr illustrative example).
Beyond the initialization of shape parameters, the remaining known causes of the difficulty when training sin networks are inconsistent gradients due to the complex shapes that activations can take, and the large degeneracy in local minima caused by symmetries [49]. We empirically found that including additional RELU component to the composite sine wave function discussed before stabilizes training. With this modification our proposed activation, which we call METASIN, can be expressed as follows:
p(z =
11 RELU (2)))
+ Σ
o sin (1,92*) - -
(4)
Equation can efficiently be implemented as a module in common deep learning frameworks, which then can be used to replace activation functions of an existing state-of-the-an network architecture. of backpropagation without requiring an additional network for predicting these particular parameters, or any other changes to the underlying architecture. This non-intrusive property enables a simple three-step procedure, where (i) we identify a offte-of-the-ar image prediction model, (ii) replace its existing activation functions with METASIN, and (iii) re- re-train the resulting METASIN network using a knowledge distillation scheme we discuss in the following section.
P. The trainable parameters 0 {c- 1I ell [0 :j E [1, / E L}} are optimized through
3.1 Training
In order to train convolutional METASIN networks, we initialize the shape parameters as Co = 1, of 0, f|ر and p) [ (0(0, T), for j E [1, K]. and I [1, L] by default. This initialization on one hand forces METASIN to initially assume the shape of RELU, and gradually introduce sin components during the course of the training, while on the other hand covering broad range of initial frequency and phase values (See Appendix C Using the default initialization we were able to reliably avoid typical issues encountered while training sin networks.
is often easier to train a shallower network compared to a deeper one as supported by previous studies [16. This observation also applies to sin activations, demonstrated in Figure where shallow SIREN network can be effectively trained while a deep SIREN network encounters difficulties. To address this, employ feature knowledge distillation (KD) from teacher network [35 to provide auxiliary signals during the training of the student METASIN network. We first train a RELU an already existing baseline RELU model) and use its intermediate feature maps as supervisory signals. This approach allows us to train the shallower blocks of the student METASIN network individually, instead of propagating gradients throughout the entire network. We refer to this phase of training as XID-Bootstrapping which comprises approximately 5 10% of the total training iterations for the METASIN network. Following the initial KD-Bootstrapping phase, the student METASIN network is trained using the same configuration as the baseline network. This approach enhances the learning process and helps improve the performance of the student network.
model as the teacher (or, as it often happens in practice,
'Wtt certain selection of parameters the resulting composite function can be shown to be equivalent to a Fourier Series, and hence the hyperparameter K can be thought as controlling the expressiveness (Appendix D)

---

3.2 Efficient Implementation
While equation is straightforward to implement in both PyTorch and Tensorflow using the corre- sponding Python APIs, such native implementations end up being inefficient compared to executing RELU function. For performance critical applications, efficiency issues can be alleviated through writing a customized operation in CUDA (Appendix F). In Table I we compare the latency induced by the native PyTorch implementation against our customized METASIN operator with fused CUDA kernel functions. Our optimized implementation significantly reduces the overhead of the native implementation. During forward pass, executing METASIN with K 10 components requires less than three times the time it takes to execute RELU function, whereas the backward computation is only slightly more expensive than a RELU activation. Our efficient implementation also uses roughly the same amount of memory during execution RELU activation, making METASIN networks feasible to use in practice on the same hardware that a comparable RELU network designed for.
Module
Impl.
Forward (rel)
Backward (rel)
PyTorch Native
26. 1x
1.58x
METASIN
PyTorch CUDA
2.9x
1.12x
Table 1: Latency of executing METASIN activation with K 10 using native efficient
implementations, relative to the latency of executing a single RELU activation on Nvidia RTX 3090.
4 Experiments
We present experiments where we apply the three step procedure discussed at the end of Section 3 Specifically, we switch existing architecture's 's activations METASIN, and re-train the resulting new model from scratch using KD-Bootstrapping.
Activation
SIREN KD-B
MRELU KD-B
METASIN
Factor
RELU
RELUg
SNAKE
MISH
SIREN
MRELU
METASIN
KD-B
Px2
33.03
33.04
32.95
32.96
31.75
32.76
32.78
33.02
33.02
33.26
Px3
29.36
29.36
29.16
29.06
28.18
28.98
29.16
29.36
29.35
29.58
Px4
27.09
27.10
26.97
26.89
26.20
26.83
26.97
27.10
27.11
27.29
Table 2: Comparison of PSNRs obtained from various image resampling models. See text for details.
4.1 Image Resampling
In image resampling our approach builds upon the state-of-the-art network proposed by Bernasconi al. (5], which incorporates ProSR [45] feature extractor with 3 residual blocks, resampling layer, and a prediction layer. The network takes an input image and a warp grid and performs image resampling. We trained baseline networks following the exact same training procedures and dataset as described in (5) The METASIN network is obtained by replacing RELU activations of the feature
extractor with METASIN with K 10 and s!l j at initialization. All other parameters are initialized as described in Section 3.1 The transition to METASIN activations increases the amount of compute by 3% (and similarly the wall-clock inference time by 3%) compared to the baseline RELU network. To rule out any improvement due to this increase in computation, we accordingly increase the number of channels of baseline RELU network, which we label as RELUg.
We evaluate the models on three predetermined sets of projective transforms with average local scaling factors of 2 (Px2), 3 (Pxx) and 4 (Px1). We report the PSNR scores of the resampled images Table 2 as well as visual examples in Figure 2 We use KD-Bootsrapping during the first 10% of the training procedure. On average the METASIN network with KD KD-tootstrapppng improves over the RELUg baseline by 0.2dB, setting new state-of-the-art image resampling. We also conducted KD- ID-Bootstrapping RELU, MISH and SNAKE networks, but observed no improvement.
5

---

word
word
word
Reference
RELU
METASIN
GG
Reference
RELU
METASIN
Reference
RELU
METASIN
Figure Example upsampling results comparing the state-of-the-ar model with RELU activations and the corresponding METASIN network.
This outcome suggests that the training process for these networks is inherently stable, making LD-Bootstrapping unnecessary for these activations. Refer Appendix K for additional results.
4.2 Denoising Monte-Carlo Rendered Images
Next, we present our experiments in denoising Monte-Carlo rendered images. Our base denoiser network U-Net architecture used in [38] and use the same procedure generate noisy and reference renderings. Since we focus on direct image prediction models rather than kernel denoising in 38) our U-Net denoiser (DPCN) directly predicts pixel values of the clean image (For completeness, the results of kernel denoising (KPCN) are presented in Appendix M). During evaluation, we denoise images rendered with [2, 4, 8, 16] samples per pixel and use SMAPE, FLIP (2] 1-MS SSIM, and 1-SSIM as our metrics. Our best METASIN network is configured with K = 5 and the frequencies are initialized as f j/2. We LD-bootsrapping during the first 5% the training procedure. We present a summary of various METASIN network configurations in Table 3 and selected visual examples in Figure
METASIN-5 (fz =f=1))
METASIN-10 KD-B (Glorot)
METASIN-5 KD-B (f, =
METASIN-5
Metric
DPCN
KD-B (f) = j/2)
SMAPE
3.351
3.16 (-5.7%)
3.118 (-6.95%)
3.088 (-7.55%)
3.081 (-8.06%)
FLIP
1.084
0.991
(-8.58%)
0.972 (-10.33%)
0.976 (-9.96%)
0.965 (-10.98%)
1-MS SSIM
3.106
2.838 (-8.63%)
2.784 (-10.37%)
2.757 (-11.24%)
2.733 (-12.01%)
1-SSIM
6.895
6.385 (-7.4%)
6.227 (-9.69%)
6.146 (-10.86%)
6.112 (-11.36%)
Table Results of various METASIN configurations. All relative values are in relation to the DPCN baseline. The set of test samples contains all samples with spp < 16. The frequency initializations are given in parenthesis, which shows that a slight improvement can made by initializing as fy = j/2 rather than the default f, j. Switching to Glorot initialization results in reduced accuracy despite the doubled K. Introducing KD-Bot BoDoootstrapping during training significantly improves accuracy.
Discussion
In this section we present ablation experiments and an exploratory study on using METASIN activa- tions for image classification and implicit image representation using ML.Ps We present preliminary investigations of further applications of METASIN activation in Appendix

---

Reference (Full)
Baseline
METASIN
Reference (Crop)
Figure 3:
Monte-Carlo denoising results. The first row shows results produced with an input noisy images rendered at 4 samples-per-pixel, whereas the input in the remaining two rows are rendered at 16 samples- cemples--er-pixxe per-pixel.
5.1
Comparison with Alternative Activations
MRELU-10 KD-B
METASIN-10
Metric
DPCN
MRELU-10
MISH
METASIN-10
KD-B
SMAPE
3.351
3.472 (+3.61%)
3.254 (-2.89%)
3.293 (-1.73%)
3.346 (-0.15%)
3.118 (-6.95%)
FLIP
1.084
1.096 (+1.11%)
1.031 (-4.89%)
1.054 (-2.77%)
1.06 (-2.21%)
0.972 (-10.33%)
1-MS SSIM
3.106
3.397 (49.37%)
2.963 (-4.6%)
3.011 (-3.06%)
3.1666 (-1.29%)
2.784 (-10.37%)
1-SSIM
6.895
7.044 (+2. 16%)
6.589 (-4.44%) 6.896 (+0.01%)
6.736 (-2.31%)
6.227 (-9.69%)
Table 4:
Comparison of various alternative activations on the DPCN model discussed in Section 4.2 All parameters are initialized using Glorot initialization. See for discussion.
ran several experiments to assess the comparative improvements that can achieved by using alternative activations instead of METASIN. In addition to RELU, we also tested SNAKE (z + sin (z))
in (49 MISH (z tanh [In + e")) in 30), and MRELU which we linearly combine several RELU
activations in the form E/= Cj RELU + by) (Similar to Adaptive Piecewise Linear Units 3 SNAKE activation application, which in turn falls below the corresponding METASIN activated network (Table 2 3 col). In case of Monte-Carlo denoising it was not possible to train the U-Net model with SNAKE activations as the training diverged. MISH similarly performs slightly worse than both RELU and SNAKE in image resampling (Table 2 4th col).
). The
49
performs slightly worse than the RELU baseline the image resampling
In Monte Carlo denoising the MISH activated network performs better than the RELU baseline by 3% in terms of MS $SSSS however still falls significantly behind the METASIN activated network that achieves 10% improvement in the experiment (Table
The MRELU activation with K 10 performs significantly worse than the RELU baseline both in denoising and resampling task (Table 4 3rd col and Table 2 7th col). We also tested applying

---

KD-Bootstrap LD-Bootstrapping to MRELU which made big improvement in all metrics, still, however, scoring far behind the corresponding METASIN network.
Overall, METASIN activations ended up consistently yielding notably better results than the activations tested in all our experiments. In Appendix H present further comparisons with ensemble activations.
5.2 The Effect of Hy Hyprraarametess
The METASIN activation as formulated in Equation 4 has a single hyperparameter K that determines the number of constituent sine waves. While in our experiments
often set K 10, we also run ablation experiments to measure the effect of different K values. In these experiments we used the exact same settings as in Section [4.2] except varying the K parameter. In Table we show the effect of changing K to 1. 5, and 12, compared to the baseline where K = 10. Not surprisingly, lowering K results in consistently worse results according all four metrics we tested. These findings also show that the results that we presented in Section 4.2 can be further improved simply by setting K = 12 (See Appendix [or further discussion). Aside from K, another decision be made when using METASIN activations is whether to train with LD-Bootstrapping or not. While in toy examples we haven't 't noticed any difference between the two alternatives, the last two columns in Table show a comparison with and without KD-B HDD-Botsrapping, which clearly demonstrates the benefit of employing KD- JD-Bootsrapping in a real-world scenario.
METASIN-10 KD-B
METASIN-1 KD-B
METASIN-5 KD-B
METASIN-12 KD-B
Metric
FLIP
0.972
0.997 (+2.57%)
0.979 (+0.72%)
0.967 (-0.51%)
1-MS_SSIN
2.784
2.825 (+1.47%)
2.805 (+0.75%)
2.767 (-0.61%) %)
Table 5: Comparison of DPCN networks (discussed in Section
4.2p with various K parameters trained
with KD- AD-Bootstrapping and Glorot frequency initialization.
5.3 Image Classification using Convolutional METASIN Networks
We ran several preliminary experiments for exploring the use of METASIN networks outside our target domain of image prediction tasks. Here we report the results various experiments in classification setting, where we trained various student Wide-ResNets using supervision from comparably larger Wide-ResNets that we used as teacher networks. As the teacher network in this case is a larger network with higher accuracy, replicating its predictions becomes a reasonable objective. Therefore, we employ feature knowledge distillation throughout the entire training procedure. The architecture [47] follows the same architectural design ResNet [5], except that the number of channels in each residual block expanded.
The METASIN activated students are identical to their RELU activated counterparts, except the activations are replaced by METASIN with K 8. We ran our experiments on the CIFAR100 [23 dataset, which consists of 50K training and 10K validation images with resolution 32 32. The student networks were trained for 300 epochs using standard cross-entropy loss following the knowledge distillation methodology discussed in (3).. Whit While not setting a new state-of-the-are Table the METASIN student networks consistently outperform their RELU counterparts, and in some cases also the corresponding SWISHI student networks, motivating further investigation of using METASIN activations in classification tasks. However, we note that the accuracy of METASIN models are less competitive compared to alternatives when models are trained from scratch without knowledge distillation (Appendix(]) D).
that
5.4 METASIN with Multi-Layer Perceptrons
While in this work we focused on applications of METASIN with convolutional neural networks, we also ran several exploratory experiments where we used METASIN activations with MLPs to perform implicit neural representation tasks as in (39]. We refer the reader Appendix A.TT a discussion on video overfitting, where the METASIN network achieves 1-4 dB PSNR improvement over the SIREN

---

Teacher
WRN-40-2
WRN-40-4
WRN-28-2
WRN-10-2
Student
WRN-4D-1
WRN-4D-1
WRN-16-2
WR-N1662
Teacher Acc.
76.08
78.58
74,82
76.08
Activ.
RELU
METASIN
SWISH|
RELU
METASIN
SWISH
RELU
METASIN
SWISH
RELU
METASIN
SWISH
70.53
72.55
71.76
72.33
73.65
Table 6: Comparing RELU student networks with METASIN activated networks with otherwise identical architecture, both trained through knowledge distillation from the same RELU activated teacher.
baseline with noticeably less visual artifacts. In Appendix A.3 we discuss a voxel grid overfitting experiment where the METASIN network exhibits significantly improved detail reconstruction over the RELU baseline. We also present preliminary novel view synthesis experiment where we compare baseline Neural Radiance Field (NeRF) method [29 with corresponding METASIN model with K 10.
We present example result from this experiment in Figure 4 and refer the reader Appendix (A.2 for more details. Despite not utilizing positional encoding, the METASIN network surpasses vanilla NeRF in performance. It has been observed that Fourier features[42]. which bear similarities to positional encoding, aid in learning feature engineering may not be crucial, superior performance. While this result is intriguing, we acknowledge recent techniques that leverage latent codes in the form of memory, combined with specialized data structures [31(41), which can significantly enhance vanilla NeRF in terms of performance and training time. While not conclusive, hope the initial results share in this section motivate further investigation of METASIN activations in the context of other architectures and application areas.
functions. This experiment implies that such METASIN network achieves comparable or
Ground Truth
NcRF-PyTorch (20.78 dB)
METASIN-10 (21.46 dB)
Figure Zoomed qualitative results on scene flower
28 Comparison between ground-truth,
NeRF (29 and METASIN-10 model, with corresponding PSNR scores in parenthesis.
6 Conclusion and Limitations
We presented a novel activation function with trainable parameters that improves upon state-of-the-ar convolutional image prediction models. Our work has several limitations. While METASIN networks can reliably be trained without training with SD-Bootstrapping helps the network to eventually converge to better minima. While it is not uncommon when training METASIN network to have access to a comparable trained RELU network, nevertheless this requirement might lead to some extra effort when a pre-trained base model is not available. Additionally, in our current formulation (Equation 4 the number of sin components K is a hyperparameter that we set manually at design time. We performed preliminary experiments with determining К automatically in adaptive fashion, which would an interesting direction to study more in the future.
The METASIN function is only CO in contrast to the infinitely differentiable SIREN. The (o property allows SIREN networks to be trained using loss functions on the gradient or Hessian of the network. This distinct characteristic of the SIREN network is used later in 24 to automatically integrate the function that appears neural volume rendering. The lack of gradient loss METASIN networks seems to limit the performance of METASIN activations for applications such as implicit 3D shape representations (Appendix Nj. would be interesting to tackle this inherent limitation of METASIN networks in future work.

---

Broader Impact
Fundamentally, neural network blocks comprise a linear transformation and a non-linear activation. Our work introduces new activation function that enables significant prediction accuracy improve- ments over previous alternatives. Being at such a fundamental level, our proposed contribution thus be applied to wide array of neural network types that perform variety of tasks and facilitate improvements upon existing state-of-the thtt--ffthhe-rt results. The resulting improved models may be leveraged for both positive and negative ends.
Acknowledgements
We thank Xianyao Zhang, Marios Pappas, Michael Bernasconi, and Aziz Djelouah for their help in Monte-Carlo denoising and resampling experiments, Katarina Tothova for proof-reading our manuscript. We also thank Wig42 for the "RoomChair" and "LivingRoom" images, Xianyao Zhang for the "furball" image in Figure nacimus for "Can" image, SlykDrako for the "BedRoom" image in Figure [5] The dataset for the Nerf experiments were provided by Matthew Tancik and Ben Mildenhall.
References
[1] The stanford 3d scanning repository. 3Dscanrep/
URL http:://rraphics..cre wtw.:ccrsssaaaoor..o.oo www.:ordddddddddeeeeeeeeeeeeeeee edu/data.
[2] Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark Fairchild. Flip: difference evaluator for alternating images. Proc. ACM Comput. Graph. Interact. Tech., 352):15-1, 2020.
[3] Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete. A survey on modern trainable activation functions. Neural Networks, 138:14-32, 2021.
[4] Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. Kerne--predictng convolutional networks for denois- ing monte carlo renderings. ACM Trans. Graph., 36(4):97-1, 2017.
[5] Michael Bernasconi, Abdelaziz Djelouah, Farnood Salehi, Markus Gross, and Christopher Schroers. Kernel aware resampler. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[6] Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai. Hu, Yun Fu, Denys Makovi- ichuk, S. Tulyakov, and Jian Ren. Real-time neural light field mobile devices. ArXiv, abs/2212.08057. 2022.
[7] Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning, 2020.
[8] Ka-Hou Chan, Sio-Kei Im, Wei Ke, and Ngan-Lin Lei. SinpIn]: A fast convergence activation function for convolutional neural networks. In 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion), pages 365-369, 2018 doi:
10.1 1 109///C/CCCC
CaAeCCCComaanio.222 Companion.2018.000 2018.00082.
[9] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8628-8638, 2021.
[10] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. Advances in neural information processing systems, 32, 2019.
[11] Drork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (elus), 2016.
10

---

[12] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks, In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 315-323. PMLR, 11-13 Apr 2011.
[13] Mark Harmon and Diego Klabjan. Activation ensembles for deep neural networks, 2017.
[14] Kaiming Hc, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification, 2015.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision ECCV 2016, pages 630-645, 2016.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2020.
[18) Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, (15):3593666 1989.
[19] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[20] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2020.
[21] Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Jyoti Mitra. Relu fields: The little non-linearity that could. ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[22] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks, 2017.
[23] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[24] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14556-14565, 2021.
[25] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022.
[26] Andrew Maas, Awni Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Atlanta, Georgia, USA, 2013.
[27] Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Man- mohan Chandraker. Modulated periodic activations for generalizable local functional represen- tations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 0-214-14223, October 2021.
[28] Ben Mildenhall, Pratul P. Srinivasan. Rodrigo Ortiz- Cazon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar, Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. 38(4), jul 2019.
[29] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[30] Diganta Misra. Mish: A self regularized non-monotonic activation function, 2020.
11

---

[31] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4): 1-15, 2022.
[32] Ashis Paul, Rajarshi Bandyopadhyay, Jin Hee Yoon, Zong Woo Geem, and Ram Sarkar. SinLU: Sinu-Sigmoidal linear unit. Sci. China Ser. A Math., 10(3):337, January 2022.
[33] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. International Conference Machine Learning, pages 5301-5310 PMLR, 2019.
[34] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.
[35]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 1412.5555 2014.
[36] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA,
May 7-9, 2015, Conference Track Proceedings, 2015.
[37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Citrventtoo-MIICAA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241, Springer, 2015.
[38] Farnood Salchi, Marco Manzi, Gerhard Roethlin, Romann Weber, Christopher Schroers, and Marios Papas. Deep adaptive sampling and reconstruction using analytic distributions. ACM Transactions Graphics (TOG), 4166):1-16, 2022.
[39] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7477, 2020.
[40] Sanghyun Son and Kyoung Mu Lee. Srwarp: Generalized image super-resolution under arbitrary transformation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7782-7791, 2021.
[41] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[42] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kei,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG), 37(4):1-15, 2018.
[45] Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine- Hornung, Olga Sorkine- Hornung, and Christopher Schroers. A fully progressive approach to single-image super- resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 864-873, 2018
[46] Lin Yen-Chen. Nerf-pytorch.
2020.
2020.
[47] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
12

---

[48] Xianyao Zhang, Marco Manzi, Thijs Vogels, Henrik Dahlberg, Markus Gross, and Marios Papas. Deep compositional denoising for high-quality monte carlo rendering. In Computer Graphics Forum, volume 40, pages 1-13. Wiley Online Library, 2021.
[49] Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 13:1583-1594 2020.
13

---

Appendix
A Use of METASIN Beyond Image Prediction
In addition to our main results presented in Section 4 of the paper. we also performed various exploratory experiments investigate further application cases of the METASIN activations. The experiments cover image classification where we show favorable results of using convolutional well various overfitting experiments to
METASIN networks over baseline RELU networks, explore the use of METASIN activations with MLPs. We present our preliminary findings here with the hope of motivating further investigation in these directions.
In all MLP experiments, we use METASIN with K = 10 sine components, and distribute the frequencies evenly across the range 35]. The initialization of the remaining parameters follows the description provided in Section
Target
SIREN
METASIN
Wall
Car
Bicycle
Figure 5: Visualization of selected reconstructed frames of the video. To fully appreciate the details and visual cues presented in the figure, we recommend visualizing the figures in color and zooming in for a more comprchensive analysis.
Video Dataset
Activation
Model
Channels
Depth
MSE
PSNR (dB)
SIREN
MLP
1024
3
2.60 10-3
32.26
Cat
METASIN
MLP
1024
3
1.96
10-3
33.32
SIREN
ML.P
1024
3
1.69
10-3
34.50
Bikes
METASIN
MLP
1024
3
5.58
10-4
39.32
Table 7: Results on implicit video representation.
A.1 Video Overfitting using METASIN MLP
In this section, we present the overfitting experiments on videos using an MLP network with 3 hidden layers and 1024 neurons to overfit video. The input is 3 dimensional vector consisting of frame ID and pixel coordinate, and the output the RGB value the corresponding pixel. The METASIN network improves the PNSR by 1-4 dB compared to the SIREN baseline shown in Table
Overall the Cat dataset has more fine details, making more challenging to accurately represent these intricate details. Consequently, the PSNR for the Cat dataset is comparatively lower in comparison to Bikes dataset. The results are depicted in Figure While RELU networks struggle to represent the signal, both SIREN network and METASIN can reconstruct the frames. Notably, the METASIN network reconstructs more details, see Figure 5 (Bicycle) and has less noise, as we see in Figure 5 (Wall and Car). It is worth mentioning that various techniques, such as positional encoding or Fourier
14

---

Ground Truth
NeRF
METASIN-LE
Figure 6: Zoomed qualitative results static forward scenes NeRF and METASIN-10 10 model.
28
Comparison between ground-truth,
features 43 42
well utilizing latent codes 31. have shown significant improvements in RELU
networks. However, exploring the impact of these techniques when applied conjunction with METASIN activation is direction we defer future work.
Method
Leaves
Orchids
Flower
T-Rex
Horns
NeRF 29
20.92
20.36
27.40
26.80
27.45
METASIN-10 10
21.46
20.52
28.20
26.81
27.98
Table 8: Per-scene PSNR1 comparison. In this table, we report the results obtained by running NeRF PyTorch implementation, based on [46] and previously used in [21]6], and the results of our proposed METASIN. Our method achieves higher PSNR all tested scenes.
A.2 Novel View Synthesis using METASIN MLP
(NeRF) enable synthesizing novel views with only sparse set of input views. NeRF methods exploit positional encoding to map the input into high-frequency space, which is the key to high-quality scene rendering, which we drop in our experiments and replace the standard RELU with 10 components METASIN-10 activation. Frequencies and phases are initialized uniformly over [1, 35) and [0, 2x). The experiment setting the same as [29], except that we extend the training process to 250K iterations instead of 200K, in order to make sure that METASIN models
Neural Radiance Fields 29
converge well. Our implementation based on 46 The results are reported in Table g We also
provide comparison of qualitative results in Figure
15

---

GT
RELU (MSE=0.252)
METASIN-JO (MSE=0.044)
Figure 7: False-color visualization of the reconstructed density grid where METASIN is configured with K 10
A.3
Density Overfitting using METASIN MLP
In this section we present an overfitting experiment on a 3D voxel grid. The smoke scene we used in this experiment contains 175 232 x 175 density values within range [0, 1]. Following the MLP architecture used in 39], we map the 3D coordinates to the corresponding density value using RELU and METASIN activated networks. Comparisons between the METASIN activated network and the baseline RELU model are presented in Figure
B Details of Overfitting Experiment with Convolutional Network
Table shows further architectural details of the overfitting experiment we discuss in Section 3f the main paper. We perform experiments with two base convolutional architectures, shallow and deep network. M-SIREN refers to the amplitude modulated STREN (27] which has significantly more parameters due to the introduction of an additional amplitude modulation network.
Input
Target
RELU
SIREN
M-SIREN
METASIN
Shallow CNN
29.81 dB
32.55 dB
30.88 dB
35.04 dB
Deep CNN
30.36 dB
12.65 dB
35.70 dB
40.01 dB
Figure Peak Signal-to-Noise Ratio (PSNR) comparison of different activations employed on CNNs
for image up-sampling (with 16x of upsampling factor).
Figure shows a visual comparison reconstructed images by the different versions of both base networks. Both the shallow and deep versions of the METASIN network can be trained reliably (unlike their sin activated counterparts that diverge in the case of the deep architecture). The METASIN networks come with relatively little parameter overhead (the relative overhead becomes even smaller with larger networks that have more channels, as discussed in Appendix and produce visually sharper results that also closer to the reference in terms of PSNR.
16

---

CNN Size
Activation
# Param.
Depth
PSNR (dB)
RELU
75 K (Ix)
4
29.81
SIREN 39
76 K (1x)
4
32.55
Shallow
M-SIREN 27
281 K (3.7x)
35.04 4
30.88
METASIN (ours)
82 K (1.1x)
RELU
408 K. (1x)
13
30.36
SIREN
39
408 K (1x)
13
12.65
Deep
M-SIREN 27
651 K (1.6x)
13
35.70
METASIN (ours)
432 K (1.1x)
13
40.01
Table 9: Comparison of different activations employed on CNNs trained for image up-sampling (16x).
1.0
0.5
Target Pred
Loss 0.252
0.0
-0.5
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
Siren f=
1.0
0.5
Target Pred
L1 Loss 0.319
-0.5
-1.0
-0.2
0.4
0.6
Siren f=
1.0
0.5
Target Fred
Loss 0.090
0.0
-0.5
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
Siren f= 10
1.0
0.5
Target Pred
L1 Loss 0.021
0.0
-05
-1.0
-0.4
-0.2
0.4
0.6
MetaSin K
1.0
0.5
Target
Loss 0.013
0055
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
Figure 9: Representing y(() = sin(10L + 5 sin(())) x
with 3 layer FC networks
C Sensitivity to Frequency Initialization
Although the sin activation function has promising properties outlined in the main paper, it is known that they can be extremely difficult to train [9 44
We illustrate this in a toy experiment where we try to reconstruct the signal y(t) = sin (10r - 5i (10)) x s10 (in(0))))) when the input z(()
Most of energy of yy() distributed between frequencies between 10 and 170. We sample / from the normal distribution and train fully connected networks with input neuron, 3 layers having neurons each, and output neuron. In one of the networks, we use RELU activation in between fully connected layers except for the last
17

---

one. In the remaining two networks, we use sin(fa). In one we initialize f = 1, in the other one we initialize f and f = 10. We use L1 loss and Adam as the optimizer. The results are shown in Figure 9 As it is seen, REL.Unettork can reconstruct part of the signal. The sin-network with the initialization f does not learn all, while the network initialized with f = is able to match y(t), and with f = 10 the model fits the original signal reasonably well. This suggests the importance of initialization and the difficulty training of the sin activation function.
On the other hand, a METASIN network with the frequency parameters from Equation 1 j and K 10 effectively covers wide range of frequencies including the one that is important to represent the signal. Consequently, using METASIN we are able to lower the loss to 0.013 and eliminate the trial and error process for initializing frequencies.
+ initialized
Figure 10 we additionally plot the results an experiment where we sweep through a wide range of we values for fully-connected SIREN network that was trained to overfit to single image. The plot illustrates that only a narrow range among possible initial we values results in a plausible training error, confirming the importance of initialization in SIREN networks.
50
10a
40
Loss
PSNR (dB)
100
Loss
20
10-2
10
0
10-3
10
10
102
10a
10a
SIREN Wo
Figure 10: Sensitivity of SIREN w.r.t. base frequency Why In this figure, reported the PSNR and MSE Loss values across a wide range of SIREN's 's base frequency wo values. For each wp. we trained SIREN network and showed that only restricted set of frequencies (around 10?) converges to sufficiently good local minima. Specifically, in this experiment, we train the networks to up-sample the image reported in Figure [ with up-scale of x16 for 100, 000 iterations using the Adam optimizer with a learning rate of 10-5
D Connection to Fourier Series
Well-behaved functions, such as continuous functions can be expressed as Fourier series with finitely many terms. Let us define function as Fourier series:
2TT
2mj p
o(x) ag + aj
by sin
(5)
where K E N+ is finite, and P denotes length of period. Using the harmonic addition rule, i.e.
a cos a + bsin ccos (a + r), where
sgn(a) a² 62
(6)
arctan
we can rewrite Equation 5 as:
2
((x)= 0g +
On cos
P x x
(7)
j=1
18

---

Substituting in the identity cosa -snn (a 2 we get:
[
ạ 2r
o(x) - ag +
On sin
x
(8)
j=l
Dropping the constant term we obtain the following:
(xx) Σ
oj
sin (J,x + pj), where
j=1
2TT
(9)
fj and
bj
P1 arctan
jj
From Equation we see that any Fourier Series be expressed METASIN activation
(x) coRELU(X)) +
Σ
oj sin (f pj).
(10)
by setting = 0 and the rest of its parameters {cz, f3. Pzj K]} accordingly.
E METASIN Overhead
METASIN parameter overhead formula for a convolutional layer compared to RELU baseline is as follows:
3 K
overhead kernel-size" input-channels
(11)
For example if we have a layer with kernel-size 3 with 128 input-channels, then overhead for K 10 is 0.026. Which means we have approximately 2.6% more parameters. If input-channels is 32 the overhead becomes 10.08%. On the other hand for kernel-size = 5 and 512 input-channels the overhead is 0.24%.
Details on C++/CUDA Implementation
To account for the inefficiency of a native Python API implementation of METASIN we implemented custom-optimized fused CUDA kernels for both forward and backward functions of the MetaSin activation in C++. Our implementation can be integrated into the PyTorch and TensorFlow Python APIs. Throughout the development process we also tested native automatic compilation functionalities provided by both frameworks (specifically: jit, torch.compile for PyTorch, and jit and XLA in TensorFlow). While notable improvements over the baseline Python API implementation can be made through the use of these out-of-the tit-off-hee-oxx facilities, obtained the best performances in terms of speed and memory consumption using our custom-designed CUDA functions.
Some of the techniques we utilized in our code are follows: in order to optimize the memory footprint and the inference speed of the METASIN, we remove the intermediate quantities that autograd engine computes and instead compute the output and gradient tensors directly from the input and the METASIN parameters. Moreover, we further optimized the computation speed with improved caching and reduction strategy warp and block levels. Mcanwhile, have exploited a 2-level reduction strategy based on pairwise sunmation in our backward kernel. In this way, we could avoid numerical errors in gradient tensors and achieve comparable accuracy to autograd in float32 precision.
The reduction in computational overhead using the aforementioned optimizations enabled running compute-intensive experiments we present throughout the paper in a feasible manner, and we believe demonstrates the viability of METASIN activations for most practical tasks. That being said, other optimizations that we were not able to explore due time constraints could help reduce the overhead even further, which is an interesting direction for future research.
19

---

Forward Timing [ms]
Backward Timing [ms)]
10a
native cuda
native cuda
Time (ms)
[ms]
10.4
22
MetaSin Components
MeteSin Components
Figure 11: Effect of METASIN components to latency.
We also investigated the computational scaling behavior of our implementation with respect the number of sin components K. To this end, we ran benchmark experiment, in which we compared forward and backward latencies of RELU, METASIN Native, and METASIN CUDA functions on the same input tensors using different K values. The results can be found in Figure
iter: 0
iter: 33K
iter: 66K
iter: 100K
iter: 100K
Low Freq.
we weight
2
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
iter:
iter:
100K
Wide Freq.
abs weight
2
0
www.
0
Ya www.wal
-2.5
-2.5
2.5
-2.5
2.5
-2.5
0.0
2.5
iter: 0
iter: 33K
iter: 66K
iter: 100K
iter: 100K
High Freq.
abs weight
2
2
-2.5
0.0
2.5
2.5
0.0
2.5
2.5
0.0
2.5
-2.
0.0
2.5
Figure 12: METASIN activation shape evolution time and final distribution of frequency param- eters. In this figure, we report the behavior of three activated neurons (of the same layer) over the training process.
G Evolution of METASIN Activations during Training
Figure [2] shows illustrative examples the evolution of META activations during the course of training in a toy example. In this experiment the weight of the RELU component is set to 1, whereas the amplitudes all sin components are set to 0. Thus a METASIN activation starts out RELU, but then evolves into various shapes depending via backpropagation. is worth noting that the shape after the initial phase of training, which agrees with the observation that weights change very slowly in overparameterized networks. Moreover, the different rows in the figure show three different METASIN neurons of the same layer that interestingly converge to complementary frequency response that can capture different frequency components of the input signal.
the training of the shape parameters METASIN activation tends to not change
To shed some light the shapes that METASIN activations take in more practically relevant setting with and without -D-tootstrapping, we additionally produced visualization of the METASIN shapes from the resampling network described in Section 4.1
The METASIN shapes obtained from the
20

---

first and last blocks of models trained from scratch and with KD Bootstrapping. This visualization in Figure [3] shows that, while there is significant local variation between individual METASIN activations, globally the rough RELU shape is still discernible even without any involvement from the RELU teacher.
With KD Bootstrapping
Trained from Scratch (w/o KD-B)
First Block
Last Block
Figure 13: Visualization of METASIN shapes with and without KD bootstrapping.
H
Comparison with Ensemble Activations
We ran additional experiments using ensemble activations as described in a recent survey [3], namely Adaptive Blending Units (abu), Variable AFs (vaf), Adaptive AFs (aaf) in the same experimental setting that we used to generate the first row of Figure | in the main paper. We present the best results obtained from multiple runs with different initializations in Table T0 also including RELU and META SIN as references:
abu
vaf
aaf
RELU
METASIN
[SNR[db]
29.79
28.98
29.06
29.81
35.04
Table 10: Comparison various enseble activations in an image overfitting setting.
We also ran two experiments in image resampling setting using abu and aaf versions of the model from Section 4.1 which we present in Table II
21

---

201
201
201
Reference
RELU
METASIN
Figure 14: Additional upsampling results comparing the state-of-the-ar model with RELU activa- tions (5] and the corresponding METASIN network.
Activation
Factor
abu
aaf
RELU METASIN w/ KD-B
Px2
33.02
32.97
33.03
33.26
Px3
29.31
29.28
29.36
29.58
Px4
27.09
27.10
27.09
27.29
Table 11: Comparison of various ensemble activations for image resampling
I Effect of Setting K to a high number
We note that further improvements in terms of model accuracy can be made by pushing K beyond what we present the paper. We did not go beyond 10-12 as throughout our experiments observed that doing so often resulted in diminishing returns. As a more concrete example we provide results from a resampling experiment in Table [2 (Note that these results are from a preliminary run with slightly inferior hyperparameters compared to the corresponding experiments we report in the main paper, and the training was stopped prematurely after 900K iterations instead of the full 2M hence the lower PSNRs compared to T). These results suggest that choosing K 10 would indeed make sense in cases where the absolute best quality desired and additional latencies are tolerable, but overall K se 10 seems to be the happy medium.
22

---

Activation/Upsample Factor
Px2
Px3
Px1
METASIN-10
33.09
29.42
27.17
METASIN-20
33.14
29.45
27.20
Table 12: The effect of setting K to 10 and 20 on resampling quality
Classification Experiment with METASIN Model Trained from Scratch
To investigate the effect of KD-Bootstrapping in the classification setting, trained various Wide- ResNets with original RELU activations and with METASIN activations entirely from scratch and without using KD-B LD-Bootstrapping, Table 13 presents test accuracy CIFAR-100.
Model/Activationn
WRN-16-2
WRN-28-2
WRN-40-2
WRN-40-4
RELU
72.85
74.82
75.95
78.99
METASIN w/o KD-B
71.82
73.88
75.44
78.44
Table 13: Comparison of METASIN and RELU models in image classification with training without DD-BBoostrapping. In this case METASIN models only become ad advantageous when train using KD-Bootstrapping, underlining the importance of the proposed training methodology.
In accordance with the above discussion these results underline the role of KD Bootstrapping for achieving the best results with METASIN networks. To give a concrete example: as we show above, 71.82 accuracy is behind its RELU counterpart
when training from scratch WRN-16-2-4 1--2-MMTASSI (WRN-16-2-ReLU) with accuracy 72.85, On the other hand, the last column of Table shows that by distilling from WRN-40-2 teacher, the accuracy of WRN-16-2-METASE 74.10, whereas WRN-16-2- RELL achieves 73.65 accuracy using the same procedure.
be brought up to
K Resampler Architecture and Additional Results
Our network architecture is based on the design proposed in (5] We utilize ProSR feature extractor [45] consisting of 3 residual blocks with 4 dense blocks within each residual block and growth rate of 40 and 160 channels. The subsequent prediction layer is implemented an MLP with 4 hidden layers and 256 neurons. In our experiments, this prediction layer remains unchanged. We apply METASIN activations only to the convolutional feature extractor. We present additional results in Figure 14 for the resampler experiment presented in Section 4.1
L DPCN Denoiser Architecture and Additional Results
The design of the U-Net architecture we use in Section detailed in Table 14
We also present
Encoder ResBlocks
Encoder Channels
Decoder ResBlocks
Decoder Channels
Scale
2
64
2
64
2
128
2
128
2
256
2
256
256
2
256
2
256
2
256
Table 14: The Configuration of the Standard DPCN U-Net. Scale 0 is the full-resolution scale, From one scale the next one, the image resolution is halved in both dimensions.
additional results in Figure 15 to the results we presented in Section 4.2 in the main paper.
23

---

Reference (Full)
DPCN
METASIN-S w/ KD-B
Reference
Figure 15: Additional results to Section 4.2
M KPAL Denoiser Results
In order to factor out the potential effects of kernel prediction to our denoising experiments, we utilized a direct prediction version of the sfate-of-the-ar Monte-Carlo denoiser presented (38). In this section we present results of training a METASIN version of the original kernel prediction network. The improvement over the baseline RELU network remains significant in this experiment across all metrics. An interesting observation from this study was the significantly reduced effect of JD-Bootstrapping, which we eventually decided not to employ to produce the results Table [5 We hypothesize that the lesser effect of KD- AD-Bootstrapping in this case can be attributed to kernel prediction, which however requires further investigation.
Metric
KPAL
METASIN-II (Glorot)
SMAPE
3.098
2.966 (-4.26%)
flip loss
0.934
0.893 (-4.39%)
1-MS SSIM
2.893
2.622 (-9.37%)
1-SSIM
6.227
5.934 (-4.71%)
Table 15: Comparison of the state-of-the-arr kernel predicting denoiser 38 and its significantly improved METASIN version.
N
Failure Cases
While our preliminary experiments on using METASIN activations with ML.Ps generally resulted in improvements over corresponding baselines, we also observed some failure cases. A particular

---

experiment where the tested METASIN network did not perform well was on signed distance field (SDF) overfitting. In this case, the SIREN reproduction of the ground-truth truth SDF better fine scale details than the corresponding METASIN reproduction. We hypothesize that this discrepancy in performance due to the gradient loss that SIREN employs and the METASIN network lacks due to not being continuously differentiable. We defer further investigation of this limitation to future work.
(a) GT
(b) SIREN
(c) fj = ((1,35)
Figure 16: SDF overfitting Thai Statue 1. Here the SIREN network produces better representa- tion of the underlying 3D shape than a comparable METASIN activated MLP.
25


Reviews:
[{"review_id": "FURqpzTEfQ", "reviewer": "Reviewer_N8tS", "summary": "The authors of this paper propose a new activation function, which relies on a parametrized sinusoidal function instead of only a piecewise linear function. The authors show how this function can lead to performance improvements in the setting of denoising.", "questions": "I would be grateful if the authors could elaborate on the issues encountered during training from scratch with their proposed activation. The way the paper is written, it seems to me that training with MetaSin would suffer from the same problems as previous work using sinusoidal activations e.g. SIREN, which is somewhat surprising given that MetaSin contains ReLU as a special case.", "limitations": "The authors have adequately addressed the limitations of their work. Furthermore, I see no immediate negative societal impact arising from this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors propose a novel formulation for an activation function, relying on a sum of sines of varying amplitude, frequency and phase (all of which are parameters learned during training). The formulation of this activation function is novel, as far as I am aware.\n\n- I appreciate the explanation that the authors provide on the usefulness of the sine activation, namely how it enables the network to capture higher frequency components. Moreover, I appreciate the inclusion of the link to Fourier series in the Appendix, and I think it may be an interesting topic that the authors may want to elaborate on.\n\n- The authors have performed several experiments to evaluate the performance of their method, not only in their main problems of interest (denoising and resampling) but also more usual classification tasks. They have also made the effort to write optimized code for their proposed activation function (although I highly encourage releasing this to assure reproducibility).", "weaknesses": "The main weaknesses of the paper are the following (mostly concerning the experimental setup):\n\n- While the experiments cover a wide range of tasks, they contain a weakness in that most of them rely on starting from a pretrained network to achieve the best results. While the authors acknowledge this limitation, it would nevertheless improve the paper if results for training without the initial model were included.\n\n- Related to the above, the classification experiment only considers the setting where the MetaSin network learns from a pretrained teacher. I believe it would be interesting to include a classification experiment where the new architecture is trained from scratch.\n\n- In the Appendix, the authors include a plot that shows how the weights of the activation change during the course of the training. We can see that most of the weight is in the ReLU activation (in other words, the initial form of the MetaSin activation). While this is not inherently bad, it makes the benefit induced by the new activation less clear in my opinion. I believe that this may also stem from the initialization via a pretrained model (which has already converged while using a ReLU activation), and thus may be alleviated if trained from scratch. I would greatly appreciate it if the authors were able to examine this.\n\n**Post rebuttal comment**: As mentioned in my response to the author rebuttal below, while most of my concerns have been addressed, the fact that KD bootstrapping is required remains a limitation. However, I have a generally positive view of this work, given that as the authors mentioned in their comments the use of KD bootstrapping for stability, while limiting, is still an interesting approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**MetaSin without KD Bootstrapping from a pre-trained network**\n\nDuring our experiments we had a chance to confirm firsthand the well-known difficulties associated with training *sin*-based activations, especially when utilized in convolutional networks. These difficulties may even lead to divergence in training: an example can be found in Figure 1 row 2. We haven’t encountered divergence issues when training MetaSin networks with or without KD bootstrapping. That said, our experiments in which we did not utilize KD bootstrapping often yielded sub-par results. For instance, in Table 2 see the last two columns: Removing KD Bootstrapping leads to worse results on average in resampling. The same behavior can be observed likewise in Table 4 for denoising. We hope that these two experiments, where we compare our full method (that is, MetaSin with KD-B), with an ablation where we do not utilize a pre-trained network (MetaSin w/o KD-B) will help the readers to put the contribution of KD Bootstrapping into proper context. If needed, we would also be happy to modify the text to point the reader’s attention to these experiments. \n\n**Classification experiment with MetaSin from scratch**\n\nTo investigate this we trained various Wide ResNets with original ReLU activations and with MetaSin activations entirely from scratch and without using any Knowledge Distillation. The table below presents test accuracy on CIFAR-100: \n\n\n| Model/Activation  | WRN-16-2  | WRN-28-2 | WRN-40-2 | WRN-40-4 |\n| :--------:                | :-------:       | :-------:       | :-------:      | :-------:       |\n| ReLU                    | 72.85          | 74.82          | 75.95         | 78.99         |\n| MetaSin from scratch | 71.82          | 73.88         | 75.44         | 78.44         |\n\n\nIn accordance with the above discussion these results underline the role of KD Bootstrapping for achieving the best results with MetaSin networks. To give a concrete example: as we show above when training from scratch WRN-16-2-MetaSin at 71.82 accuracy lacks behind its ReLU counterpart (WRN-16-2-ReLU) with accuracy 72.85. On the other hand, Table 6 last column of our main paper shows that by distilling from a WRN-40-2 teacher, the accuracy of WRN-16-2-MetaSin can be brought up to **74.10**, whereas WRN-16-2-ReLU achieves 73.65 accuracy using the same procedure. We’d be happy to modify the paper accordingly to ensure that our side investigation into image classification is as informative as possible.\n\n\n**Shape variation of MetaSin activations**\n   \nTo shed some light on the shapes that MetaSin activations take when training from scratch, we produced a visualization of the MetaSin shapes from the resampling network described in Section 4.1. The MetaSin shapes are obtained from the first and last blocks of models trained from scratch and with KD Bootstrapping. This visualization can be found in the pdf file attached to the Author Rebuttal section. The figure shows that, while there is significant local variation between individual MetaSins, globally the rough ReLU shape is still discernible even without any involvement from the ReLU teacher.\n\n\n**Issues when training from scratch:**\n\nOne of the main challenges when training *sin*-based activations such as SIREN is their   dependence on initialization. In Appendix C we present an illustration of this behavior through a set of toy examples. Another challenge when training *sin*-based activations are inconsistent gradients due to complex shapes the activation can take, and large degeneracy in local minima caused by symmetries. These make the training of  deep networks especially unstable and may lead to divergence (See Figure 1. Deep CNN/SIREN).  \n\nWith MetaSin we avoid the aforementioned issues by having better coverage of plausible ranges of the shape parameters and introducing the additional ReLU component that stabilizes the training. As such, when training MetaSin models (either with and without KD Bootstrapping) we haven’t encountered dramatic inconsistencies in model accuracy due to initialization, nor encountered any further stability issues during training. \n\nThe main issue with training from scratch without KD Bootstrapping is that in challenging real-world problems (such as training direct prediction models for denoising and resampling) the accuracy of the model is inferior to the alternative of training with KD Bootstrapping. On the other hand, in simpler tasks, such as the various overfitting experiments we present throughout the paper, models trained from scratch tend to perform just as well. Finally, in Appendix I we present an interesting finding that a kernel-predicting version of the denoiser from Section 4.2 (as opposed to direct prediction) also trains fine without KD Bootstrapping, which we hypothesize is due to the reduced dimensionality of the problem space, but nevertheless remains an interesting direction for further investigation. \n"}, {"review_id": "9EnqPQhqae", "reviewer": "Reviewer_9soJ", "summary": "**SUMMARY AFTER REBUTTAL**: the authors have addressed most of my concerns and I have increased my score during the rebuttal phase. I believe the novelty of the paper is small if moving beyond their specialized subfield, which is why the overall score remains low.\n\n---\n\nThe paper proposes a variant of the sin activation function proposed in SIREN, specifically for image reconstruction and image denoising. \n\nInstead of using a single sin function, they consider a linear combination of a ReLU function and multiple sin functions, where the weights of the linear combination and the parameters of the sin functions are shared across layers and trained with back-propagation. The networks are trained via a simple knowledge distillation procedure after a drop-in replacement of the activation functions of the corresponding ReLU networks. \n\nA series of experiments shows better image reconstruction capabilities with a very small overhead (thanks to a custom CUDA kernel) on two major benchmarks.", "questions": "Apart from the previous weaknesses, I would be curious to see more details on their CUDA kernel. Is this done manually, or is it just the result of PyTorch compile procedures?\n\nI am also skeptical of their initialization range, since they set to 0 all weights multiplying the sin AFs. Wouldn't this prevent a good gradient flow? More plots of the resulting sin functions before and after training can improve the discussion here.\n\nWhile the paper is well written, there are some citations that appear in a wrong way (e.g., \"state-of-the-art resampler Bernasconi et al. [4]\"). There is also a small typo on P3 (\"it’s predictions\").", "limitations": "All limitations are correctly described in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well written and easy to follow.\n- Experiments are good (see some remarks below), and the improvements are consistent.\n- The motivation for the proposed AFs is only discursive, but clear.\n- As far as I know, this formulation is novel with some caveats (see below).", "weaknesses": "- My main concern is that the idea of building an AF from a linear combination of base AFs whose weights are trained is quite known in the literature. Considering for example the survey on AFs by Apicella et al., 2021 [A survey on modern trainable activation functions], they have an entire chapter on this idea including Adaptive AFs, Variable AFs, Kernel-based AFs, Adaptive Blending Unit, Adaptive Piecewise Linear Unit, etc.\n- The core contribution of this paper is to apply this idea to sin AFs, i.e., considering a different base set from the papers cited above (which combined other types of functions, such as kernels with different weights, ReLUs and sigmoids, etc.), showing this is useful in the context of image reconstruction. While interesting, this is a niche result that is best suited for a smaller conference or journal with applicative interest.\n- In addition, the authors are only comparing to standard AFs, although they consider a \"MReLU\" AF which seems similar to the ensemble AFs described above. It would be helpful to have a stronger comparison to ensemble AFs.\n\nNote that a similar paper vertical on PINNs was rejected from last ICLR (https://openreview.net/forum?id=MpGP-z07TmM) due to similar reasons.\n\nAn additional weakness is that the authors claim their method is a complete drop-in replacement, while it requires a KD step to work correctly. I believe these claims should be amended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison with ensemble activations**\n\nThroughout the paper we present comparisons against popular baselines Snake, Mish, Siren, as well as an ensemble activation we call MReLU that is similar to the Adaptive Piecewise Linear Units presented by Agostinelli et al., 2015 and mentioned in the survey by Apicella et al., 2021 (we will add citations accordingly). Tables 2 and 4 in our main paper show that MReLU consistently performs worse than MetaSin in both resampling and denoising applications. Following up on the reviewer’s comments we ran additional experiments using other ensemble activations Adaptive Blending Units (abu), Variable AFs (vaf), Adaptive AFs (aaf) in the same experimental setting that we used to generate the first row of Figure 1 in the main paper. We present the best results we obtained from multiple runs with different initializations below, also including ReLU and MetaSin as reference:\n\n| Activation |  abu | vaf | aaf | ReLU | MetaSin|\n| :--------:      | :-------:       | :-------:       | :-------:      | :-------:       | :-------:      \n| PSNR [db] |  29.79      | 28.98      |  29.06     | 29.81       |  35.04 |\n\nWe also were able to run two experiments in image resampling setting using abu and aaf versions of the model from Section 4.1, which we present below:\n\n| Activation | x2  | x3 | x4 |\n| :--------:      | :-------:       | :-------:       | :-------:      | \n| abu |  33.02       | 29.31      | 27.09 | \n| aaf  |  32.97      | 29.28       |  27.10 |\n| ReLU | 33.03   | 29.36   | 27.09 |\n| Metasin KD-B |  33.26 | 29.58 | 27.29 |\n\nOverall the ensemble activations we tested tend to perform roughly similar to ReLU. We would be happy to include the above data points in the final manuscript.\n\n**Rejected ICLR submission on PINNs**\n\nThe main criticism that led to this decision (despite otherwise favorable reviews) appears to be the argument strongly put forth by Reviewer `E27E` stating that: \n“The physics-informed activation function (PIAC, Eq. (10)) …”, which the submission claims as a novel contribution,  “... is **identical** to, among others, the soft-normalized version of the adaptive blending unit (ABU) [1]” (emphasis by `E27E`)\nAs such, we don’t believe the analogy to our work applies, since to the best of our knowledge the MetaSin formulation is novel and all of our reviewers seem to agree with us on this point.\n\n[1] Sütfeld et al., Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks, 2017\n\n\n**Results are Niche**\n\nOur main contributions in this paper include the specific formulation of the MetaSin activations that we arrive at by gradually addressing issues associated with *sin* activations as described in Section 3 in or paper, as well as the training methodology that we call KD Bootstrapping that is described in Section 3.1, which to our knowledge has not been explored in the context neither *sin*-based nor ensemble activations.\n\nWhile, on one hand, we focused our efforts on thoroughly investigating the use of MetaSin activations in two arguably core topics within our target domain of image prediction applications, namely resampling and denoising, on the other hand we believe that it is not unreasonable to assume other applications within this vast domain might benefit from the techniques we present in this paper. In support of this argument, we present various preliminary results throughout the paper, which show the effectiveness of MetaSin activations in NeRF models (Table 8 in Appendix) for predicting novel views, as well as in 2D (Table 7 in Appendix) and 3D (Figure 7 in Appendix) signal representation tasks. \n\nThe domain of image prediction applications spans a big chunk of classical research problems in image/video processing and computer graphics/vision: segmentation, matting deblurring, tone mapping, depth estimation, view interpolation, inpainting, to name a few. Moreover, one can speculate that our work might find applications in contemporary generative models relying on diffusion models considering our strong results in denoising. Taking all these exciting directions into account, we firmly believe that our work would be interesting to the broader research community. \n\n**”Drop-in Replacement” and Details on CUDA kernel**\n\nWe address both topics in detail in the Author Rebuttal section as other reviewers made similar remarks. \n\n**Initialization of MetaSin and Gradient propagation**\n\nWe initialize the weights to be 0 intentionally, as we aim to prevent the activation from having arbitrary frequencies during the initialization phase. Instead, we allow the network to determine which frequencies to use during training, as we illustrate in Figure 11 in the Appendix. This initialization approach constrains the frequencies to remain relatively stable in the initial stages, akin to a Fourier series. However, as training progresses, all parameters become freely updatable, allowing the frequencies to adapt and evolve over time.\n\n**On Presentation**\n\nWe noticed that the reviewer mentioned that our paper is “very well written and easy to follow” despite assigning a “fair” presentation score. Please let us know In case there are any outstanding points we can improve on the writing side.  \n"}, {"review_id": "ZhjhRGAPIX", "reviewer": "Reviewer_C7v1", "summary": "The paper proposes a modification of the sine activation function and show that this results in improved performance, compared to RELU and others. The new activation, called METASIN, is a superposition of RELU with several sinusoidal functions, and is motivated by the observation that RELU has a spectral bias to low frequencies. To ensure that training is stable in deep networks, the authors propose a distillation approach, where the activation function parameters, such as amplitudes, are initialized based on a teacher RELU model. The authors show that by combining these two techniques, they outperform RELU and achieve state-of-the-art results on denoising and image resampling tasks.  ", "questions": "- In Line 160, the authors say that the KD Bootstrapping approach comprises only 10% of the total training budget. However, you are also training another model with RELU activations. Wouldn't this increase the training compute by about a factor of 2? Or is the 10% here only for training METASIN shape parameters based on the teacher model? \n\n- How do you select the KD-bootsrapping duration in each experiment? Is it based on a separate validation split? If so, please mention this explicitly in the paper. \n\n- References are missing parentheses throughout the paper. Consider replacing \\cite with \\citep.\n\n\n\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces a new activation function that can be of interest to the community, which contains RELU as a special case. \n\n- The authors provide an optimized implementation of the activation function that is 3 times faster compared to native PyTorch implementation without increasing the memory overhead. In addition, the authors report that the new activation increases overall training compute by only 3% compared to RELU. However, this does not account for the impact of distillation, which seems to be needed.\n\n- The empirical results are strong, achieving SoTA on a few tasks, such as image resampling and denoising. \n\n- The paper is well-written and is easy to follow. \n\n", "weaknesses": "- It seems that all experiments are carried out with distillation. The authors state that they also apply distillation to RELU networks but how does that work when RELU does not contain any shape parameters? If the authors introduce an amplitude to RELU, its effect may vanish when also using normalization layers, which may explain why the authors do not observe any impact of distillation in RELU networks. This needs to be clarified. \n\n- The experimental setup seems to be different in each experiment. For example, the authors use KD-bootsrapping during the first 5% of training in the denoising experiment but switch to 10% in image resampling. They also use a different initialization of the frequencies. Are these chosen based on a separate validation set? It is not clear if this is the case.  \n\n- The authors claim in the abstract that the improvement is obtained by \"simply replacing the activations\". However, the new activations contain trainable parameters that are also initialized/trained using distillation. It is not a simple matter of replacing the activations. \n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Distillation and ReLU Networks**\n\nThe reviewer is correct that our best results in both image resampling and denoising applications are obtained using distillation, specifically KD Bootstrapping as discussed in Section 3.1 of our manuscript. In short: the role of distillation in these experiments is to stabilize training in the early phases and guide the network towards good local minima. Later during the training we turn it off and allow the network to learn the final solution independently without any additional constraints. This is a specific use case for distillation tailored to training MetaSin activations.\n\nIn general though, Knowledge Distillation is used to enhance accuracy of a student network (usually with fewer parameters) by utilizing a pre-trained teacher network (usually with more parameters). The image classification experiment in Section 5.3 is carried out in this general knowledge distillation setting, where the teacher-student pairs and corresponding accuracies are listed in Table 6. Thus, to make the comparison between MetaSin and ReLU student networks fair, we apply distillation to both student variants. This experiment shows that MetaSin student networks are capable of imbibing more information than their ReLU counterparts from identical teacher models.\n\nWe’d be happy to modify the text accordingly to clarify the use of knowledge distillation in the aforementioned experiments.\n\n**Frequency initialization**\n\nThrough grid search across many experiments we performed for this work (on fixed validation sets), we empirically found that initializing the frequency shape parameters of MetaSin activations as $f_j = j$, where $j \\in [1, K]$ ($K$ being the number of *sin* components) works well in most of our experiments involving convolutional models, which is what we recommend in Section 3.1 as a sensible default. That being said, as with any hyperparameter, slight improvements can be made by tweaking the default value: For instance the last two columns of Table 3 show that $f_j = j/2$ yields better results when training the DPCN denoiser described in Section 4.2.  \n\n**Duration of KD Bootstrapping**\n\nIn order to produce our denoising and resampling results we performed KD Bootstrapping for 200K iterations, which correspond to 5 and 10% of the total training time, respectively. Similarly to the frequency initialization, we determined that the 5-10% of total training iterations we recommend as default in Section 3.1 through our observations across various experiments we performed during this work. This duration has been sufficient to stabilize training and guide the network towards good local minima in the beginning, and we observed from that point onwards more KD Bootstrapping steps do not improve the results.\n\n**Total training time with KD Bootstrapping**\n\nThe percentage in Line 160 refers to the ratio of the total training iterations in which we utilize distillation, i.e. incur an additional distillation loss during training. As the computational overhead of the distillation loss is minimal the total training time with or without KD Bootstrapping is roughly the same. That being said, in both our denoising and resampling experiments we had access to the trained baseline ReLU networks. If that is not the case, a ReLU network needs to be trained beforehand, and this should be included in the training budget. We would be happy to modify the text to clarify this point. \n\n**”Simply replacing the activations”**\n\nWe address this more in detail in the Author Rebuttal section as another reviewer raised a similar concern. In short: Although the procedure for replacing the activations (including re-training) is described later in the text in Lines 140-143, we fully agree that our phrasing in Line 12 might be misleading and will modify the text accordingly. \n"}, {"review_id": "gdu0sKeylb", "reviewer": "Reviewer_pU6A", "summary": "The paper proposes MetaSin, a new activation function for deep learning. MetaSin essentially consist of a relu function plus a sum of parametrized sin activations functions. The MetaSin function is developed specifically to work in the domain of image prediction. The paper presents multiple experiments that support MetaSin as the new state-of-the-art activation function, that also investigates different training setups and hyperparameters.", "questions": "In addition to some of the weaknesses pointed out in the previous section, I would like the authors to answer:\n\n1. Table 1: what are the computational scaling behaviour of K? It seems important as it potentially could indicate if there is a performance-computationally trade-off for this hyperparameter.\n\n2. Table 5: It seems like increasing K is beneficial for performance. I wonder how far this behaviour can be pushed. What about increasing it further, like 20 or even higher?\n\nSmaller correction:\nL88: there is missing an end bracket in the equation.", "limitations": "I think the authors are fairly on point in their limitations section that MetaSin.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe work seems original and novel\n\nQuality: \nThe paper is of high quality and provides both a more theoretical reasoning for why MetaSin should be able to perform better than ReLU, but also backs this up with multiple experiments. \n\nClarity: \nThe paper is clearly written and easy to follow. \n\nSignificance:\nAs the authors mention themself in their \"Broader Impact\" section, a potential new SOTA activation function can have a large impact on the community with the potential to increase performance across a large selection of tasks and models. ", "weaknesses": "While I agree with the underlying hypothesis that ReLU networks are most commonly used and therefore it is the most relevant baseline to compare to, I still think the others baselines are relevant, especially for section 5.3 on image classification. The statement in L52-53 is simply wrong, and misrepresent the conclusion in [1]. It is correct that ReLU is sometimes better, but in [1] it is only true for machine translation. For image classification Swish seems to be the better choice, and I am really missing this as a baseline in at least section 5.3.\n\nIn relation to that in L55-56 the authors mention that most other activation functions are untested on image prediction. In that case it seem obvious that the authors could have tested out more baselines than ReLU.\n\nIn general standard deviations are missing for most experiments to show if the results are indeed significant at all. I would be therefore be very careful using the term \"state-of-the-art\".\n\nRegarding section 3.2: more details on the CUDA implementation, specifically regarding the kernel fusion, and what the actual gradient of the MetaSin operator is missing\n\n[1] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Swish as a Baseline for Classification**\n\nAs the reviewer suggested we ran the image classification experiments reported in Table 6 using Swish versions of the baselines. We report the validation accuracies, in comparison to the MetaSin and ReLU results from earlier:\n\n| Teacher  | WRN-40-2  | WRN-40-4 | WRN-28-2 | WRN-40-2 |\n| :--------:                | :-------:       | :-------:       | :-------:      | :-------:       |\n| **Student**  | **WRN-40-1**  | **WRN-40-1** | **WRN-16-2** | **WRN-16-2** |\n| ReLU                  | 73.39          | 70.53          | 71.76         | 73.65         |\n| MetaSin              | **73.74**          | **72.55**         | 72.33        | **74.10**         |\n| Swish\t      | 73.28\t    | **72.54** \t | **72.98**        | 72.60       |\n\nWe’d be happy to update Table 6 with the additional Swish baseline and modify L255 as “... MetaSin student networks consistently outperform their ReLU counterparts, *and in some cases also the Swish student networks*, …”. We believe that the latter part  “... motivating further investigation of using MetaSin activations in classification tasks” is still appropriate and would keep it unchanged.\n\nWe also greatly appreciate the reviewer pointing our attention to the statement in Lines 52-53. In fact, Swish outperforms ReLU in the image classification experiments presented in Ramachandran et al. 2017, whereas performing comparable to ReLU in others. We will replace the text in those lines accordingly with the following:\n“... A comparative study of these activations re-affirmed ReLU as a strong baseline, while suggesting Swish might be a better alternative for image classification.”\n \n\n**Significance of Experiments**\n\nTo investigate this while staying within our compute budget, we ran an experiment on a slight variation of the resampling model from Section 4.1, which uses the same loss, batch and patch sizes for training, and the exact same feature extractor as the original model. The only difference is that the model is trained for 2x upsampling (instead of generic resampling) and the MLP part following the feature extractor is different to accommodate that. As a reminder, we only use MetaSin activations to replace ReLUs in the feature extractor, both in the resampling model we presented in Section 4.1, and in the newly trained upsampling model. The mean and standard deviation computed from 5 independent training runs are:\n\n| Upsampling Model  | PSNR [db]  |\n| :--------:                | :-------:       |\n| ReLU                  | 31.82    $\\pm$ 0.02   |\n| MetaSin (mean/std) | 32.00 $\\pm$ 0.01  |\n\nOn the denoising side, unfortunately due to the resource intensive training process of the denoiser model from 4.2, repeating experiments was not an option. That being said, in Tables 3, 4, and 5 we present a total of 9 different runs that differ in their initializations, number of *sin* components, and use of KD-B. In all these experiments MetaSin models consistently outperform the DPCN baseline presented, often by a large margin. Additionally, in Appendix I we report similar levels of improvement from a kernel prediction version of the same denoiser augmented with MetaSin. We firmly believe that these consistently strong improvements are very unlikely to be due to randomness in the training procedure.\n\n**Details on CUDA kernel**\n\nAs another reviewer also asked for more details, we address this in the Author Rebuttal section\n\n**Computational scaling behavior of K**\n\nWe ran a benchmark experiment, in which we compare forward and backward latencies of ReLU, MetaSin Native, and MetaSin CUDA functions on the same input tensors using different K values. The results can be found in the pdf file attached to the Author Rebuttal section. As the Reviewer suggested, the results show that the computational cost gets higher as we increase K. \n\n\n\n**Effect of setting K to a higher number**\n\nThe reviewer is correct that further improvements in terms of model accuracy can certainly be made by pushing K beyond what we present in the paper. We did not go beyond 10-12 as  throughout our experiments we observed that doing so often resulted in diminishing returns. As a more concrete example we provide results from a resampling experiment below:\n\n| Activation / Upsample Factor | x2  | x3 | x4 |\n| :--------:      | :-------:       | :-------:       | :-------:  | \n| MetaSin-10 | 33.09       | 29.42      | 27.17     | \n| MetaSin-20 | 33.14       | 29.45      | 27.20     | \n\n(Note that this is from an older, but in our opinion still sufficiently representative run with slightly inferior hyperparameters, and the training was stopped prematurely after 900K iterations instead of the full 2M - hence the lower PSNRs compared to Table 2)\n\nThese results suggest that choosing K > 10 would indeed make sense in cases where the absolute best quality is desired and additional latencies are tolerable, but overall K $\\approx$ 10 seems to be the happy medium.\n"}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments. We are encouraged that the reviewers are unanimously leaning towards accepting our submission for publication. In the following we start by briefly summarizing our motivations, then address similar remarks by two reviewers about the phrasing we use to describe the usage of MetaSin activations, and finally provide more details on our optimized CUDA implementation.


**Motivation and broader implications**

Our motivation for this work came from the promising recent results obtained by using MLPs with *sin* activations in visual representation applications including image, video, and 3D shapes. In spite of these findings, however, in the general area of image prediction models, i.e. the large family of models that predict colors of image pixels as their output, ReLU activations still tend to be the default choice. From our own experience, this tendency stems from various issues associated with *sin* networks, such as the training stability and sensitivity to initialization, as well as the lack of methods for utilizing *sin* activations in *convolutional* networks, which enjoy heavy use in the domain of image prediction applications. The aim of the techniques we present in this paper is to pave the way to enable practitioners of this vast domain to reap the benefits of *sin* activations while maintaining the relative training stability of ReLU networks. 

Our results suggest that there is room for significant improvements even in cutting-edge image prediction models, which can be leveraged through a targeted approach such as MetaSin with KD-Bootstrapping. We believe that our findings might be highly useful to practitioners in related application areas including and beyond the ones we touch on in this paper.  


**MetaSin as a “drop-in” ReLU replacement** (`9soJ`, `C7v1`)

This phrase refers to the process of switching to MetaSin in existing code, which consists of replacing `relu` functions with `metasin(K)`. However, after reading their comments and revisiting our own initial text, we agree with the reviewers that it can be misleading as there are additional steps of KD-Bootstrapping and training the model and shape parameters. We will accordingly replace the two total occurrences of term “drop-in replacement” in lines 9 and 136 with “convenient replacement”, and remove the word “simply” in line 12.

**Details on C++/CUDA Implementation**  (`9soJ`, `pU6A`)

To account for the inefficiency of a naive Python API implementation of MetaSin we implemented custom-optimized fused CUDA kernels for both forward and backward functions of the MetaSin activation in C++. Our implementation can be integrated into the PyTorch and TensorFlow Python APIs. Throughout the development process we also tested native automatic compilation functionalities provided by both frameworks (specifically: jit, torch.compile for PyTorch, and jit and XLA in TensorFlow). While notable improvements over the baseline Python API implementation can be made through the use of these out-of-the-box facilities, we obtained best performances in terms of speed and memory consumption using our custom-designed CUDA functions.

Some of the techniques we utilized in our code are as follows: In order to optimize the memory footprint and the inference speed of the MetaSin, we remove the intermediate quantities that the autograd engine computes and instead compute the output and gradient tensors directly from the input and the MetaSin parameters. Moreover, we further optimized the computation speed with improved caching and reduction strategy on warp and block levels. Meanwhile, we have exploited a 2-level reduction strategy based on *pairwise summation* in our backward kernel. This way we could avoid numerical errors in gradient tensors and achieve comparable accuracy to autograd in float32 precision.

The reduction in computational overhead using the aforementioned optimizations enabled running the compute intensive experiments we present throughout the paper in a feasible manner, and we believe demonstrates the viability of MetaSin activations for most practical tasks. That being said, other optimizations that we were not able to explore due to time constraints could help reduce the overhead even further.

**Small Corrections**

Finally, we greatly appreciate that the reviewers took the time to point out various typos, spelling errors, minor latex issues, and so on. We noted them all and will fix them in the final manuscript. We address other remarks individually for each reviewer in the corresponding Official Comment sections.


Meta-review: The reviewers mentioned several positive aspects of the paper, e.g.:
- The paper proposes a novel formulation for an activation function, relying on a sum of sines of varying amplitude, frequency and phase (all of which are parameters learned during training).
- The experiments are promising.
- The paper is very well written and easy to follow.

However, they also raised multiple doubts about it, such as:
- There are some deficiencies in the experiments (e.g., settings, other baselines like MReLU).

All reviewers agreed that the paper could be accepted (6-6-6-6). It seems the paper is a solid submission, and I tend to agree with that.
Decision: Accept (poster)