Title: Empowering Convolutional Neural Nets with MetaSin Activation
Venue: NeurIPS 2023
Accepted: yes
Keywords: sin activation

image prediction

image resampling

monte-carlo denoising

knowledge distillation
Abstract: ReLU networks have remained the default choice for models in the area of image prediction despite their well-established spectral bias towards learning low frequencies faster, and consequently their difficulty of reproducing high frequency visual details. As an alternative, sin networks showed promising results in learning implicit representations of visual data. However training these networks in practically relevant settings proved to be difficult, requiring careful initialization, dealing with issues due to inconsistent gradients, and a degeneracy in local minima. In this work, we instead propose replacing a baseline network’s existing activations with a novel ensemble function with trainable parameters. The proposed MetaSin activation can be trained reliably without requiring intricate initialization schemes, and results in consistently lower test loss compared to alternatives. We demonstrate our method in the areas of Monte-Carlo denoising and image resampling where we set new state-of-the-art through a knowledge distillation based training procedure. We present ablations on hyper-parameter settings, comparisons with alternative activation function formulations, and discuss the use of our method in other domains, such as image classification.

Empowering Convolutional Neural Networks with MetaSin Activation

Abstract

RELU networks have remained the default choice for models in the area of image prediction despite their well-established spectral bias towards learning low frequencies faster, and consequently their difficulty of reproducing high frequency visual details. As an alternative, sin networks showed promising results in learning implicit representations of visual data. However training these networks in practically relevant settings proved to be difficult, requiring careful initialization, dealing with issues due to inconsistent gradients, and a degeneracy in local minima. In this work, we instead propose replacing a baseline network's existing activations with a novel ensemble function with trainable parameters. The proposed METASIN activation can be trained reliably without requiring intricate initialization schemes, and results in consistently lower test loss compared to alternatives. We demonstrate our method in the areas of Monte-Carlo denoising and image resampling where we set new state-of-the-art through a knowledge distillation based training procedure. We present ablations on hyper-parameter settings, comparisons with alternative activation function formulations, and discuss the use of our method in other domains, such as image classification.

1 Introduction

Deep convolutional neural networks are highly proficient in a wide array of tasks including image prediction applications. While there is no scarcity of inventive methods when it comes to designing convolutional blocks (e.g. [25]), another crucial design element, namely the activation function, has seen little change over the years. The RELU activation, despite (and arguably because of) its simplicity has remained the default choice for majority of the models in the area of image prediction.

It is often desirable for models that produce visual results to produce sharp imagery with abundant high frequency details. The widespread use of RELU networks for image prediction applications is somewhat surprising given their spectral bias towards learning low frequencies first [33], and consequently their difficulty in reproducing high frequency visual details. While sin activations have been shown to be superior in simple overfitting experiments on image reconstruction [39], empirical evidence showed that training sin networks in practically relevant settings is challenging even when employing meticulous initialization schemes. While some exploration on using sin activations with convolutional networks exists [8], their use has been mostly limited to fully connected architectures.

In this work we make the following contributions:

We shed light on challenges associated with training networks with sin -based activations through an extensive set of experiments, and address those by formulating the METASIN activation coupled with a stable training procedure.

We show that METASIN networks consistently enable improvements over alternatives when tested on comparable architectures in a variety of image prediction applications, including Monte-Carlo denoising and image resampling where we set new state-of-the-art.

We present a wide range of ablations on hyperparamter settings, comparisons with various alternative activations, and discuss the use of METASIN networks in other domains such as classification and neural signal representations.

2 Related Work

Activation Functions In addition to standard non-polynomial functions such as Sigmoid, tanh , sin , RELU, and exp , numerous alternative activation functions have been explored in the literature. Previous work explored RELU variants such as Leaky RELU [26], Parametric rectified linear unit (PReLU) [14], and Gaussian error linear unit (GELU) [17]. Other authors presented activations with exponential components, such as the Exponential linear unit (ELU) [11], Scaled exponential linear unit (SELU) [22], as well as Sigmoid variants such as Softplus [12], and SWISH [34]. A comparative study of these activations re-affirmed RELU as a strong baseline, while suggesting SWISH might be a better alternative for image classification [34]. Another study [25] reported no accuracy change when switching from RELU to GELU in image classification. The aforementioned body of work has mostly focused on classification experiments, whereas their use in image prediction models was left mostly unexplored.

Previous investigations [34] pointed out periodic activations as a promising direction, which was later explored in the form of sin activations for implicit neural representations of continuous signals [39]. More recent work proposed modulating the amplitudes of sin functions using a secondary fully connected network [27]. Another sin variant is Snake [49], which takes the form x + sin 2 ( x ) and focuses on improving extrapolation of periodic functions. Finally, several papers explored combining multiple primitive functions, such as Mish ( x tanh ( softplus ( x ) ) [30], SinLU ( sin and Sigmoid) [32], and activation ensembles [13]. Preliminary experiments [39] suggest that activations alternative to RELU, with the exception of sin , tend to perform sub-par on image reconstruction. The popularity of RELU networks in image prediction tasks contradicts the theoretical analysis presented in previous work [33] that established the spectral bias of RELU networks towards learning low frequencies faster. Later work [7] re-affirmed this finding through a neural tangent kernel [20] analysis.

Image Resampling Tasks such as correcting lens distortion, retargeting, and upsampling are some image resampling examples that are considered as key operations in real-world computer vision and image processing pipelines. Recent neural methods [5, 40, 9] have been shown to outperform traditional techniques, such as upsampling with a fixed kernel. These neural networks often comprise convolutional blocks [16, 19] with RELU activations. Typically utilizing deep networks, current resampling methods can learn complex representations of input images and produce high-quality resampled outputs. In this work, we employ METASIN to improve upon a recent state-of-the-art resampler [5].

Monte-Carlo Denoising Generating high quality rendered images may require prohibitive amount of compute resources. This can be alleviated by stopping the rendering process prematurely and employing a specialized denoiser to remove the residual noise. Neural denoisers have demonstrated superior performance compared to classical denoising techniques [4, 44, 48]. These denoisers often leverage U-Net with RELU activation [37], which allows them to process noisy images across multiple scales. We likewise demonstrate the use of METASIN in such U-Nets and achieve significant improvements in denoising quality.

3 METASIN Activation

Let us define a generic neural network g ( x ) of depth L that predicts a target image y ∈ R d given input x ∈ R d ′ as:

where T [ l ] denotes a linear transformation that results from either a convolutional or fully connected layer. We would like to find a set of activations ϕ [1] . . . ϕ [ L ] such that they minimize the average reconstruction error ∑ n i L ( g ( x ( i ) ) -y ( i ) ) over a dataset { x ( i ) , y ( i ) ∈ [1 , n ] } .

In hypothetical limit cases where the network g has infinite width or depth, few constraints on activation functions are necessary for the resulting architecture to have the universal approximation capability [18]. However, in practice the selection of the activation functions can play a significant role in the training of neural networks, specifically in the resulting model's prediction accuracy and, in case of image prediction models, the visual characteristics of the predictions. While it is unclear how a principled method can be developed for rank ordering all possible selections of activations, empirical evidence suggests that multi-layer perceptron (MLP) networks with sin activations exhibit desirable characteristics both in expressiveness and their ability to reproduce rich high frequency details [39], especially in comparison to RELU networks that suffer from spectral bias towards reproducing low frequencies [33].

Shallow CNN

Input

Target

RELU

SIREN

METASIN

29.81 dB

32.55 dB

35.04 dB

Deep CNN

Diverged

30.36 dB

12.65 dB

40.01 dB

However, practical experience also revealed the difficulty of training deep convolutional sin networks in real-world scenarios. Figure 1 shows an illustrative example that on one hand demonstrates the clear advantage of sin activations over RELU when training shallow networks, but on the other hand illustrates the fragility of training deeper sin networks that are closer representatives of real-world models. To examine further we express an intermediate segment from a sin activated network as:

where a [ l ] is the output of layer l after applying the corresponding linear transformation and activation function. We can also express the usual parameterization of sine waves as f ( z ; c, f, p ) = c sin ( f z + p ) , where c , f , and p are the amplitude, frequency, and phase of the sine wave that we collectively refer as the shape parameters .

In overparameterized networks the initial parameter values tend to change very slowly [10] (also known as lazy training). Therefore ensuring the selection of plausible initial values for the shape parameters in sin networks is crucial for training. Some exploration has been done in this direction: For instance, SIREN [39] introduces a scaling constant w 0 that is applied to the weight matrix W of the first layer of the underlying fully connected network, i.e. sin ( w 0 W x + b ) , whereas recent work explored introducing an explicit amplitude parameter whose value is predicted through a secondary neural network [27]. Yet initialization remains a challenge as in practice only a very narrow range of values leads to plausible average loss (Refer to Figure 10 in Appendix C).

In this work, we instead introduce explicit shape parameters for directly controlling amplitude, frequency, and phase. These parameters are completely disentangled from other network parameters, and can be initialized intuitively by reasoning about the shape of resulting sine wave. Moreover, in order to provide better initial coverage of plausible ranges of the shape parameters, we construct a composite periodic function by linearly combining K sine waves with individual shape parameters:

The shape parameters are shared among individual channels of layers in convolutional neural networks, and are optimized through backpropagation along with other network parameters 2 . In our experiments we observed that initializing the frequency parameters f [ l ] j of the K constituent sine waves of the composite function to cover a large range of frequencies consistently yields improved results, without necessitating lengthy trial and error procedures that are often not feasible when training large models in practice (Refer to Appendix C for an illustrative example).

Beyond the initialization of shape parameters, the remaining known causes of the difficulty when training sin networks are inconsistent gradients due to the complex shapes that activations can take, and the large degeneracy in local minima caused by symmetries [49]. We empirically found that including an additional RELU component to the composite sine wave function discussed before stabilizes training. With this modification our proposed activation, which we call METASIN, can be expressed as follows:

Equation 4 can efficiently be implemented as a module in common deep learning frameworks, which then can be used to replace activation functions of an existing state-of-the-art network architecture. The trainable parameters θ = { c [ l ] 0 , c [ l ] j , f [ l ] j , p [ l ] j ; j ∈ [1 , K ] , l ∈ [1 , L ] } are optimized through backpropagation without requiring an additional network for predicting these particular parameters, or any other changes to the underlying architecture. This non-intrusive property enables a simple three-step procedure, where (i) we identify a state-of-the-art image prediction model, (ii) replace its existing activation functions with METASIN, and (iii) re-train the resulting METASIN network using a knowledge distillation scheme we discuss in the following section.

3.1 Training

In order to train convolutional METASIN networks, we initialize the shape parameters as c 0 = 1 , c [ l ] j = 0 , f [ l ] j = j , and p [ l ] j = U (0 , π ) , for j ∈ [1 , K ] , and l ∈ [1 , L ] by default. This initialization on one hand forces METASIN to initially assume the shape of RELU, and gradually introduce sin components during the course of the training, while on the other hand covering a broad range of initial frequency and phase values (See Appendix G). Using the default initialization we were able to reliably avoid typical issues encountered while training sin networks.

It is often easier to train a shallower network compared to a deeper one as supported by previous studies [16]. This observation also applies to sin activations, as demonstrated in Figure 1, where a shallow SIREN network can be effectively trained while a deep SIREN network encounters difficulties. To address this, we employ feature knowledge distillation (KD) from a teacher network [35] to provide auxiliary signals during the training of the student METASIN network. We first train a RELU model as the teacher (or, as it often happens in practice, use an already existing baseline RELU model) and use its intermediate feature maps as supervisory signals. This approach allows us to train the shallower blocks of the student METASIN network individually, instead of propagating gradients throughout the entire network. We refer to this phase of training as KD-Bootstrapping , which comprises approximately 5 -10% of the total training iterations for the METASIN network. Following the initial KD-Bootstrapping phase, the student METASIN network is trained using the same configuration as the baseline network. This approach enhances the learning process and helps improve the performance of the student network.

3.2 Efficient Implementation

While equation 4 is straightforward to implement in both PyTorch and Tensorflow using the corresponding Python APIs, such native implementations end up being inefficient compared to executing a RELU function. For performance critical applications, efficiency issues can be alleviated through writing a customized operation in CUDA (Appendix F). In Table 1 we compare the latency induced by the native PyTorch implementation against our customized METASIN operator with fused CUDA kernel functions. Our optimized implementation significantly reduces the overhead of the native implementation. During forward pass, executing a METASIN with K = 10 components requires less than three times the time it takes to execute a RELU function, whereas the backward computation is only slightly more expensive than a RELU activation. Our efficient implementation also uses roughly the same amount of memory during execution as a RELU activation, making METASIN networks feasible to use in practice on the same hardware that a comparable RELU network is designed for.

4 Experiments

We present experiments where we apply the three step procedure discussed at the end of Section 3. Specifically, we switch an existing architecture's activations to METASIN, and re-train the resulting new model from scratch using KD-Bootstrapping.

4.1 Image Resampling

In image resampling our approach builds upon the state-of-the-art network proposed by Bernasconi et al. [5], which incorporates ProSR [45] feature extractor with 3 residual blocks, a resampling layer, and a prediction layer. The network takes an input image and a warp grid and performs image resampling. We trained baseline networks following the exact same training procedures and dataset as described in [5]. The METASIN network is obtained by replacing RELU activations of the feature extractor with METASIN with K = 10 and f [ l ] j = j at initialization. All other parameters are initialized as described in Section 3.1. The transition to METASIN activations increases the amount of compute by 3% (and similarly the wall-clock inference time by 3%) compared to the baseline RELU network. To rule out any improvement due to this increase in computation, we accordingly increase the number of channels of baseline RELU network, which we label as RELU E .

We evaluate the models on three predetermined sets of projective transforms with average local scaling factors of 2 ( P × 2 ), 3 ( P × 3 ) and 4 ( P × 4 ). We report the PSNR scores of the resampled images in Table 2, as well as visual examples in Figure 2 We use KD-Bootsrapping during the first 10% of the training procedure. On average the METASIN network with KD-Bootstrapping improves over the RELU E baseline by 0 . 2 dB, setting a new state-of-the-art in image resampling. We also conducted KD-Bootstrapping on RELU, MISH and SNAKE networks, but observed no improvement.

Reference

RELU

METASIN

Reference

RELU

METASIN

Reference

RELU

METASIN

word

word

This outcome suggests that the training process for these networks is inherently stable, making KD-Bootstrapping unnecessary for these activations. Refer to Appendix K for additional results.

4.2 Denoising Monte-Carlo Rendered Images

Next, we present our experiments in denoising Monte-Carlo rendered images. Our base denoiser network is a U-Net architecture used in [38] and we use the same procedure to generate noisy and reference renderings. Since we focus on direct image prediction models rather than kernel denoising as in [38], our U-Net denoiser (DPCN) directly predicts pixel values of the clean image (For completeness, the results of kernel denoising (KPCN) are presented in Appendix M). During evaluation, we denoise images rendered with [2 , 4 , 8 , 16] samples per pixel and use SMAPE, FLIP [2], 1 -MS_SSIM, and 1 -SSIM as our metrics. Our best METASIN network is configured with K = 5 and the frequencies are initialized as f [ l ] j = j/ 2 . We use KD-bootsrapping during the first 5% of the training procedure. We present a summary of various METASIN network configurations in Table 3, and selected visual examples in Figure 3.

5 Discussion

In this section we present ablation experiments and an exploratory study on using METASIN activations for image classification and implicit image representation using MLPs. We present preliminary investigations of further applications of METASIN activation in Appendix A

Reference (Full)

Baseline

METASIN

Reference (Crop)

5.1 Comparison with Alternative Activations

We ran several experiments to assess the comparative improvements that can be achieved by using alternative activations instead of METASIN. In addition to RELU, we also tested SNAKE ( x +sin 2 ( x ) ) in [49]), MISH ( x tanh(ln(1+ e x )) in [30]), and MRELU in which we linearly combine several RELU activations in the form ∑ K j =1 c j RELU ( x + b j ) (Similar to Adaptive Piecewise Linear Units [3]). The SNAKE activation [49] performs slightly worse than the RELU baseline in the image resampling application, which in turn falls below the corresponding METASIN activated network (Table 2 3 nd col). In case of Monte-Carlo denoising it was not possible to train the U-Net model with SNAKE activations as the training diverged. MISH similarly performs slightly worse than both RELU and SNAKE in image resampling (Table 2 4 th col).

In Monte Carlo denoising the MISH activated network performs better than the RELU baseline by 3% in terms of MS_SSIM, however still falls significantly behind the METASIN activated network that achieves 10% improvement in the same experiment (Table 4).

The MRELU activation with K = 10 performs significantly worse than the RELU baseline both in denoising and resampling task (Table 4 3 rd col and Table 2 7 th col). We also tested applying

KD-Bootstrapping to MRELU which made a big improvement in all metrics, still, however, scoring far behind the corresponding METASIN network.

Overall, METASIN activations ended up consistently yielding notably better results than the activations we tested in all our experiments. In Appendix H we present further comparisons with ensemble activations.

5.2 The Effect of Hyperparameters

The METASIN activation as formulated in Equation 4 has a single hyperparameter K that determines the number of constituent sine waves. While in our experiments we often set K = 10 , we also run ablation experiments to measure the effect of different K values. In these experiments we used the exact same settings as in Section 4.2, except varying the K parameter. In Table 5 we show the effect of changing K to 1 , 5 , and 12 , compared to the baseline where K = 10 . Not surprisingly, lowering K results in consistently worse results according to all four metrics we tested. These findings also show that the results that we presented in Section 4.2 can be further improved simply by setting K = 12 (See Appendix I for further discussion). Aside from K , another decision to be made when using METASIN activations is whether to train with KD-Bootstrapping or not. While in toy examples we haven't noticed any difference between the two alternatives, the last two columns in Table 4 show a comparison with and without KD-Bootsrapping, which clearly demonstrates the benefit of employing KD-Bootsrapping in a real-world scenario.

Table 5: Comparison of DPCN networks (discussed in Section 4.2) with various K parameters trained with KD-Bootstrapping and Glorot frequency initialization.

5.3 Image Classification using Convolutional METASIN Networks

We ran several preliminary experiments for exploring the use of METASIN networks outside our target domain of image prediction tasks. Here we report the results of various experiments in a classification setting, where we trained various student Wide-ResNets using supervision from comparably larger Wide-ResNets that we used as teacher networks. As the teacher network in this case is a larger network with higher accuracy, replicating its predictions becomes a reasonable objective. Therefore, we employ feature knowledge distillation throughout the entire training procedure. The Wide-ResNet architecture [47] follows the same architectural design as ResNet [15], except that the number of channels in each residual block is expanded.

The METASIN activated students are identical to their RELU activated counterparts, except the activations are replaced by METASIN with K = 8 . We ran our experiments on the CIFAR100 [23] dataset, which consists of 50K training and 10K validation images with resolution 32 × 32 . The student networks were trained for 300 epochs using standard cross-entropy loss following the knowledge distillation methodology discussed in [36]. While not setting a new state-of-the-art, Table 6 shows that the METASIN student networks consistently outperform their RELU counterparts, and in some cases also the corresponding SWISH student networks, motivating further investigation of using METASIN activations in classification tasks. However, we note that the accuracy of METASIN models are less competitive compared to alternatives when models are trained from scratch without knowledge distillation (Appendix J).

5.4 METASIN with Multi-Layer Perceptrons

While in this work we focused on applications of METASIN with convolutional neural networks, we also ran several exploratory experiments where we used METASIN activations with MLPs to perform implicit neural representation tasks as in [39]. We refer the reader to Appendix A.1 for a discussion on video overfitting, where the METASIN network achieves 1-4 dB PSNR improvement over the SIREN

baseline with noticeably less visual artifacts. In Appendix A.3 we discuss a voxel grid overfitting experiment where the METASIN network exhibits significantly improved detail reconstruction over the RELU baseline. We also present a preliminary novel view synthesis experiment where we compare a baseline Neural Radiance Field (NeRF) method [29] with a corresponding METASIN model with K = 10 .

We present an example result from this experiment in Figure 4, and refer the reader to Appendix A.2 for more details. Despite not utilizing positional encoding, the METASIN network surpasses vanilla NeRF in performance. It has been observed that Fourier features[42], which bear similarities to positional encoding, aid in learning high-frequency functions. This experiment implies that such feature engineering may not be crucial, as our METASIN network achieves comparable or even superior performance. While this result is intriguing, we acknowledge recent techniques that leverage latent codes in the form of memory, combined with specialized data structures [31, 41], which can significantly enhance vanilla NeRF in terms of performance and training time. While not conclusive, we hope the initial results we share in this section motivate further investigation of METASIN activations in the context of other architectures and application areas.

四

Ground Truth

20

NeRF-PyTorch (

.

78

dB)

METASIN-10

(

21

.

46

dB)

6 Conclusion and Limitations

We presented a novel activation function with trainable parameters that improves upon state-of-the-art convolutional image prediction models. Our work has several limitations. While METASIN networks can reliably be trained without it, training with KD-Bootstrapping helps the network to eventually converge to better minima. While it is not uncommon when training a METASIN network to have access to a comparable trained RELU network, nevertheless this requirement might lead to some extra effort when a pre-trained base model is not available. Additionally, in our current formulation (Equation 4) the number of sin components K is a hyperparameter that we set manually at design time. We performed preliminary experiments with determining K automatically in an adaptive fashion, which would be an interesting direction to study more in the future.

The METASIN function is only C 0 in contrast to the infinitely differentiable SIREN. The C ∞ ) property allows SIREN networks to be trained using loss functions on the gradient or Hessian of the network. This distinct characteristic of the SIREN network is used later in [24] to automatically integrate the function that appears in neural volume rendering. The lack of a gradient loss in METASIN networks seems to limit the performance of METASIN activations for applications such as implicit 3D shape representations (Appendix N). It would be interesting to tackle this inherent limitation of METASIN networks in future work.

Broader Impact

Fundamentally, neural network blocks comprise a linear transformation and a non-linear activation. Our work introduces a new activation function that enables significant prediction accuracy improvements over previous alternatives. Being at such a fundamental level, our proposed contribution thus can be applied to a wide array of neural network types that perform a variety of tasks and facilitate improvements upon existing state-of-the-art results. The resulting improved models may be leveraged for both positive and negative ends.

References

The stanford 3d scanning repository. URL http://graphics.stanford.edu/data/ 3Dscanrep/ .

Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D Fairchild. Flip: A difference evaluator for alternating images. Proc. ACM Comput. Graph. Interact. Tech. , 3(2):15-1, 2020.

Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete. A survey on modern trainable activation functions. Neural Networks , 138:14-32, 2021.

Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. Kernel-predicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph. , 36(4):97-1, 2017.

Michael Bernasconi, Abdelaziz Djelouah, Farnood Salehi, Markus Gross, and Christopher Schroers. Kernel aware resampler. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makoviichuk, S. Tulyakov, and Jian Ren. Real-time neural light field on mobile devices. ArXiv , abs/2212.08057, 2022.

Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning, 2020.

Ka-Hou Chan, Sio-Kei Im, Wei Ke, and Ngan-Lin Lei. Sinp[n]: A fast convergence activation function for convolutional neural networks. In 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion) , pages 365-369, 2018. doi: 10.1109/UCC-Companion.2018.00082.

Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8628-8638, 2021.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems , 32, 2019.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus), 2016.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 315-323. PMLR, 11-13 Apr 2011.

Mark Harmon and Diego Klabjan. Activation ensembles for deep neural networks, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision - ECCV 2016 , pages 630-645, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2020.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks , 2(5):359-366, 1989.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks, 2020.

Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Jyoti Mitra. Relu fields: The little non-linearity that could. ACM SIGGRAPH 2022 Conference Proceedings , 2022.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks, 2017.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14556-14565, 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml , volume 30, page 3. Atlanta, Georgia, USA, 2013.

Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 14214-14223, October 2021.

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. , 38(4), jul 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Diganta Misra. Mish: A self regularized non-monotonic activation function, 2020.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) , 41(4): 1-15, 2022.

Ashis Paul, Rajarshi Bandyopadhyay, Jin Hee Yoon, Zong Woo Geem, and Ram Sarkar. SinLU: Sinu-Sigmoidal linear unit. Sci. China Ser. A Math. , 10(3):337, January 2022.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning , pages 5301-5310. PMLR, 2019.

Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 , 2014.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Farnood Salehi, Marco Manzi, Gerhard Roethlin, Romann Weber, Christopher Schroers, and Marios Papas. Deep adaptive sampling and reconstruction using analytic distributions. ACM Transactions on Graphics (TOG) , 41(6):1-16, 2022.

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems , 33:7462-7473, 2020.

Sanghyun Son and Kyoung Mu Lee. Srwarp: Generalized image super-resolution under arbitrary transformation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7782-7791, 2021.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) , 37(4):1-15, 2018.

Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga SorkineHornung, and Christopher Schroers. A fully progressive approach to single-image superresolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 864-873, 2018.

Lin Yen-Chen. Nerf-pytorch. https://github.com/yenchenlin/nerf-pytorch/ , 2020.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC , 2016.

Xianyao Zhang, Marco Manzi, Thijs Vogels, Henrik Dahlberg, Markus Gross, and Marios Papas. Deep compositional denoising for high-quality monte carlo rendering. In Computer Graphics Forum , volume 40, pages 1-13. Wiley Online Library, 2021.

Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems , 33:1583-1594, 2020.

Appendix

A Use of METASIN Beyond Image Prediction

In addition to our main results presented in Section 4 of the paper, we also performed various exploratory experiments to investigate further application cases of the METASIN activations. The experiments cover image classification where we show favorable results of using convolutional METASIN networks over baseline RELU networks, as well as various overfitting experiments to explore the use of METASIN activations with MLPs. We present our preliminary findings here with the hope of motivating further investigation in these directions.

In all MLP experiments, we use METASIN with K = 10 sine components, and distribute the frequencies evenly across the range [1 , 35] . The initialization of the remaining parameters follows the description provided in Section 3.

Wall

Target

SIREN

METASIN

Car

Bicycle

A.1 Video Overfitting using METASIN MLP

In this section, we present the overfitting experiments on videos using an MLP network with 3 hidden layers and 1024 neurons to overfit a video. The input is a 3 dimensional vector consisting of a frame ID and pixel coordinate, and the output is the RGB value of the corresponding pixel. The METASIN network improves the PNSR by 1 -4 dB compared to the SIREN baseline as shown in Table 7.

Overall the Cat dataset has more fine details, making it more challenging to accurately represent these intricate details. Consequently, the PSNR for the Cat dataset is comparatively lower in comparison to the Bikes dataset. The results are depicted in Figure 5. While RELU networks struggle to represent the signal, both SIREN network and METASIN can reconstruct the frames. Notably, the METASIN network reconstructs more details, see Figure 5 (Bicycle) and has less noise, as we see in Figure 5 (Wall and Car). It is worth mentioning that various techniques, such as positional encoding or Fourier

Ground Truth

NeRF

METASIN-10

A.2 Novel View Synthesis using METASIN MLP

Neural Radiance Fields [29] (NeRF) enable synthesizing novel views with only a sparse set of input views. NeRF methods exploit positional encoding to map the input into high-frequency space, which is the key to high-quality scene rendering, which we drop in our experiments and replace the standard RELU with a 10 components METASIN10 activation. Frequencies and phases are initialized uniformly over [1 , 35] and [0 , 2 π ] . The experiment setting is the same as [29], except that we extend the training process to 250 K iterations instead of 200 K, in order to make sure that METASIN models converge well. Our implementation is based on [46], The results are reported in Table 8. We also provide a comparison of qualitative results in Figure 6.

GT

RELU (MSE=

0

.

252

)

METASIN-10 (MSE=

0

.

044

)

A.3 Density Overfitting using METASIN MLP

In this section we present an overfitting experiment on a 3 D voxel grid. The smoke scene we used in this experiment contains 175 × 232 × 175 density values within range [0 , 1] . Following the MLP architecture used in [39], we map the 3 D coordinates to the corresponding density value using RELU and METASIN activated networks. Comparisons between the METASIN activated network and the baseline RELU model are presented in Figure 7.

B Details of Overfitting Experiment with Convolutional Network

Table 9 shows further architectural details of the overfitting experiment we discuss in Section 3 of the main paper. We perform experiments with two base convolutional architectures, a shallow and a deep network. M-SIREN refers to the amplitude modulated SIREN [27], which has significantly more parameters due to the introduction of an additional amplitude modulation network.

Shallow CNN

Input

Target

RELU

SIREN

M-SIREN

METASIN

29.81 dB

32.55 dB

30.88 dB

35.04 dB

Deep CNN

30.36 dB

12.65 dB

35.70 dB

40.01 dB

Figure 8 shows a visual comparison of reconstructed images by the different versions of both base networks. Both the shallow and deep versions of the METASIN network can be trained reliably (unlike their sin activated counterparts that diverge in the case of the deep architecture). The METASIN networks come with relatively little parameter overhead (the relative overhead becomes even smaller with larger networks that have more channels, as discussed in Appendix E), and produce visually sharper results that also are closer to the reference in terms of PSNR.

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.5

0.0

0.5

1.0

L1 Loss

0.252

ReLU

Target

Pred

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.5

0.0

0.5

1.0

L1 Loss

0.319

f

=1

Target

Pred

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.5

0.0

0.5

1.0

L1 Loss

0.090

f

=5

Target

Pred

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.5

0.0

0.5

1.0

L1 Loss

0.021

f

=10

Target

Pred

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.5

0.0

0.5

1.0

L1 Loss

0.013

MetaSin

K

=10

Target

Pred

C Sensitivity to Frequency Initialization

Although the sin activation function has promising properties as outlined in the main paper, it is known that they can be extremely difficult to train [39, 49].

We illustrate this in a toy experiment where we try to reconstruct the signal y ( t ) = sin(10 t + 5 sin(10 t )) × sin(10 sin(10 t )) when the input is x ( t ) = t . Most of the energy of y ( t ) is distributed between frequencies between 10 and 170 . We sample t from the normal distribution and train fully connected networks with 1 input neuron, 3 layers having 20 neurons each, and 1 output neuron. In one of the networks, we use RELU activation in between fully connected layers except for the last

one. In the remaining two networks, we use sin( fx ) . In one we initialize f = 1 , in the other one we initialize f = 5 , and f = 10 . We use L 1 loss and Adam as the optimizer. The results are shown in Figure 9. As it is seen, RELU-network can reconstruct part of the signal. The sin -network with the initialization f = 1 does not learn at all, while the network initialized with f = 5 is able to match y ( t ) , and with f = 10 the model fits the original signal reasonably well. This suggests the importance of initialization and the difficulty of training of the sin activation function.

On the other hand, a METASIN network with the frequency parameters from Equation 4 initialized as f [ l ] j = j and K = 10 effectively covers a wide range of frequencies including the one that is important to represent the signal. Consequently, using a METASIN we are able to lower the loss to 0 . 013 and eliminate the trial and error process for initializing frequencies.

In Figure 10 we additionally plot the results of an experiment where we sweep through a wide range of w 0 values for a fully-connected SIREN network that was trained to overfit to a single image. The plot illustrates that only a narrow range among possible initial w 0 values results in a plausible training error, confirming the importance of initialization in SIREN networks.

10

1

10 0

10 1

10 2

10 3

10 4

SIREN

w

0

10

0

10

20

30

40

50

PSNR (dB)

10

3

10

2

10

1

10 0

10 1

Loss

PSNR

Loss

D Connection to Fourier Series

Well-behaved functions, such as continuous functions can be expressed as a Fourier series with finitely many terms. Let us define a function ϕ as a Fourier series:

where K ∈ N + is finite, and P denotes length of one period. Using the harmonic addition rule, i.e.

we can rewrite Equation 5 as:

Substituting in the identity cos α = sin ( α + π 2 ) we get:

Dropping the constant term we obtain the following:

From Equation 9 we can see that any Fourier Series can be expressed as a METASIN activation

by setting c 0 = 0 and the rest of its parameters { c j , f j , p j ; j ∈ [1 , K ] } accordingly.

E METASIN Overhead

METASIN parameter overhead formula for a convolutional layer compared to RELU baseline is as follows:

For example if we have a layer with kernel-size = 3 with 128 input-channels, then overhead for K = 10 is 0 . 026 . Which means we have approximately 2 . 6% more parameters. If input-channels is 32 the overhead becomes 10 . 08% . On the other hand for kernel-size = 5 and 512 input-channels the overhead is 0 . 24% .

F Details on C++/CUDA Implementation

To account for the inefficiency of a native Python API implementation of METASIN we implemented custom-optimized fused CUDA kernels for both forward and backward functions of the MetaSin activation in C++. Our implementation can be integrated into the PyTorch and TensorFlow Python APIs. Throughout the development process we also tested native automatic compilation functionalities provided by both frameworks (specifically: jit, torch.compile for PyTorch, and jit and XLA in TensorFlow). While notable improvements over the baseline Python API implementation can be made through the use of these out-of-the-box facilities, we obtained the best performances in terms of speed and memory consumption using our custom-designed CUDA functions.

Some of the techniques we utilized in our code are as follows: in order to optimize the memory footprint and the inference speed of the METASIN, we remove the intermediate quantities that the autograd engine computes and instead compute the output and gradient tensors directly from the input and the METASIN parameters. Moreover, we further optimized the computation speed with improved caching and reduction strategy on warp and block levels. Meanwhile, we have exploited a 2 -level reduction strategy based on pairwise summation in our backward kernel. In this way, we could avoid numerical errors in gradient tensors and achieve comparable accuracy to autograd in float32 precision.

The reduction in computational overhead using the aforementioned optimizations enabled running the compute-intensive experiments we present throughout the paper in a feasible manner, and we believe demonstrates the viability of METASIN activations for most practical tasks. That being said, other optimizations that we were not able to explore due to time constraints could help reduce the overhead even further, which is an interesting direction for future research.

Forward Timing[ms]

100

relu

native

cuda

[sw]

Time

10-1

2°

21

22

23

24

25

26

MetaSin Components

Backward Timing[ms]

relu

native

cuda

[sw]

100

Time

10-1

20

21

22

23

24

25

26

MetaSinComponents

We also investigated the computational scaling behavior of our implementation with respect to the number of sin components K . To this end, we ran a benchmark experiment, in which we compared forward and backward latencies of RELU, METASIN Native, and METASIN CUDA functions on the same input tensors using different K values. The results can be found in Figure 11.

2.5

0.0

2.5

0

2

iter: 0

2.5

0.0

2.5

0

2

iter: 0

2.5

0.0

2.5

0

2

iter: 0

2.5

0.0

2.5

0

2

iter: 33K

2.5

0.0

2.5

0

2

iter: 33K

2.5

0.0

2.5

0

2

iter: 33K

2.5

0.0

2.5

0

2

iter: 66K

2.5

0.0

2.5

0

2

iter: 66K

2.5

0.0

2.5

0

2

iter: 66K

2.5

0.0

2.5

0

2

iter: 100K

2.5

0.0

2.5

0

2

iter: 100K

2.5

0.0

2.5

0

2

iter: 100K

f

j

abs weight

iter: 100K

f

j

abs weight

iter: 100K

f

j

abs weight

iter: 100K

G Evolution of METASIN Activations during Training

Figure 12 shows illustrative examples of the evolution of METASIN activations during the course of training in a toy example. In this experiment the weight of the RELU component is set to 1 , whereas the amplitudes of all sin components are set to 0 . Thus a METASIN activation starts out as RELU, but then evolves into various shapes depending on the training of the shape parameters via backpropagation. It is worth noting that the shape of a METASIN activation tends to not change after the initial phase of training, which agrees with the observation that weights change very slowly in overparameterized networks. Moreover, the different rows in the figure show three different METASIN neurons of the same layer that interestingly converge to a complementary frequency response that can capture different frequency components of the input signal.

To shed some light on the shapes that METASIN activations take in a more practically relevant setting with and without KD-Bootstrapping, we additionally produced a visualization of the METASIN shapes from the resampling network described in Section 4.1. The METASIN shapes are obtained from the

first and last blocks of models trained from scratch and with KD Bootstrapping. This visualization in Figure 13 shows that, while there is significant local variation between individual METASIN activations, globally the rough RELU shape is still discernible even without any involvement from the RELU teacher.

With KD Bootstrapping

Trained from Scratch (w/o KD-B)

First Block

Last Block

H Comparison with Ensemble Activations

We ran additional experiments using ensemble activations as described in a recent survey [3], namely Adaptive Blending Units (abu), Variable AFs (vaf), Adaptive AFs (aaf) in the same experimental setting that we used to generate the first row of Figure 1 in the main paper. We present the best results we obtained from multiple runs with different initializations in Table 10, also including RELU and METASIN as references:

We also ran two experiments in image resampling setting using abu and aaf versions of the model from Section 4.1, which we present in Table 11:

201

Reference

RELU

METASIN

I Effect of Setting K to a high number

We note that further improvements in terms of model accuracy can be made by pushing K beyond what we present in the paper. We did not go beyond 10 -12 as throughout our experiments we observed that doing so often resulted in diminishing returns. As a more concrete example we provide results from a resampling experiment in Table 12 (Note that these results are from a preliminary run with slightly inferior hyperparameters compared to the corresponding experiments we report in the main paper, and the training was stopped prematurely after 900 K iterations instead of the full 2 M- hence the lower PSNRs compared to Table 2). These results suggest that choosing K > 10 would indeed make sense in cases where the absolute best quality is desired and additional latencies are tolerable, but overall K ≈ 10 seems to be the happy medium.

J Classification Experiment with METASIN Model Trained from Scratch

To investigate the effect of KD-Bootstrapping in the classification setting, we trained various WideResNets with original RELU activations and with METASIN activations entirely from scratch and without using KD-Bootstrapping. Table 13 presents test accuracy on CIFAR-100.

Table 13: Comparison of METASIN and RELU models in image classification with training without KD-Bootstrapping. In this case METASIN models only become advantageous when train using KD-Bootstrapping, underlining the importance of the proposed training methodology.

In accordance with the above discussion these results underline the role of KD Bootstrapping for achieving the best results with METASIN networks. To give a concrete example: as we show above, when training from scratch WRN-16-2-METASIN at 71 . 82 accuracy is behind its RELU counterpart (WRN-16-2-ReLU) with accuracy 72 . 85 . On the other hand, the last column of Table 6 shows that by distilling from a WRN-40-2 teacher, the accuracy of WRN-16-2-METASIN can be brought up to 74 . 10 , whereas WRN-16-2-RELU achieves 73 . 65 accuracy using the same procedure.

K Resampler Architecture and Additional Results

Our network architecture is based on the design proposed in [5]. We utilize a ProSR feature extractor [45], consisting of 3 residual blocks with 4 dense blocks within each residual block and a growth rate of 40 and 160 channels. The subsequent prediction layer is implemented as an MLP with 4 hidden layers and 256 neurons. In our experiments, this prediction layer remains unchanged. We apply METASIN activations only to the convolutional feature extractor. We present additional results in Figure 14 for the resampler experiment presented in Section 4.1.

L DPCN Denoiser Architecture and Additional Results

The design of the U-Net architecture we use in Section 4.2 is detailed in Table 14. We also present

additional results in Figure 15 to the results we presented in Section 4.2 in the main paper.

DPCN

METASIN-5 w/ KD-B

M KPAL Denoiser Results

In order to factor out the potential effects of kernel prediction to our denoising experiments, we utilized a direct prediction version of the state-of-the-art Monte-Carlo denoiser presented in [38]. In this section we present results of training a METASIN version of the original kernel prediction network. The improvement over the baseline RELU network remains significant in this experiment across all metrics. An interesting observation from this study was the significantly reduced effect of KD-Bootstrapping, which we eventually decided not to employ to produce the results in Table 15. We hypothesize that the lesser effect of KD-Bootstrapping in this case can be attributed to kernel prediction, which however requires further investigation.

N Failure Cases

While our preliminary experiments on using METASIN activations with MLPs generally resulted in improvements over corresponding baselines, we also observed some failure cases. A particular

experiment where the tested METASIN network did not perform well was on signed distance field (SDF) overfitting. In this case, the SIREN reproduction of the ground-truth SDF is better at fine scale details than the corresponding METASIN reproduction. We hypothesize that this discrepancy in performance is due to the gradient loss that SIREN employs and the METASIN network lacks due to not being continuously differentiable. We defer further investigation of this limitation to future work.

(a) GT

(b) SIREN

(c)

f

j

=

U

(1

,

35)

UWU

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes MetaSin, a new activation function for deep learning. MetaSin essentially consist of a relu function plus a sum of parametrized sin activations functions. The MetaSin function is developed specifically to work in the domain of image prediction. The paper presents multiple experiments that support MetaSin as the new state-of-the-art activation function, that also investigates different training setups and hyperparameters.\n\nSTRENGTHS:\nOriginality:\nThe work seems original and novel\n\nQuality: \nThe paper is of high quality and provides both a more theoretical reasoning for why MetaSin should be able to perform better than ReLU, but also backs this up with multiple experiments. \n\nClarity: \nThe paper is clearly written and easy to follow. \n\nSignificance:\nAs the authors mention themself in their \"Broader Impact\" section, a potential new SOTA activation function can have a large impact on the community with the potential to increase performance across a large selection of tasks and models.\n\nWEAKNESSES:\nWhile I agree with the underlying hypothesis that ReLU networks are most commonly used and therefore it is the most relevant baseline to compare to, I still think the others baselines are relevant, especially for section 5.3 on image classification. The statement in L52-53 is simply wrong, and misrepresent the conclusion in [1]. It is correct that ReLU is sometimes better, but in [1] it is only true for machine translation. For image classification Swish seems to be the better choice, and I am really missing this as a baseline in at least section 5.3.\n\nIn relation to that in L55-56 the authors mention that most other activation functions are untested on image prediction. In that case it seem obvious that the authors could have tested out more baselines than ReLU.\n\nIn general standard deviations are missing for most experiments to show if the results are indeed significant at all. I would be therefore be very careful using the term \"state-of-the-art\".\n\nRegarding section 3.2: more details on the CUDA implementation, specifically regarding the kernel fusion, and what the actual gradient of the MetaSin operator is missing\n\n[1] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.\n\nQUESTIONS:\nIn addition to some of the weaknesses pointed out in the previous section, I would like the authors to answer:\n\n1. Table 1: what are the computational scaling behaviour of K? It seems important as it potentially could indicate if there is a performance-computationally trade-off for this hyperparameter.\n\n2. Table 5: It seems like increasing K is beneficial for performance. I wonder how far this behaviour can be pushed. What about increasing it further, like 20 or even higher?\n\nSmaller correction:\nL88: there is missing an end bracket in the equation.\n\nLIMITATIONS:\nI think the authors are fairly on point in their limitations section that MetaSin.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a modification of the sine activation function and show that this results in improved performance, compared to RELU and others. The new activation, called METASIN, is a superposition of RELU with several sinusoidal functions, and is motivated by the observation that RELU has a spectral bias to low frequencies. To ensure that training is stable in deep networks, the authors propose a distillation approach, where the activation function parameters, such as amplitudes, are initialized based on a teacher RELU model. The authors show that by combining these two techniques, they outperform RELU and achieve state-of-the-art results on denoising and image resampling tasks.\n\nSTRENGTHS:\n- The paper introduces a new activation function that can be of interest to the community, which contains RELU as a special case. \n\n- The authors provide an optimized implementation of the activation function that is 3 times faster compared to native PyTorch implementation without increasing the memory overhead. In addition, the authors report that the new activation increases overall training compute by only 3% compared to RELU. However, this does not account for the impact of distillation, which seems to be needed.\n\n- The empirical results are strong, achieving SoTA on a few tasks, such as image resampling and denoising. \n\n- The paper is well-written and is easy to follow.\n\nWEAKNESSES:\n- It seems that all experiments are carried out with distillation. The authors state that they also apply distillation to RELU networks but how does that work when RELU does not contain any shape parameters? If the authors introduce an amplitude to RELU, its effect may vanish when also using normalization layers, which may explain why the authors do not observe any impact of distillation in RELU networks. This needs to be clarified. \n\n- The experimental setup seems to be different in each experiment. For example, the authors use KD-bootsrapping during the first 5% of training in the denoising experiment but switch to 10% in image resampling. They also use a different initialization of the frequencies. Are these chosen based on a separate validation set? It is not clear if this is the case.  \n\n- The authors claim in the abstract that the improvement is obtained by \"simply replacing the activations\". However, the new activations contain trainable parameters that are also initialized/trained using distillation. It is not a simple matter of replacing the activations.\n\nQUESTIONS:\n- In Line 160, the authors say that the KD Bootstrapping approach comprises only 10% of the total training budget. However, you are also training another model with RELU activations. Wouldn't this increase the training compute by about a factor of 2? Or is the 10% here only for training METASIN shape parameters based on the teacher model? \n\n- How do you select the KD-bootsrapping duration in each experiment? Is it based on a separate validation split? If so, please mention this explicitly in the paper. \n\n- References are missing parentheses throughout the paper. Consider replacing \\cite with \\citep.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**SUMMARY AFTER REBUTTAL**: the authors have addressed most of my concerns and I have increased my score during the rebuttal phase. I believe the novelty of the paper is small if moving beyond their specialized subfield, which is why the overall score remains low.\n\n---\n\nThe paper proposes a variant of the sin activation function proposed in SIREN, specifically for image reconstruction and image denoising. \n\nInstead of using a single sin function, they consider a linear combination of a ReLU function and multiple sin functions, where the weights of the linear combination and the parameters of the sin functions are shared across layers and trained with back-propagation. The networks are trained via a simple knowledge distillation procedure after a drop-in replacement of the activation functions of the corresponding ReLU networks. \n\nA series of experiments shows better image reconstruction capabilities with a very small overhead (thanks to a custom CUDA kernel) on two major benchmarks.\n\nSTRENGTHS:\n- The paper is very well written and easy to follow.\n- Experiments are good (see some remarks below), and the improvements are consistent.\n- The motivation for the proposed AFs is only discursive, but clear.\n- As far as I know, this formulation is novel with some caveats (see below).\n\nWEAKNESSES:\n- My main concern is that the idea of building an AF from a linear combination of base AFs whose weights are trained is quite known in the literature. Considering for example the survey on AFs by Apicella et al., 2021 [A survey on modern trainable activation functions], they have an entire chapter on this idea including Adaptive AFs, Variable AFs, Kernel-based AFs, Adaptive Blending Unit, Adaptive Piecewise Linear Unit, etc.\n- The core contribution of this paper is to apply this idea to sin AFs, i.e., considering a different base set from the papers cited above (which combined other types of functions, such as kernels with different weights, ReLUs and sigmoids, etc.), showing this is useful in the context of image reconstruction. While interesting, this is a niche result that is best suited for a smaller conference or journal with applicative interest.\n- In addition, the authors are only comparing to standard AFs, although they consider a \"MReLU\" AF which seems similar to the ensemble AFs described above. It would be helpful to have a stronger comparison to ensemble AFs.\n\nNote that a similar paper vertical on PINNs was rejected from last ICLR (https://openreview.net/forum?id=MpGP-z07TmM) due to similar reasons.\n\nAn additional weakness is that the authors claim their method is a complete drop-in replacement, while it requires a KD step to work correctly. I believe these claims should be amended.\n\nQUESTIONS:\nApart from the previous weaknesses, I would be curious to see more details on their CUDA kernel. Is this done manually, or is it just the result of PyTorch compile procedures?\n\nI am also skeptical of their initialization range, since they set to 0 all weights multiplying the sin AFs. Wouldn't this prevent a good gradient flow? More plots of the resulting sin functions before and after training can improve the discussion here.\n\nWhile the paper is well written, there are some citations that appear in a wrong way (e.g., \"state-of-the-art resampler Bernasconi et al. [4]\"). There is also a small typo on P3 (\"it’s predictions\").\n\nLIMITATIONS:\nAll limitations are correctly described in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper propose a new activation function, which relies on a parametrized sinusoidal function instead of only a piecewise linear function. The authors show how this function can lead to performance improvements in the setting of denoising.\n\nSTRENGTHS:\n- The authors propose a novel formulation for an activation function, relying on a sum of sines of varying amplitude, frequency and phase (all of which are parameters learned during training). The formulation of this activation function is novel, as far as I am aware.\n\n- I appreciate the explanation that the authors provide on the usefulness of the sine activation, namely how it enables the network to capture higher frequency components. Moreover, I appreciate the inclusion of the link to Fourier series in the Appendix, and I think it may be an interesting topic that the authors may want to elaborate on.\n\n- The authors have performed several experiments to evaluate the performance of their method, not only in their main problems of interest (denoising and resampling) but also more usual classification tasks. They have also made the effort to write optimized code for their proposed activation function (although I highly encourage releasing this to assure reproducibility).\n\nWEAKNESSES:\nThe main weaknesses of the paper are the following (mostly concerning the experimental setup):\n\n- While the experiments cover a wide range of tasks, they contain a weakness in that most of them rely on starting from a pretrained network to achieve the best results. While the authors acknowledge this limitation, it would nevertheless improve the paper if results for training without the initial model were included.\n\n- Related to the above, the classification experiment only considers the setting where the MetaSin network learns from a pretrained teacher. I believe it would be interesting to include a classification experiment where the new architecture is trained from scratch.\n\n- In the Appendix, the authors include a plot that shows how the weights of the activation change during the course of the training. We can see that most of the weight is in the ReLU activation (in other words, the initial form of the MetaSin activation). While this is not inherently bad, it makes the benefit induced by the new activation less clear in my opinion. I believe that this may also stem from the initialization via a pretrained model (which has already converged while using a ReLU activation), and thus may be alleviated if trained from scratch. I would greatly appreciate it if the authors were able to examine this.\n\n**Post rebuttal comment**: As mentioned in my response to the author rebuttal below, while most of my concerns have been addressed, the fact that KD bootstrapping is required remains a limitation. However, I have a generally positive view of this work, given that as the authors mentioned in their comments the use of KD bootstrapping for stability, while limiting, is still an interesting approach.\n\nQUESTIONS:\nI would be grateful if the authors could elaborate on the issues encountered during training from scratch with their proposed activation. The way the paper is written, it seems to me that training with MetaSin would suffer from the same problems as previous work using sinusoidal activations e.g. SIREN, which is somewhat surprising given that MetaSin contains ReLU as a special case.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work. Furthermore, I see no immediate negative societal impact arising from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}