SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Empowering Convolutional Neural Networks with MetaSin Activation
Farnood Salehi
Tunç Ozan Aydin $^{1,3}$
Andre Gaillard3
Guglielmo Campores<sup>1,2*</sup>
Yuxuan Wang<sup>3</sup>
<sup>1</sup>Disney Research | Studios
$^{2}$ University of Padova
$^{3}$ ETH Zürich
RELU networks have remained the default choice for models in the area of image prediction despite their well-established spectral bias towards learning low frequencies faster, and consequently their difficulty of reproducing high frequency visual details. As an alternative, sin networks showed promising results in learning implicit representations of visual data. However training these networks in practically relevant settings proved to be difficult, requiring careful initialization, dealing with issues due to inconsistent gradients, and a degeneracy in local minima. In this work, we instead propose replacing a baseline network's existing activations with a novel ensemble function with trainable parameters. The proposed METASIN activation can be trained reliably without requiring intricate initialization schemes, and results in consistently lower test loss compared to alternatives. We demonstrate our method in the areas of Monte-Carlo denoising and image resampling where we set new state-of-the-art through a knowledge distillation based training procedure. We present ablations on hyper-parameter settings, comparisons with alternative activation function formulations, and discuss the use of our method in other domains, such as image classification.
Deep convolutional neural networks are highly proficient in a wide array of tasks including image prediction applications. While there is no scarcity of inventive methods when it comes to designing convolutional blocks (e.g. [25]), another crucial design element, namely the activation function, has seen little change over the years. The RELU activation, despite (and arguably because of) its simplicity has remained the default choice for majority of the models in the area of image prediction.
It is often desirable for models that produce visual results to produce sharp imagery with abundant high frequency details. The widespread use of RELU networks for image prediction applications is somewhat surprising given their spectral bias towards learning low frequencies first [33], and consequently their difficulty in reproducing high frequency visual details. While sin activations have been shown to be superior in simple overfitting experiments on image reconstruction [39], empirical evidence showed that training sin networks in practically relevant settings is challenging even when employing meticulous initialization schemes. While some exploration on using sin activations with convolutional networks exists [8], their use has been mostly limited to fully connected architectures.
In this work we make the following contributions:
*This work was done while the author was an intern at Disney Research | Studios
- We shed light on challenges associated with training networks with sin-based activations through an extensive set of experiments, and address those by formulating the METASIN activation coupled with a stable training procedure.
- We show that METASIN networks consistently enable improvements over alternatives when tested on comparable architectures in a variety of image prediction applications, including Monte-Carlo denoising and image resampling where we set new state-of-the-art.
- We present a wide range of ablations on hyperparameter settings, comparisons with various alternative activations, and discuss the use of METASIN networks in other domains such as classification and neural signal representations.
Activation Functions In addition to standard non-polynomial functions such as Sigmoid, tanh, sin, RELU, and exp, numerous alternative activation functions have been explored in the literature. Previous work explored RELU variants such as Leaky RELU [26], Parametric rectified linear unit (PReLU) [14], and Gaussian error linear unit (GELU) [17]. Other authors presented activations with exponential components, such as the Exponential linear unit (ELU) [11], Scaled exponential linear unit (SELU) [22], as well as Sigmoid variants such as Softplus [12], and SWISH [34]. A comparative study of these activations re-affirmed RELU as a strong baseline, while suggesting SWISH might be a better alternative for image classification [34]. Another study [25] reported no accuracy change when switching from RELU to GELU in image classification. The aforementioned body of work has mostly focused on classification experiments, whereas their use in image prediction models was left mostly unexplored.
Previous investigations [34] pointed out periodic activations as a promising direction, which was later explored in the form of sin activations for implicit neural representations of continuous signals [39]. More recent work proposed modulating the amplitudes of sin functions using a secondary fully connected network [27]. Another sin variant is Snake [49], which takes the form  $x + \sin^2 (x)$  and focuses on improving extrapolation of periodic functions. Finally, several papers explored combining multiple primitive functions, such as Mish ( $x \tanh \left(\text{softplus}(x)\right)$ ) [30], SinLU ( $\sin$  and Sigmoid) [32], and activation ensembles [13]. Preliminary experiments [39] suggest that activations alternative to RELU, with the exception of sin, tend to perform sub-par on image reconstruction. The popularity of RELU networks in image prediction tasks contradicts the theoretical analysis presented in previous work [33] that established the spectral bias of RELU networks towards learning low frequencies faster. Later work [7] re-affirmed this finding through a neural tangent kernel [20] analysis.
Image Resampling Tasks such as correcting lens distortion, retargeting, and upsampling are some image resampling examples that are considered as key operations in real-world computer vision and image processing pipelines. Recent neural methods [5, 40, 9] have been shown to outperform traditional techniques, such as upsampling with a fixed kernel. These neural networks often comprise convolutional blocks [16, 19] with RELU activations. Typically utilizing deep networks, current resampling methods can learn complex representations of input images and produce high-quality resampled outputs. In this work, we employ METASIN to improve upon a recent state-of-the-art resampler [5].
Monte-Carlo Denoising Generating high quality rendered images may require prohibitive amount of compute resources. This can be alleviated by stopping the rendering process prematurely and employing a specialized denoiser to remove the residual noise. Neural denoisers have demonstrated superior performance compared to classical denoising techniques [4, 44, 48]. These denoisers often leverage U-Net with RELU activation [37], which allows them to process noisy images across multiple scales. We likewise demonstrate the use of METASIN in such U-Nets and achieve significant improvements in denoising quality.
3 METASIN Activation
Let us define a generic neural network  $g(\mathbf{x})$  of depth  $L$  that predicts a target image  $\mathbf{y} \in \mathbb{R}^d$  given input  $\mathbf{x} \in \mathbb{R}^{d'}$  as:
$$
\mathbf {y} \approx g (\mathbf {x}) = \left(\phi^ {[ L ]} \circ T ^ {[ L ]} \circ \phi^ {[ L - 1 ]} \circ T ^ {[ L - 1 ]} \circ \dots \circ \phi^ {[ 1 ]} \circ T ^ {[ 1 ]}\right) (\mathbf {x}), \tag {1}
$$
where  $T^{[l]}$  denotes a linear transformation that results from either a convolutional or fully connected layer. We would like to find a set of activations  $\phi^{[1]}\ldots \phi^{[L]}$  such that they minimize the average reconstruction error  $\sum_{i}^{n}\mathcal{L}\left(g\left(\mathbf{x}^{(i)}\right) - \mathbf{y}^{(i)}\right)$  over a dataset  $\{\mathbf{x}^{(i)},\mathbf{y}^{(i)}\in [1,n]\}$ .
In hypothetical limit cases where the network  $g$  has infinite width or depth, few constraints on activation functions are necessary for the resulting architecture to have the universal approximation capability [18]. However, in practice the selection of the activation functions can play a significant role in the training of neural networks, specifically in the resulting model's prediction accuracy and, in case of image prediction models, the visual characteristics of the predictions. While it is unclear how a principled method can be developed for rank ordering all possible selections of activations, empirical evidence suggests that multi-layer perceptron (MLP) networks with sin activations exhibit desirable characteristics both in expressiveness and their ability to reproduce rich high frequency details [39], especially in comparison to RELU networks that suffer from spectral bias towards reproducing low frequencies [33].
Figure 1: Peak Signal-to-Noise Ratio (PSNR) comparison of different activations employed on CNNs to overfit an image from its downsampled version (with downsampling factor  $16 \times$ ). Notably, while the shallow sin network performs better than its RELU counterpart, the deep sin network fails to converge, demonstrating the fragility of training deep convolutional networks with sin activations. See Appendix B for details.
However, practical experience also revealed the difficulty of training deep convolutional sin networks in real-world scenarios. Figure 1 shows an illustrative example that on one hand demonstrates the clear advantage of sin activations over RELU when training shallow networks, but on the other hand illustrates the fragility of training deeper sin networks that are closer representatives of real-world models. To examine further we express an intermediate segment from a sin activated network as:
$$
\mathbf {a} ^ {[ l ]} = \sin \left(T ^ {[ l ]} \left(\mathbf {a} ^ {[ l - 1 ]}\right)\right), \tag {2}
$$
where  $\mathbf{a}^{[l]}$  is the output of layer  $l$  after applying the corresponding linear transformation and activation function. We can also express the usual parameterization of sine waves as  $f(\mathbf{z};c,f,p) = c\sin (f\mathbf{z} + p)$ , where  $c$ ,  $f$ , and  $p$  are the amplitude, frequency, and phase of the sine wave that we collectively refer as the shape parameters.
In overparameterized networks the initial parameter values tend to change very slowly [10] (also known as lazy training). Therefore ensuring the selection of plausible initial values for the shape parameters in sin networks is crucial for training. Some exploration has been done in this direction: For instance, SIREN [39] introduces a scaling constant  $w_{0}$  that is applied to the weight matrix  $W$  of the first layer of the underlying fully connected network, i.e.  $\sin(w_{0}W\mathbf{x} + \mathbf{b})$ , whereas recent work explored introducing an explicit amplitude parameter whose value is predicted through a secondary neural network [27]. Yet initialization remains a challenge as in practice only a very narrow range of values leads to plausible average loss (Refer to Figure 10 in Appendix C).
In this work, we instead introduce explicit shape parameters for directly controlling amplitude, frequency, and phase. These parameters are completely disentangled from other network parameters, and can be initialized intuitively by reasoning about the shape of resulting sine wave. Moreover, in order to provide better initial coverage of plausible ranges of the shape parameters, we construct a composite periodic function by linearly combining  $K$  sine waves with individual shape parameters:
$$
\sum_ {j = 1} ^ {K} c _ {j} ^ {[ l ]} \sin \left(\mathbf {z} ^ {[ l ]} f ^ {[ l ]} + p ^ {[ l ]}\right), \text {w h e r e} \mathbf {z} ^ {[ l ]} = T ^ {[ l ]} \left(\mathbf {a} ^ {[ l ]}\right). \tag {3}
$$
The shape parameters are shared among individual channels of layers in convolutional neural networks, and are optimized through backpropagation along with other network parameters<sup>2</sup>. In our experiments we observed that initializing the frequency parameters  $f_{j}^{[l]}$  of the  $K$  constituent sine waves of the composite function to cover a large range of frequencies consistently yields improved results, without necessitating lengthy trial and error procedures that are often not feasible when training large models in practice (Refer to Appendix C for an illustrative example).
Beyond the initialization of shape parameters, the remaining known causes of the difficulty when training sin networks are inconsistent gradients due to the complex shapes that activations can take, and the large degeneracy in local minima caused by symmetries [49]. We empirically found that including an additional RELU component to the composite sine wave function discussed before stabilizes training. With this modification our proposed activation, which we call METASIN, can be expressed as follows:
$$
\phi (\mathbf {z} ^ {[ l ]}) = c _ {0} ^ {[ l ]} \operatorname {R E L U} \left(\mathbf {z} ^ {[ l ]}\right) + \sum_ {j = 1} ^ {K} c _ {j} ^ {[ l ]} \sin \left(f _ {j} ^ {[ l ]} \mathbf {z} ^ {[ l ]} + p _ {j} ^ {[ l ]}\right). \tag {4}
$$
Equation 4 can efficiently be implemented as a module in common deep learning frameworks, which then can be used to replace activation functions of an existing state-of-the-art network architecture. The trainable parameters  $\theta = \{c_0^{[l]}, c_j^{[l]}, f_j^{[l]}, p_j^{[l]}; j \in [1, K], l \in [1, L]\}$  are optimized through backpropagation without requiring an additional network for predicting these particular parameters, or any other changes to the underlying architecture. This non-intrusive property enables a simple three-step procedure, where (i) we identify a state-of-the-art image prediction model, (ii) replace its existing activation functions with METASIN, and (iii) re-train the resulting METASIN network using a knowledge distillation scheme we discuss in the following section.
In order to train convolutional METASIN networks, we initialize the shape parameters as  $c_{0} = 1$ ,  $c_{j}^{[l]} = 0$ ,  $f_{j}^{[l]} = j$ , and  $p_{j}^{[l]} = \mathcal{U}(0,\pi)$ , for  $j \in [1,K]$ , and  $l \in [1,L]$  by default. This initialization on one hand forces METASIN to initially assume the shape of RELU, and gradually introduce sin components during the course of the training, while on the other hand covering a broad range of initial frequency and phase values (See Appendix G). Using the default initialization we were able to reliably avoid typical issues encountered while training sin networks.
It is often easier to train a shallower network compared to a deeper one as supported by previous studies [16]. This observation also applies to sin activations, as demonstrated in Figure 1, where a shallow SIREN network can be effectively trained while a deep SIREN network encounters difficulties. To address this, we employ feature knowledge distillation (KD) from a teacher network [35] to provide auxiliary signals during the training of the student METASIN network. We first train a RELU model as the teacher (or, as it often happens in practice, use an already existing baseline RELU model) and use its intermediate feature maps as supervisory signals. This approach allows us to train the shallower blocks of the student METASIN network individually, instead of propagating gradients throughout the entire network. We refer to this phase of training as KD-Bootstrapping, which comprises approximately  $5 - 10\%$  of the total training iterations for the METASIN network. Following the initial KD-Bootstrapping phase, the student METASIN network is trained using the same configuration as the baseline network. This approach enhances the learning process and helps improve the performance of the student network.
2With a certain selection of parameters the resulting composite function can be shown to be equivalent to a Fourier Series, and hence the hyperparameter  $K$  can be thought as controlling the expressiveness (Appendix D).
3.2 Efficient Implementation
While equation 4 is straightforward to implement in both PyTorch and Tensorflow using the corresponding Python APIs, such native implementations end up being inefficient compared to executing a RELU function. For performance critical applications, efficiency issues can be alleviated through writing a customized operation in CUDA (Appendix F). In Table 1 we compare the latency induced by the native PyTorch implementation against our customized METASIN operator with fused CUDA kernel functions. Our optimized implementation significantly reduces the overhead of the native implementation. During forward pass, executing a METASIN with  $K = 10$  components requires less than three times the time it takes to execute a RELU function, whereas the backward computation is only slightly more expensive than a RELU activation. Our efficient implementation also uses roughly the same amount of memory during execution as a RELU activation, making METASIN networks feasible to use in practice on the same hardware that a comparable RELU network is designed for.
Module	Impl.	Forward (rel)	Backward (rel)
METASIN	PyTorch Native	26.1x	1.58x
PyTorch CUDA	2.9x	1.12x
We present experiments where we apply the three step procedure discussed at the end of Section 3. Specifically, we switch an existing architecture's activations to METASIN, and re-train the resulting new model from scratch using KD-Bootstrapping.
Table 1: Latency of executing a METASIN activation with  $K = 10$  using native vs. efficient implementations, relative to the latency of executing a single RELU activation on Nvidia RTX 3090.
Factor	Activation
RELU	RELUE	SNAKE	MISH	SIREN	SIREN KD-B	MRELU	MRELU KD-B	METASIN	METASIN KD-B
P×2	33.03	33.04	32.95	32.96	31.75	32.76	32.78	33.02	33.02	33.26
P×3	29.36	29.36	29.16	29.06	28.18	28.98	29.16	29.36	29.35	29.58
P×4	27.09	27.10	26.97	26.89	26.20	26.83	26.97	27.10	27.11	27.29
Table 2: Comparison of PSNRs obtained from various image resampling models. See text for details.
4.1 Image Resampling
In image resampling our approach builds upon the state-of-the-art network proposed by Bernasconi et al. [5], which incorporates ProSR [45] feature extractor with 3 residual blocks, a resampling layer, and a prediction layer. The network takes an input image and a warp grid and performs image resampling. We trained baseline networks following the exact same training procedures and dataset as described in [5]. The METASIN network is obtained by replacing RELU activations of the feature extractor with METASIN with  $K = 10$  and  $f_{j}^{[l]} = j$  at initialization. All other parameters are initialized as described in Section 3.1. The transition to METASIN activations increases the amount of compute by  $3\%$  (and similarly the wall-clock inference time by  $3\%$ ) compared to the baseline RELU network. To rule out any improvement due to this increase in computation, we accordingly increase the number of channels of baseline RELU network, which we label as  $\mathrm{RELU}_E$ .
We evaluate the models on three predetermined sets of projective transforms with average local scaling factors of  $2(P_{\times 2})$ ,  $3(P_{\times 3})$  and  $4(P_{\times 4})$ . We report the PSNR scores of the resampled images in Table 2, as well as visual examples in Figure 2. We use KD-Bootstrapping during the first  $10\%$  of the training procedure. On average the METASIN network with KD-Bootstrapping improves over the  $\mathrm{RELU}_E$  baseline by  $0.2\mathrm{dB}$ , setting a new state-of-the-art in image resampling. We also conducted KD-Bootstrapping on RELU, MISH and SNAKE networks, but observed no improvement.
RELU
METASIN
Reference
Figure 2: Example upsampling results comparing the state-of-the-art model with RELU activations [5] and the corresponding METASIN network.
This outcome suggests that the training process for these networks is inherently stable, making KD-Bootstrapping unnecessary for these activations. Refer to Appendix K for additional results.
4.2 Denoising Monte-Carlo Rendered Images
Next, we present our experiments in denoising Monte-Carlo rendered images. Our base denoiser network is a U-Net architecture used in [38] and we use the same procedure to generate noisy and reference renderings. Since we focus on direct image prediction models rather than kernel denoising as in [38], our U-Net denoiser (DPCN) directly predicts pixel values of the clean image (For completeness, the results of kernel denoising (KPCN) are presented in Appendix M). During evaluation, we denoise images rendered with [2, 4, 8, 16] samples per pixel and use SMAPE, FLIP [2], 1-MS_SSIM, and 1-SSIM as our metrics. Our best METASIN network is configured with  $K = 5$  and the frequencies are initialized as  $f_{j}^{[l]} = j / 2$ . We use KD-boostrapping during the first  $5\%$  of the training procedure. We present a summary of various METASIN network configurations in Table 3, and selected visual examples in Figure 3.
Metric	DPCN	METASIN-5 (fj = j)	METASIN-10 KD-B (Glorot)	METASIN-5 KD-B (fj = j)	METASIN-5 KD-B (fj = j/2)
SMAPE	3.351	3.16 (-5.7%)	3.118 (-6.95%)	3.088 (-7.85%)	3.081 (-8.06%)
FLIP	1.084	0.991 (-8.58%)	0.972 (-10.33%)	0.976 (-9.96%)	0.965 (-10.98%)
1-MS_SSIM	3.106	2.838 (-8.63%)	2.784 (-10.37%)	2.757 (-11.24%)	2.733 (-12.01%)
1-SSIM	6.895	6.385 (-7.4%)	6.227 (-9.69%)	6.146 (-10.86%)	6.112 (-11.36%)
Table 3: Results of various METASIN configurations. All relative values are in relation to the DPCN baseline. The set of test samples contains all samples with  $\mathrm{spp} \leq 16$ . The frequency initializations are given in parenthesis, which shows that a slight improvement can be made by initializing as  $f_{j} = j / 2$  rather than the default  $f_{j} = j$ . Switching to Glorot initialization results in reduced accuracy despite the doubled  $K$ . Introducing KD-Bootstrapping during training significantly improves accuracy.
In this section we present ablation experiments and an exploratory study on using METASIN activations for image classification and implicit image representation using MLPs. We present preliminary investigations of further applications of METASIN activation in Appendix A
Figure 3: Monte-Carlo denoising results. The first row shows results produced with an input noisy images rendered at 4 samples-per-pixel, whereas the input in the remaining two rows are rendered at 16 samples-per-pixel.
5.1 Comparison with Alternative Activations
Metric	DPCN	MRELU-10	MRELU-10 KD-B	MISH	METASIN-10	METASIN-10 KD-B
SMAPE	3.351	3.472 (+3.61%)	3.254 (-2.89%)	3.293 (-1.73%)	3.346 (-0.15%)	3.118 (-6.95%)
FLIP	1.084	1.096 (+1.11%)	1.031 (-4.89%)	1.054 (-2.77%)	1.06 (-2.21%)	0.972 (-10.33%)
1-MS_SSIM	3.106	3.397 (+9.37%)	2.963 (-4.6%)	3.011 (-3.06%)	3.066 (-1.29%)	2.784 (-10.37%)
1-SSIM	6.895	7.044 (+2.16%)	6.589 (-4.44%)	6.896 (+0.01%)	6.736 (-2.31%)	6.227 (-9.69%)
Table 4: Comparison of various alternative activations on the DPCN model discussed in Section 4.2. All parameters are initialized using Glorot initialization. See text for discussion.
We ran several experiments to assess the comparative improvements that can be achieved by using alternative activations instead of METASIN. In addition to RELU, we also tested SNAKE  $(x + \sin^2 (x))$  in [49]), MISH  $(x\tanh (\ln (1 + e^{x}))$  in [30]), and MRELU in which we linearly combine several RELU activations in the form  $\sum_{j = 1}^{K}c_{j}\mathrm{RELU}(x + b_{j})$  (Similar to Adaptive Piecewise Linear Units [3]). The SNAKE activation [49] performs slightly worse than the RELU baseline in the image resampling application, which in turn falls below the corresponding METASIN activated network (Table  $2 - 3^{nd}$  col). In case of Monte-Carlo denoising it was not possible to train the U-Net model with SNAKE activations as the training diverged. MISH similarly performs slightly worse than both RELU and SNAKE in image resampling (Table  $2 - 4^{th}$  col).
In Monte Carlo denoising the MISH activated network performs better than the RELU baseline by  $3\%$  in terms of MS_SSIM, however still falls significantly behind the METASIN activated network that achieves  $10\%$  improvement in the same experiment (Table 4).
The MReLU activation with  $K = 10$  performs significantly worse than the RELU baseline both in denoising and resampling task (Table 4 -  $3^{rd}$  col and Table 2 -  $7^{th}$  col). We also tested applying