SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Visual comparisons of the generalization capabilities of a scale-equivariant neural network (left) and its normalization-equivariant counterpart (right) for Gaussian noise. Both networks were trained for Gaussian noise at noise level  $\sigma = 25$  exclusively. The adaptive filters (rows of  $A_{\theta}^{y_r}$  in Prop. 3) are indicated for two particular pixels as well as the sum of their coefficients (note that some weights are negative, indicated in red). The scale-equivariant network tends to excessively smooth out the image when evaluated at a lower noise level, whereas the normalization-equivariant network is more adaptable and considers the underlying texture to a greater extent.
Proposition 3 Let  $f_{\theta}^{NE}:\mathbb{R}^n\mapsto \mathbb{R}^m$  be a CNN composed of only:
- affine convolution kernels with no bias and where padding is made of existing features,
sort pooling nonlinearities,
- possibly skip or affine residual connections, and max or average pooling layers.
Then,  $f_{\theta}^{NE}$  is a normalization-equivariant continuous piecewise-linear function with finitely many pieces. Moreover, on each piece represented by the vector  $y_r$ ,
$$
f _ {\theta} ^ {N E} (y) = A _ {\theta} ^ {y _ {r}} y, \text {w i t h} A _ {\theta} ^ {y _ {r}} \in \mathbb {R} ^ {m \times n} \text {s u c h t h a t} A _ {\theta} ^ {y _ {r}} \mathbf {1} _ {n} = \mathbf {1} _ {m}.
$$
In Prop. 3, the subscripts on  $A_{\theta}^{y_r}$  serve as a reminder that this matrix depends on the sort pooling activation patterns, which in turn depend on both the input vector  $y$  and the weights  $\theta$ . As already revealed for bias-free networks with ReLU [33],  $A_{\theta}^{y_r}$  is the Jacobian matrix of  $f_{\theta}^{\mathrm{NE}}$  taken at any point  $y$  in the interior of the piece represented by vector  $y_r$ . Moreover, as  $A_{\theta}^{y_r}\mathbf{1}_n = \mathbf{1}_m$ , the output vector of such networks are locally made of fixed affine combinations of the entries of the input vector. And since a CNN has a limited receptive field centered on each pixel,  $f_{\theta}^{\mathrm{NE}}$  can be thought of as an adaptive filter that produces an estimate of each pixel through a custom affine combination of pixels. By examining these filters in the case of image denoising (see Fig. 3), it becomes apparent that they vary in their characteristics and are intricately linked to the contents of the underlying images. Indeed, these filters are specifically designed to cater to the specific local features of the noisy image: averaging is done over uniform areas without affecting the sharpness of edges. Note that this behavior has already been extensively studied by [33] for unconstrained filters.
The total number of fixed adaptive affine filters depends on the weights  $\theta$  of the network  $f_{\theta}^{\mathrm{NE}}$  and is bounded by  $2^{S}$  where  $S$  represents the total number of sort pooling patterns traversed to get from the receptive filed to its final pixel (assuming no max pooling layers). Obviously, this upper bound grows exponentially with  $S$ , suggesting that a limited number of sort pooling operations may generate an extremely large number of filters. Interestingly, if ReLU activation functions where used instead, the upper bound would reach  $2^{2S}$ .
We demonstrate the effectiveness and versatility of the proposed methodology in the case of image denoising. To this end, we modify two well-established neural network architectures for image denoising, chosen for both their simplicity and efficiency, namely DRUNet [46]: a state-of-the-art
Figure 4: Comparison of the performance of our normalization-equivariant alternative with its scale-equivariant and ordinary counterparts for Gaussian denoising with the same architecture on Set12 dataset. The vertical blue line indicates the unique noise level on which the "blind" networks were trained exclusively (from left to right:  $\sigma = 50$ ,  $\sigma = 25$  and  $\sigma = 10$ ). In all cases, normalization-equivariant networks generalize much more robustly beyond the training noise level.
U-Net with residual connections [17]; and FDnCNN, the unpublished flexible variant of the popular DnCNN [47]: a simple feedforward CNN that chains "conv+ReLU" layers with no downsampling, no residual connections and no batch normalization during training [19], and with a tunable noise level map as additional input [48]. We show that adapting these networks to become normalization-equivariant does not adversely affect performance and, better yet, increases their generalization capabilities. For each scenario, we train three variants of the original Gaussian denoising network for grayscale images: ordinary (original network with additive bias), scale-equivariant (bias-free variation with ReLU [33]) and our normalization-equivariant architecture (see Fig. 2). Details about training and implementations can be found in Appendix A and Appendix B. Unless otherwise noted, all results presented in this paper are obtained with DRUNet [46]; similar outcomes can be achieved with FDnCNN [47] architecture (see Appendix D).
Finally, note that both DRUNet [46] and FDnCNN [47] can be trained as "blind" but also as "non-blind" denoisers and thus achieve increased performance, by passing an additional noisemap as input. In the case of additive white Gaussian noise of variance  $\sigma^2$ , the noisemap is constant equal to  $\sigma \mathbf{1}_n$  and the resulting parameterized functions can then be put mathematically under the form  $f_{\theta}:(y,\sigma)\in \mathbb{R}^{n}\times \mathbb{R}_{*}^{+}\mapsto \mathbb{R}^{n}$ . In order to integrate this feature to normalization-equivariant networks as well, a slight modification of the first affine convolutional layer must be made. Indeed, by adapting the proof (5) to the case (1), we can show that the first convolutional layer must be affine with respect to the input image  $y$  only - the coefficients of the kernels acting on the image pixels add up to 1 - while the other coefficients of the kernels need not be constrained.
5.1 The proposed architectural modifications do not degrade performance
The performance, assessed in terms of PSNR values, of our normalization-equivariant alternative (see Fig. 2) and of its scale-equivariant and ordinary counterparts is compared in Table 2 for "non-blind" architectures on two popular datasets [32]. We can notice that the performance gap between two different variants is less than  $0.05\mathrm{dB}$  at most for all noise levels, which is not significant. This result suggests that the class of parameterized functions  $(f_{\theta})$  currently used in image denoising can drastically be reduced at no cost. Moreover, it shows that it is possible to dispense with activation functions, such as the popular ReLU: nonlinearities can simply be brought by sort pooling patterns. In terms of subjective visual evaluation, we can draw the same conclusion since images produced by two architectural variants inside the training range are hardly distinguishable (see Fig. 3 at  $\sigma = 25$ ).
5.2 Increased robustness across noise levels
S. Mohan et al. [33] revealed that bias-free neural networks with ReLU, which are scale-equivariant, could much better generalize when evaluated at new noise levels beyond their training range, than their counterparts with bias that systematically overfit. Even if they do not fully elucidate how such networks achieve this remarkable generalization, they suggest that scale-equivariance certainly plays a major role. What about normalization-equivariance then? We have compared the robustness faculties of the three variants of networks when trained at a fixed noise level  $\sigma$  for Gaussian noise. Figure
Table 2: The PSNR (dB) results of "non-blind" deep learning-based methods applied to popular grayscale datasets corrupted by synthetic white Gaussian noise with  $\sigma = {15},{25}$  and 50 .
Dataset	Set12	BSD68
Noise level σ	15 / 25 / 50	15 / 25 / 50
DRUNet [46]	ordinary	33.23 / 30.92 / 27.87	31.89 / 29.44 / 26.54
scale-equiv	33.25 / 30.94 / 27.90	31.91 / 29.48 / 26.59
norm-equiv	33.20 / 30.90 / 27.85	31.88 / 29.45 / 26.55
FDnCNN [47]	ordinary	32.87 / 30.49 / 27.28	31.69 / 29.22 / 26.27
scale-equiv	32.85 / 30.49 / 27.29	31.67 / 29.20 / 26.25
norm-equiv	32.85 / 30.50 / 27.27	31.69 / 29.22 / 26.25
y
(b)
$y$ $(x\in [0.3,1]^n)$
Figure 5: Denoising results for example images of the form  $y = x + \lambda \varepsilon$  (see notations of subsection 5.2) with  $\sigma = 25 / 255$  and  $x \in [0,1]^n$  by "blind" CNNs specialized for noise level  $\sigma$  only.  $f_{\theta}^{\emptyset}$ ,  $f_{\theta}^{\mathrm{SE}}$  and  $f_{\theta}^{\mathrm{NE}}$  denote the ordinary, scale-equivariant and normalization-equivariant variants, respectively. In order to get the best results with  $f_{\theta}^{\emptyset}$  and  $f_{\theta}^{\mathrm{SE}}$ , it is necessary to know the renormalization parameters  $(\lambda, \mu)$  such that  $(x - \mu) / \lambda$  belongs to  $\mathcal{D} \subset [0,1]^n$  (see subsection 5.2). Note that for  $f_{\theta}^{\mathrm{SE}}$ , it is however sufficient to know only  $\mu$  as  $\lambda$  is implicit by construction. In contrast,  $f_{\theta}^{\mathrm{NE}}$  can be applied directly.
$f_{\theta}^{\varnothing}(y)$
$f_{\theta}^{\mathrm{SE}}(y)$
$3f_{\theta}^{\varnothing}(y / 3)$
$f_{\theta}^{\mathrm{SE}}(y - 0.3) + 0.3$
$f_{\theta}^{\mathrm{NE}}(y)$
$f_{\theta}^{\mathrm{NE}}(y)$
4 summarizes the explicit results obtained: normalization-equivariance pushes generalization capabilities of neural networks one step further. While performance is identical to their scale-equivariant counterparts when evaluated at higher noise levels, the normalization-equivariant networks are, however, much more robust at lower noise levels. This phenomenon is also illustrated in Fig. 3.
Demystifying robustness Let  $x$  be a clean patch of size  $n$ , representative of the training set on which a CNN  $f_{\theta}$  was optimized to denoise its noisy realizations  $y = x + \varepsilon$  with  $\varepsilon \sim \mathcal{N}(0, \sigma^2 I_n)$  (denoising at a fixed noise level  $\sigma$  exclusively). Formally, we note  $x \in \mathcal{D} \subset \mathbb{R}^n$ , where  $\mathcal{D}$  is the space of representative clean patches of size  $n$  on which  $f_{\theta}$  was trained. We are interested in the output of  $f_{\theta}$  when it is evaluated at  $x + \lambda \varepsilon$  (denoising at noise level  $\lambda \sigma$ ) with  $\lambda > 0$ . Assuming that  $f_{\theta}$  encodes a normalization-equivariant function, we have:
$$
\forall \lambda \in \mathbb {R} _ {*} ^ {+}, \forall \mu \in \mathbb {R}, f _ {\theta} (x + \lambda \varepsilon) = \lambda f _ {\theta} ((x - \mu) / \lambda + \varepsilon) + \mu . \tag {7}
$$
The above equality shows how such networks can deal with noise levels  $\lambda \sigma$  different from  $\sigma$ : normalization-equivariance simply brings the problem back to the denoising of an implicitly renormalized image patch with fixed noise level  $\sigma$ . Note that this artificial change of noise level does not make this problem any easier to solve as the signal-to-noise ratio is preserved by normalization.
Obviously, the denoising result of  $x + \lambda \varepsilon$  will be all the more accurate as  $(x - \mu) / \lambda$  is a representative patch of the training set. In other words, if  $(x - \mu) / \lambda$  can still be considered to be in  $\mathcal{D}$ , then  $f_{\theta}$  should output a consistent denoised image patch. For a majority of methods [46-48], training is performed within the interval [0, 1] and therefore  $x / \lambda$  still belongs generally to  $\mathcal{D}$  for  $1 < \lambda < 10$  (contraction), but this is much less true for  $\lambda < 1$  (stretching) for the reason that it may exceed the bounds of the interval [0, 1]. This explains why scale-equivariant functions do not generalize well to noise levels lower than their training one. In contrast, normalization-equivariant functions can benefit from the implicit extra adjustment parameter  $\mu$ . Indeed, there exists some cases where the stretched patch  $x / \lambda$  is not in  $\mathcal{D}$  but  $(x - \mu) / \lambda$  is (see Fig. 5b). This is why normalization-equivariant networks are more able to generalize at low noise levels. Note that, based on this argument, ordinary neural networks trained at a fixed noise level  $\sigma$  can also be used to denoise images at noise level  $\lambda \sigma$ , provided that a correct normalization is done beforehand [42]. However, this time the normalization is explicit: the exact scale factor  $\lambda$ , and possibly the shift  $\mu$ , must be known (see Fig. 5a).
It turns out that this theoretical argument is valid for a wide range of noise types, not only Gaussian noise. Indeed, the same argument holds for any additive noise  $\varepsilon$  that possesses the scaling property:  $\lambda \varepsilon$  belongs to the same family of probability distributions as  $\varepsilon$  (e.g., Gaussian, uniform, Laplace or even Rayleigh noise which is not zero-mean). By the way, the authors of [33] had already verified the noise generalization capabilities of scale-equivariant networks for uniform noise in addition to Gaussian noise, without fully elucidating why it works. In Appendix D, we checked experimentally that "blind" normalization-equivariant networks trained on additive uniform, Laplace or Rayleigh noise at a single noise level are much more robust at unseen noise levels than their scale-equivariant and ordinary counterparts.
6 Conclusion and perspectives
In this work, we presented an original approach to adapt the architecture of existing neural networks so that they become normalization-equivariant, a property highly desirable and expected in many applications such that image denoising. We argue that the classical pattern "conv+ReLU" can be favorably replaced by the two proposed innovations: affine convolutions that ensure that all coefficients of the convolutional kernels sum to one; and channel-wise sort pooling nonlinearities as a substitute for all activation functions that apply element-wise, including ReLU or sigmoid functions. Despite these two important architectural changes, we show that the performance of these alternative networks is not affected in any way. On the contrary, thanks to their better-conditioning, they benefit, in the context of image denoising, from an increased interpretability and especially robustness to variable noise levels both in practice and in theory.
We would like to mention that the proposed architectural modifications for enforcing normalization-equivariance require a longer training for achieving comparable performance with its original counterparts (see Appendix B), and may be incompatible with some specific network layers such as batch-norm [19] or attention-based modules [7, 24, 26, 45]. Moreover, our method has shown its potential mainly to image denoising as it stands, even though in principle normalization-equivariance may be applicable and helpful in other tasks as well (see preliminary results about image classification in Appendix D). Discovering similar advantages of normalization-equivariance in other computer vision tasks, possibly related to outlier robustness, is an interesting avenue of research for future work.
This work was supported by Bpifrance agency (funding) through the LiChIE contract. Computations were performed on the Inria Rennes computing grid facilities partly funded by France-BioImaging infrastructure (French National Research Agency - ANR-10-INBS-04-07, "Investments for the future"). We would like to thank R. Faisse (Airbus) for fruitful discussions.
[42] Y.-Q. Wang and J.-M. Morel. Can a single image denoising neural network handle all levels of Gaussian noise? IEEE Signal Processing Letters, 21(9):1150-1153, 2014.
[43] M. Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant CNNs. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 849-858, 2018.
[44] G. Yu and G. Sapiro. DCT image denoising: a simple and effective image denoising algorithm. Image Processing On Line, 1:292-296, 2011.
[45] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. Yang. Restormer: Efficient transformer for high-resolution image restoration. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5718-5729, 2022.
[46] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-Play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6360-6376, 2022.
[47] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7):3142-3155, 2017.
[48] K. Zhang, W. Zuo, and L. Zhang. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, 27(9):4608-4622, 2018.
A Description of the denoising architectures and implementation
A.1 Description of models
DRUNet: DRUNet [46] is a U-Net architecture, and as such has an encoder-decoder type pathway, with residual connections [17]. Spatial downsampling is performed using  $2 \times 2$  convolutions with stride 2, while spatial upsampling leverages  $2 \times 2$  transposed convolutions with stride 2 (which is equivalent to a  $1 \times 1$  sub-pixel convolution [39]). The number of channels in each layer from the first scale to the fourth scale are 64, 128, 256 and 512, respectively. Each scale is composed of 4 successive residual blocks “ $3 \times 3$  conv + ReLU +  $3 \times 3$  conv”.
FDnCNN: FDnCNN [47] is the unpublished flexible variant of the popular DnCNN [47]. It consists of 20 successive  $3 \times 3$  convolutional layers with 64 channels each and ReLU nonlinearities. As opposed to DnCNN, FDnCNN does not use neither batch normalization [19] for training, nor residual connections [17] and can handle an optional noisemap (concatenated with the input noisy image). Note that this architecture does not use downsampling or upsampling. Finally, the authors [47] recommend to train it by minimizing the  $\ell_1$  loss instead of the mean squared error (MSE).
A.2 Description of variants
Ordinary: The ordinary variant is built by appending additive constant ("bias") terms after each convolution of the original architecture. Note that the original FDnCNN [47] model is already in the ordinary mode.
Scale-equivariant: Since both models (DRUNet and FDnCNN) use only ReLU activation functions, removing all additive constant ("bias") terms is sufficient to ensure scale-equivariance [33]. Note that the original DRUNet [46] model is already in the scale-equivariant mode.
Normalization-equivariant: All convolutions are replaced by the proposed affine-constrained convolutions without "bias" and with reflect padding, and the proposed channel-wise sort pooling patterns supersede ReLU nonlinearities. Moreover, classical residual connections are replaced by affine residual connections (the sum of two layers  $l_{1}$  and  $l_{2}$  is replaced by their affine combination  $(1 - t)l_{1} + tl_{2}$  where  $t$  is a trainable scalar parameter).
A.3 Practical implementation of normalization-equivariant networks
The channel-wise sort pooling operations can be efficiently implemented by concatenating the sub-layer obtained with channel-wise one-dimensional max pooling with kernel size 2 and its counterpart obtained with min pooling. Note that intertwining these two sub-layers to comply with the original definition is not necessary in practice (although performed anyway in our implementation), since the order of the channels in a CNN is arbitrary.
Regarding the implementation of affine convolutions for training, each unconstrained kernel can be in practice "telescoped" with its circular shifted version (this way, the sum of the resulting trainable coefficients cancels out) and then the inverse of the kernel size is added element-wise as a nontrainable offset. Despite this over-parameterized form (involving an extra degree of freedom), we found this solution to be more easy to use in practice. Moreover, it ensures that all coefficients of the affine kernels follow the same law at initialization. Another possibility is to set an arbitrary coefficient of the kernel (the last one for instance) equal to one minus the sum of all the other coefficients. Note that the solution consisting in dividing each kernel coefficient by the sum of all the other coefficients does not work because it generates numerical instabilities as the divisor may be zero, or close to zero.
All our implementations are written in Python and are based on the PyTorch library [35]. The code and pre-trained models can be downloaded here: https://github.com/sherbret/normalization-equivariant_nn/.
Table 3: Training parameters. * indicates that it is divided by half every 100,000 iterations.
Model	Batch size	Patch size	Loss function	Learning rate	Number of iterations
DRUNet [46]	ordinary	16	128 × 128	\( \ell_1 \)	1e-4*	800,000
scale-equiv	16	128 × 128	\( \ell_1 \)	1e-4*	800,000
norm-equiv	16	128 × 128	MSE	1e-4	1,800,000
FDnCNN [47]	ordinary	128	70 × 70	\( \ell_1 \)	1e-4	500,000
scale-equiv	128	70 × 70	\( \ell_1 \)	1e-4	500,000
norm-equiv	128	70 × 70	MSE	1e-4	900,000
Table 4: Execution time (in seconds) comparison on a training batch of size  $16 \times 1 \times 128 \times 128$  for different variants of the same DRUNet architecture [46] (GPU: Quadro RTX 6000, CPU: 2,3 GHz Intel Core i7). The difference in speed between variants is more pronounced on GPU than on CPU.
B Description of datasets and training details
We use the same large training set as in [46] for all the models and all the experiments, composed of 8, 694 images, including 400 images from the Berkeley Segmentation Dataset BSD400 [32], 4, 744 images from the Waterloo Exploration Database [28], 900 images from the DIV2K dataset [1], and 2, 750 images from the Flickr2K dataset [25]. This training set is augmented via random vertical and horizontal flips and random  $90^{\circ}$  rotations. The dataset BSD32 [32], composed of the 32 images, is used as validation set to control training and select the best model at the end. Finally, the two datasets Set12 and BSD68 [32], strictly disjoint from the training and validation sets, are used for testing.
All the models  $f_{\theta}$  are optimized by minimizing the average reconstruction error between the denoised images  $\hat{x} = f_{\theta}(x + \varepsilon)$ , where  $\varepsilon \sim \mathcal{N}(0, \sigma^2 I_n)$ , and ground-truths  $x$  with Adam algorithm [22]. For "non-blind" models, the noise level  $\sigma$  is randomly chosen from [1, 50] during training. The training parameters, specific to each model and its variants, are guided by the instructions of the original papers [46, 47], to the extent possible, and are summarized in Table 3. Note that each training iteration consists in a gradient pass on a batch composed of patches randomly cropped from training images. Normalization-equivariant variants need a longer training and always use a constant learning rate (speed improvements are however certainly possible by adapting the learning rate throughout optimization, but we did not investigate much about it). Furthermore, contrary to [46] where the  $\ell_1$  loss function is recommended to achieve better performance, supposedly due to its outlier robustness properties, we obtained slightly better results with the usual mean squared error (MSE) loss when dealing with normalization-equivariant networks. Training was performed with a Quadro RTX 6000 GPU.
In Table B, we compare the computational costs of different variants for training and inference. Interestingly, the computational cost on GPU for training is much more sensitive to the "affine mode" (involving affine-constrained convolutions with reflect padding and affine residual connection) than to sort pooling nonlinearities, while it is the opposite for inference. All in all, for gaining normalization-equivariance, the learning and inference time is almost doubled for the DRUNet architecture [46] on GPU. Note however that we do not claim to have the most optimized implementation and there