Normalization-Equivariant Neural Networks with Application to Image Denoising
Authors: Sébastien Herbreteau
Emmanuel Moebel
Charles Kervrann
Venue: NeurIPS 2023 poster (2023)
Keywords: equivariance
normalization
image denoising
activation functions
ReLU
interpretability
robustness
deep learning
analysis of neural networks
TL;DR: Replacing the classic “conv+ReLU” scheme by affine convolutions and sort pooling nonlinearities allows normalization-equivariance for the same performance, which is beneficial for increased generalization across noise levels in image denoising.¬

Abstract: In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or  scaling, results in a corresponding change in the system response.  While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element-wise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.

Normalization-Qquiivriaa Neural Networks with Application to Image Denoising
Sébastien Herbreteau
Emmanuel Moebel
Charles Kervrann
Centre Inria de l'Université de Rennes, France (sebastien herbreteau, emmanuel... comanuel.moebel,, charles kervrann] fr
Abstract
In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting scaling, results in corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normaaization-equiiaaaaaa (scale shift) property, which can be detrimental in many applications. To address this issue, we propose methodology for adapting existing neural networks so that mammalizatioo-cquvvaiaana
holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element- wise the pre- activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce diffin-consstrinne convolutions and channel-wise sort pooling layers surrogates and show that these two architectural modifications do preserve normalization- equivariance without loss of performance. Experimental results in image denoising show that normaaliation-equiaariaa neural networks, in addition to their hetter conditioning, also provide much better generalization across noise levels.
Introduction
Sometimes wrongly confused with the invariance property which designates the characteristic of function f not to be affected by a specific transformation T applied beforehand, the equivariance property, on the other hand, means that reacts accordance with T. Formally, invariance is foT= whereas equivariance reads f o T = T f, where o denotes the function composition operator. Both invariance and equivariance play a crucial role in many areas of study, including physics, computer vision, signal processing and have recently been studied in various settings for deep learning-based models
13
16
In this paper, we focus on the equivariance of neural networks So to specific transformation T, namely normalization. Although highly desirable in many applications and spite of its omnipres- ence in machine learning, current neural network architectures do not equivary to normalization. With application to image denoising, for which normalizaatonn-equivee
is generally guaranteed for we propose methodology for adapting existing neural networks, and in particular denoising CNNs
lot of conventional methods [5] 18
20 37
46 48 so that normalizzaion-equuvariinee equivariance holds by design. In short, the proposed adaptation is based on two innovations:
1. affine convolutions: the weights from one layer to each neuron from the next layer, the convolution kernels in a CNN, are constrained to encode affine combinations of neurons (the sum of the weights is equal to 1).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2. channel-wise sort pooling: all activation functions that apply blement-wise such as the ReLU, are substituted with higher-ding ditherr-imensional nonlinearities, namely two by two sorting along channels that constitutes a fast and efficient normadization-equivaaa alternative.
Despite strong architectural constraints, we show that these simple modifications do not degrade performance and, even better, increase robustness to noise levels in image denoising both in practice and in theory.
2 Related Work
non-exhaustive list of application fields where equivariant neural networks were studied includes graph theory, point cloud analysis and image processing. Indeed, graph neural networks are usually expected to equívary, the sense that permutation of the nodes of the input graph should permute the output nodes accordingly. Several specific architectures were investigated to guarantee such a property point cloud data were proposed in recent line of research [6] [3 40). A typical application is the ability for these networks to produce direction vectors consistent with the arbitrary orientation the input point clouds, thus eliminating the need for data augmentation. Finally, in the domain of image processing, it may be desirable that neural networks produce outputs that equivary with regard to rotations of the input image, whether these outputs are vector fields B11, segmentation maps 41 or even bounding boxes for object tracking [[6].
3 21 38
In parallel, rotation and translation cetiiiirnnnivvaiiin networks for dealing with
In addition to their better conditioning, equivariant neural networks by design are expected to be more robust to outliers. A spectacular example has been revealed by S. Mohan al. [33 in the field of image denoising. By simply removing the additive constant ("bias") terms in neural networks with ReLU activation functions, they showed that a much better generalization at noise levels outside the training range was ensured. Although they do not fully elucidate why biases prevent generalization, and their removal allows it, the authors establish some clues that the answer is probably linked to the scale- ccale-equivariant property of the resulting encoded function: rescaling the input image by positive constant value rescales the output by the same amount.
Overview of normaaliration-qquivaaiaeee
3.1 Definitions and properties of three types of fundamental equivariances
We start with formal definitions of the different types of equivariances studied in this paper. Please note that our definition of "scale" and "shift" may differ from the definitions given by some authors in the image processing literature.
Definition 1 A function f IR" + R is said to be:
scale- eane-eequvariannt Vr R" Me Rt =()x)=1((c)
shift-qquivariant Ya Ra Vu ER, [(x f(x)+ p.
normalization-equivare if is both seale-equivariant and shift-equivariant: Vx E RR"," Va E RØ, Vu E R, f(xr + p) >f(x) + t,
where addition with the scalar shift is applied element-wise.
Note that the ccale-equivariacce property is more often referred to positive homogeneity in pure mathematics. Like linear maps that are completely determined by their values on basis, the above described equivariant functions are actually entirely characterized by the values their take specific subsets of IR, as stated by the following lemma (see proof in Appendix C.T).
Lemma (Characterizations)
R + R is entirely determined by its values on the: unit sphere S of RD if it scale equvvernuia
orthogonal complement of Span(1,) (1,), i.e. Sp, if is shift-equivarian,
intersection& S n Span(1,) 11 if it is normalication-eqiivvrinn,
where 1, denotes the all-ones vector of _
Finally, Lemma 2 highlights three basic ceniiarranceepreserviin mathematical operations that can be used as building blocks for designing neural network architectures (see proof in Appendix C.I

---

Lemma 2 (Operations preserving equivariance) Let f and be two equivariant functions of the same rype (either scale, shift normalization). Then, subject to dimensional compatibility, all of the following functions are still equivariant:
f g (f composed with g).
+ (f(x)) g(x)T)² (concatenation of f and g).
. (1 - [)f + ly for all I. E IR (affine combination of and g).
3.2 Examples of normalization-equivvaa
conventional denoisers
A ("blind")) denoiser is basically function f R" + R" which, given noisy image y E R", tries to map the corresponding noise-free image x E I Since scaling up an image by positive factor adding it up constant shift l does not change its contents, it is natural to expect scale and shift equivariance, iee normalization equivariance, from the denoising procedure emulated by f. In image denoising, majority of methods usually assume an additive white Gaussian noise model with variance o². The corruption model then reads y ~ W(x,, I²),), where I, denotes the identity matrix of size n. and the noise standard deviation
is generally passed additional argument the denoiser ("non-blind" denoising). In this case, the augmented function ʃ (v, E R x R + R said normalization eetpattiiinneevviiiiinnnnt if:
V(y. ER x RR, VUER, J(Ay + u. Ao) = Af(, +M,
as, according to the laws of statistics, xy + N(xr μu, (A)) In what follows, we give some kell-known examples of traditional denoisers that are nermalization-quuivrraan (see proofs in Appendix C.2
Noise-reduction filters:
The most rudimentary methods for image denoising are the smoothing filters, among which we can mention the averaging filter the Gaussian filter for the linear filters and the median filter which is nonlinear. These elementary "blind" denoisers all implement a aammaaliaaton-equvvariinno function. More generally, equivariant and only if its coefficients add up to 1. In others words, normalization-equivee linear filters process images by affine combinations of pixels.
can prove that linear filter is normalization-
Patch-based denoising: The popular N(on)-L(ocal) M(eans) algorithm S and its variants 20
27 consist in computing, for each pixel, an average of its neighboring noisy pixels, weighted by the degree of similarity of the patches to which they belong. In other words, they process images by convex combinations of pixels. More precisely, NLM can be defined as:
fNLM(Y,O), =
Σ (
[ix(r))) 2
Uj with W =
Σ M5E((()))
[p(u,) P(x))
(2)
W
where yi denotes the 7th component of vector y. 12y::) is the set of its neighboring pixels, p(v:) represents the vectorized patch centered at yi. and the smoothing parameter h is proportional to as proposed by several authors [ 12 29]. Defined as such, fNLM is a normalization -eutiiii-quivaiiann function. More recently. (ocal))(ocal Ridge [18] proposes to process images by linear combinations of similar patches and achieves state-of-the-art performance in unsupervised denoising. When restricting the coefficients of the combinations to sum to that is imposing affine combination constraints, the resulting algorithm encodes a
function as well.
TV denoising: Total variation (TV) denoising 37 is finally one of the most famous image denoising algorithm, appreciated for its edge-preserving properties. In its original form (37], TV denoiser is defined as function f _ R + _ that solves the following equality problem:
frvly, σ) arg min ||œ]|
s.t. lly - 2112== =====" no²
=ER"
where | == Vall2 the total variation of T Defined as such, fry is a normalization- equivariant function.

---

Denoising with "blind" DRCNN
PSNR: 36 No DR
PSNR: 31 nR
Ground truth
PSNR: 32.04 D3
PSNR: 35.R: DD
PSNR: 97.95
Noisy
Figure I: Influence of normalization for deep-learning-basseee image denoising. The raw input data a publicly available real-world noisy image of the Convallaria dataset [36]. "Blind" DnCNN [77 with official pre-traine pre-trained weights used for denoising and applied on four different normalization intervals displayed in red, each of which being included in [0. 1] over which it was learned. PSNR calculated with the average of 100 independent noisy static acquisitions of the same sample (called ground truth). Interestingly, the straightforward interval [0, does not give the best results. Normalization intervals are (a) 00 1), (b) [0.08, 0.12], (c) [0.48, 0, 52] and (d) [0.64, 0.96]. In the light of the denoising results (b)-(c) and (b)-(d), DrCNN is neither shift-qquivariant nor scale- equivariant.
3.3
The case of neural networks
Deep learning hides subtlety about normalization equivariance that deserves to be highlighted. Usually, the weights of neural networks are learned on training set containing data normalized to the same arbitrary interval (ag, bo] This training procedure improves the performance and allows for more stable optimization of the model. At inference, unseen data are processed within the interval (ao, bu] via a-ò linear normalization with ag S a < b S by denoted Ta,b and defined by:
a) - min(y)
Topb
+
max(y) min(y)
Note that this transform is actually the unique linear one with positive slope that exactly bounds the output to (a, bj. The data is then passed to the trained network and its response is finally returned to the original range via the inverse operator Tas This proven pipeline is actually relevant in light of the following proposition.
Proposition! 1 Va be B.ERV R" Tad oooofooooo is normalization coomalliation-euuvvvaiinn function.
While normalization-equivaii the hyperparameters a and for given function f Obviously, natural choice for neural networks is to take the same parameters and b as in the learning phase whatever the input image is, i.e. = dg and by. but are they really optimal? The answer to this question is generally negative. Figure depicts an example of the phenomenon in image denoising, taken from real-world application. In this example, straightforward choice largely sub-optimal. This suggests that there are always inherent performance leaks for deep neural networks due to the two degrees of freedom induced by the normalization (i.e., choice of and choice of b). In addition, this poor conditioning can be source of confusion and misinterpretation in critical applications.
appears to be solved, question is still remaining: how to choose
3.4 Categorizing image denoisers
Table summarizes the equivariance properties of several popular denoisers, either conventional [1
37 44 or deep samming-based [24] 26 47]. Interestingly. if scale-equivariance is generally guaranteed for traditional denoisers, not all of them are equivariant to shifts. particular, the widely used algorithms DCT 44 and BM3D II] are sensitive to offsets, mainly because the hard thresholding function at their core is not shifl-equivariant, Regarding the deep-learning--asee networks, only DRUNet [46] is insensitive to scale because it is bias-free convolutional neural
18

---

Table 1: Equivariance properties of several image denoisers (left: traditional, right: beariin-basee)
TV NLM NLR DCT BM3D WNNM DrCNN NLRN SwinIR Restormer DRUNet
Scale Shift
X
X
X
X X
1
x
X
X
X
X
network with only ReLU activation functions [3]. In particular, all transformer models 7 24 26 45 even bias-free, are not scale-equivarian due to their inherent attention- -hased modules. In the next section, we show how adapt existing neural architectures guarantee ammalliation-eeqiiarrinn without loss of performance and study the resulting class of parameterized functions (fo).
Design of Normaliraation-qqiivaana Networks
4.1 Affine convolutions
To justify the introduction of new type of convolutional layers, let us study one of the most basic neural network, namely the linear (parameterized) function fe & are a matrix of IR mx,, Indeed, fe can be interpreted as a dense neural network with no bias, no hidden layer and no activation function. Obviously, fe is always scale-equivariant, whatever the weights θ. As for the shift- nhfftequivarriace,, a simple calculation shows that:
IR θz, where parameters
x + Oz is shifl-equivariant < Vr RR" Vu E R (1,, = Ox ulm + 01, 1m (5)
if and only if each row of matrix θ sums to 1. In other words, for the normalization -ew.tttiiinneiiiiainnneee to hold, the rows of θ must encode weights of affine combinations. Transposing the demonstration any convolutional neural network follows from observation that a convolution from an input layer size x W x C' to an output layer of size H' x W' C' can always be represented with a dense connection by vectorizing the input and output layers. The elements of the C' convolutional kernels of size l x x each are then stored separately along the rows of the (sparse) transition matrix @ of size (H' W' x C') (H x W C). gommlliation-equiiarraaae equivariance and only if the weights
Therefore, fe is
Therefore, a convolutional layer preserves the normalization of the each convolutional kernel sums to 1. In the following, call such convolutional layers "affine convolutions".
In order to guarantee the affine constraint on each convolutional kernel throughout the training phase, one possibility is to "telescope" the circular shifted version an unconstrained kernel to itself (this way, the sum of the resulting trainable coefficients cancels out) and then add the inverse of the kernel size element-wise as a non-trainable offset. Despite this over-parameterized form (involving an extra degree of freedom), all coefficients of the affine kernels follow the same law at initialization.
found this solution to be easier use in practice. Moreover, it ensures that
Since normalization-enn eetiaaaaiiiiieiiivviinaaeeee is preserved through function composition, concatenation and affine combination (see Lemma 2). h. a (linear) convolutional neural network composed only affine convolutions with no bias and possibly skip affine residual connections (trainable affine combination of two layers), guaranteed to normalization-equiiee with existing features (reflect, replicate circular padding for example). Obviously, in their current state, these neural networks are little interest, as linear functions do not encode best-performing functions for many applications, image denoising being no exception. Nevertheless, based on such networks, we show the next subsection how to introduce nonlinearities without breaking
provided that padding is performed
4.2 Channel-wise sort pooling as normaliiztiion-qquvaiiaa alternative to ReLU
The first idea that comes mind is apply nonlinear activation function şp R preserving cermaaliaatio-eqqivvree
after each affine convolution. In other words, we look for nonlinear solution y of the characteristic functional equation of normullzation-equivvaitt
functions (see Def.
[ for = 1. Unfortunately, according to Prop. 2 (see proof in Appendix C. which is based on Temma []. the unique solution is the identity function which is linear. Therefore, activation functions that apply element-wing wise are to be excluded.
5

---

Affine convolution
Channel- sort pooling
by sorting in all noo-overlapping pools:
Σ
Figure 2: Illustration of the proposed alternative for replacing the traditional scheme "convolution + element-wise activation function" convolutional neural networks: affine convolutions supersede ordinary ones by restricting the coefficients of each kernel to sum to one and the proposed sort pooling patterns introduce nonlinearities by sorting two by two the pre-activated neurons along the channels.
Proposition 2 Let NE(n) be the set of purmaalization-equivvaiiat functions from IR" to R"
NE(1) {z + x} and
NE( = (x1, x22) + A
if S x2 else
I1 x2
A, A. R2 s.t. Al, = B12 = 12
x2
To find interesting nonlinear functions, one needs to examine multi-dimensional activation functions, iee. ones of the form y R" + R" with n 2 2. In order to preserve the dimensions of the neural layers and to limit the computational costs, focus on the case processes pre pre-activated neurons by pairs. According to Prop. 2 the normadization-cquivrian functions from R to R22 parameterized by two matrices A, B linear mapping depending on whether or not the input is
= 2, meaning that
T 2x2 such that A12 B1, 1
and apply a different aaffine-constainee in ascending order. As long A * B, the resulting function is nonlinear and makes it de facto candidate to replace the conventional one-dimensional activation functions such as the popular ReLU (rectified linear unit) function. Interestingly, when arbitrarily choosing A and B to be the permutation function simply reads:
matrices of p2x2 the resulting carmmalizaiio-equivvan
min((21, T22) max(211, 222)
s (x), (22
E R22
which is nothing else than the sorting function in IR... Clearly, it is among the simplest normalization- equivariant nonlinear function from I² to I² and the one we consider as surrogate for the one-dimensional activation functions (choosing other functions, that is considering other choices for and B, does not bring improvements in terms performance our experiments). More generally, it is easy to show that all the sorting functions of R² as soon as n 2 2. Note that such sorting operators have been promoted by [ in totally different contexts for their norm- preserving properties of the backpropagated gradients.
aataaaaiiiaiioneeiivvvraaaee and are nonlinear
Since the sorting function (6) is be applied on non-overlapping pairs of neurons, the partitioning of layers needs to be determined. In order not to mix unrelated neurons, we propose to apply this wooddieensional activation function channel-wisely across layers and call this operation "sort pooling" in reference to the max pooling operation, widely used for downsampling, and from which it can be effectively implemented. Figure 2 illustrates the sequence of the two proposed innovations, namely affine convolution followed by channel-wise sort pooling, to replace the traditional scheme "conv+ReLU", while guaranteeing normalization iiiiiiaatiiieuuuvvviiaaaeee
4.3 Encoding adaptive affine filters
Based on Lemma? 2 we can formulate the following proposition which tells more about the class of parameterized functions (fo) encoded by the proposed networks.

---

NOISY
SCALE-EQUIVARIANT
NORMALIZATION NORSALLZATIONEQUUAAAANY
Denoised
Adaptive filters
Denoised
Adaptive filters
NR
PSNR:
Σ U.99
PSNK:
-=
25
Σ 1.00
==1
PSNR: 34.15 DB
PSNR: 30,56 DB
0.94
PSNR:
x=
36.72 bB
0.99
E=1
Figure 3: Visual comparisons of the generalization capabilities of scale-equivarian neural network (left) and its normalization-equuvvrr
counterpart (right) for Gaussian noise. Both networks were trained for Gaussian noise at noise level a 25 exclusively. The adaptive filters (rows of A in Prop. 3 are indicated for two particular pixels as well as the sum of their coefficients (note that some weights are negative, indicated in red). The calleequuvvaiiat network tends to excessively smooth out the image when evaluated at a lower noise level, whereas the pormalization-equivveee more adaptable and considers the underlying texture to greater extent.
network
Proposition 3
Let : Rm be a CNN composed of only:
affine convolution kernels with no bias and where padding is made of existing features, sort pooling nonlinearities,
possibly skip or affine residual connections, and max or average pooling layers.
Then, F is cermalicatonnequuvaae
continuous piecewise-lineal function with finitely many pieces. Moreover, on each piece represented by the vector yr-
(1y A y. with AV- mm such that A 1,, = 1m
In Prop. 3 the subscripts on A serve as reminder that this matrix depends on the sort pooling activation patterns, which in turn depend on both the input vector y and the weights t. As already revealed for bias-free networks with ReLU (33), A is the Jacobian matrix of fNE taken at any point in the interior of the piece represented by vector yr. Moreover, as A 1, = 1m the output vector of such networks are locally made of fixed affine combinations of the entries of the input vector. And since a CNN has limited receptive field centered on each pixel, fNF can be thought of as an adaptive filter that produces an estimate of each pixel through a custom affine combination of pixels. By examining these filters in the case of image denoising (see Fig. 3}. it becomes apparent that they vary their characteristics and are intricately linked to the contents of the underlying images. Indeed, these filters are specifically designed to cater to the specific local features of the noisy image: averaging done over uniform areas without affecting the sharpness of edges. Note that this behavior has already been extensively studied by 333 for unconstrained filters.
The total number of fixed adaptive affine filters depends on the weights 0 of the network f and is bounded by 2 where S represents the total number of sort pooling patterns traversed to get from the receptive filed to its final pixel (assuming no max pooling layers). Obviously, this upper bound grows exponentially with S, suggesting that limited number of sort pooling operations may generate an extremely large number of filters. Interestingly, if ReLU activation functions where used instead, the upper bound would reach 228
Experimental results
We demonstrate the effectiveness and versatility of the proposed methodology in the case of image denoising. To this end, we modify two well-established neural network architectures for image denoising, chosen for both their simplicity and efficiency, namely DRUNet [46): state-of-the-art
7

---

365
32
Onipet ESNR
28
24
21
20
ordinary
scale-equiv
16
norm-equiv
ideerity
8
16
20
24
32
12
16
20
28
Input PSNR
Figure Comparison of the performance of our normalizztiion-equivrraa alternative with its scale-equiven equvv-equuvariinnt and ordinary counterparts for Gaussian denoising with the same architecture on Set12 dataset. The vertical blue line indicates the unique noise level on which the "blind" networks were trained exclusively (from left to right: = 50, = 25 and a 10). In all cases, normalization- equivariant networks generalize much more robustly beyond the training noise level.
U-Net with residual connections 17: and FDnCNN, the unpublished flexible variant of the popular DnCNN 447: simple feedforward CNN that chains "conv+ReLU" layers with downsampling, no residual connections and no batch normalization during training [1]] and with tunable noise level map as additional input [48]. We show that adapting these networks to become normalization- equivariant does not adversely affect performance and, better yet, increases their generalization capabilities. For each scenario, we train three variants of the original Gaussian denoising network for grayscale images: ordinary (original network with additive bias), scale-equivariant (bias-free variation with ReL.U 33 and our normalization--qii vettttttooneeiiviiiannn.m architecture (see Fig. D Details about training and implementations can be found in Appendix A and Appendix [ Unless otherwise noted, all results presented in this paper are obtained with DRUNet [46]; similar outcomes can be achieved with FDrCNN [47 architecture (see Appendix D
Finally, note that both DRUNet 46 and FD1CNN 47 can be trained "blind" but also "non- blind" denoisers and thus achieve increased performance, by passing an additional noisemap as input. the case of additive white Gaussian noise of variance o², the noisemap constant equal to oln and the resulting parameterized functions can then be put mathematically under the form fo (y,o) T R r> TR." In order to integrate this feature networks well, slight modification of the first affine convolutional layer must be made. Indeed, by adapting the proof (5) to the case ① we can show that the first convolutional layer must be affine with respect to the input image only - the coefficients of the kernels acting on the image pixels add up to while the other coefficients of the kernels need not be constrained.
cormalizationeqqiivee
5.1 The proposed architectural modifications do not degrade performance
The performance, assessed in terms of PSNR values, of our normalization -eeiiiaatiio--equuvviiiat alternative (see Fig. ( and of its scale-equivariant and ordinary counterparts compared in Table for "non-blind" architectures on two popular datasets 32 We can notice that the performance gap between two different variants is less than 0.05 dB most for all noise levels, which is not significant. This result suggests that the class of parameterized functions (fe) currently used in image denoising can drastically reduced at no cost. Moreover, it shows that it is possible to dispense with activation functions, such as the popular ReLU: nonlinearities can simply be brought by sort pooling patterns. terms of subjective visual evaluation, we can draw the same conclusion since images produced by two architectural variants inside the training range are hardly distinguishable (see Fig. 3 at 25).
5.2 Increased robustness across noise levels
Mohan er al. revealed that bias-free neural networks with ReLU, which are scale-equivariant could much better generalize when evaluated at new noise levels beyond their training range, than their counterparts with bias that systematically overfit. Even if they not fully elucidate how such networks achieve this remarkable generalization, they suggest that scale-equivariunce certainly plays a major role. What about www.aaiiaatoneequuvvriaaa then? We have compared the robustness facul- ties of the three variants of networks when trained at fixed noise level for Gaussian noise. Figure

---

Table 2: The PSNR (dB) results of "non-blind" deep learning- seaning-basedd methods applied to popular grayscale datasets corrupted by synthetic white Gaussian noise with o = 15, 25 and 50.
Dataset
Set12
BSD68
Noise level
15
25 / 50
15 / 25
50
ordinary
33.23 30.92 27.87
31.89/ / 29 44 26.34
DRUNet 46
scale-equiv
33.25
30.94 / 27.90
31.91 / 29.48 26.59
norm-equiv
33.20
30.90 27.85
31.88 29.45 26.55
ordinary
32.87
30.49 27.28
31.69 29.22 / 26.27
FD&CNN 47
scale-equiv
32.85 / 30.49 / 27.29
31,67 / 29.20 26.25
norm-equiv
32.85 30.50 27.27
31.69 29.22 26.25
PSNR: 25 28.28 DB
PSNR: 25.20
(a)
fo (v)
3f/ (v/3)
ANE (V)
PSNR 22-67 OR 0.7
PSNR: 31.68 DB
PSNR: DB
PSNR: 32.45 DB
(b)
y (x E [0.3, 1]")
(v)
f0 0.3) + 0.3
NE (v)
Figure 5: Denoising results for example images of the form y x + As (see notations of subsection 5.2 with = 25/255 and x E [0, 1]" by "blind" CNNs specialized for noise level only. f f and fN denote the ordinary, scale-equivarian and normalization-equiieen In order get the best results with f and S it is necessary know the renormalization parameters (X, ()) such that (x p)/X belongs to DD [0, 1]" (see subsection 5.2). Note that for fSE, it is however sufficient to know only μ as À is implicit by construction. In contrast, (NE can be applied directly.
variants, respectively.
4 (summarizes the explicit results obtained: wamauliiaaion-qqivvarinnn pushes generalization capa- bilities of neural networks step further. While performance is identical their scale-equivariant counterparts when evaluated at higher noise levels, the normalization- eeqivvaiiiiiiii networks are, however, much more robust at lower noise levels. This phenomenon is also illustrated in Fig.
Demystifying robustness Let x be a clean patch of size n, representative of the training set on which CNN fo was optimized denoise its noisy realizations y I E with e (0 o² Ia) (denoising at a fixed noise level a exclusively). Formally, we note x E D C R", where 'D is the space of representative clean patches of size n on which fe was trained. We are interested in the output of fo when it is evaluated at z + As (denoising at noise level Xα) with À > 0. Assuming that fy encodes normalization setplliaatioo-euuvvviiiii equivariant function, we have:
AA R Vu E R, fo(z + Xe) Afo((z - /)/A+c)++1
The above equality shows how such networks can deal with noise levels Ao different from a:: simply brings the problem back to the denoising of an implicitly renor- malized image patch with fixed noise level a. Note that this artificial change of noise level does not make this problem any easier to solve as the signal-to-noise ratio is preserved by normalization.

---

Obviously, the denoising result of + λE will be all the more accurate as (x (x- is representative patch of the training set. In other words, if (x 3- =u/A can still be considered be in D. then fo should output consistent denoised image patch. For majority of methods 48] training is performed within the interval [0, 1] and therefore x/ still belongs generally to D for < X < 10 (contraction), but this is much less true for À < (stretching) for the reason that it may exceed the bounds of the interval [o, 1]. This explains why scale-equivariant functions do not generalize well to noise levels lower than their training one. In contrast, pormalization-equivvrre benefit from the implicit extra adjustment parameter J.. Indeed, there exists some cases where the stretched patch 2 /À is not in D but (x ni/AA A is (see Fig. 5). This is why networks are more able to generalize at low noise levels. Note that, based on this argument, ordinary neural networks trained at fixed noise level provided that a correct normalization is done beforehand 42 However, this time the normalization is explicit: the exact scale factor X, and possibly the shift fr, must be known (see Fig. 5p).
functions can
also be used to denoise images at noise level λo,
turns out that this theoretical argument is valid for a wide range of noise types, not only Gaussian noise. Indeed, the same argument holds for any additive noise that possesses the scaling property: As belongs to the same family of probability distributions as E (e.g., Gaussian, uniform, Laplace or even Rayleigh noise which is not zero-mean). By the way, the authors of [33] had already verified the noise generalization capabilities of equivaraivariinnt networks for uniform noise in addition to Gaussian noise, without fully elucidating why it works. In Appendix D checked experimentally normulization-qqiivraantt that "blind" normalization equivariant networks trained on additive uniform, Laplace or Rayleigh
noise at single noise level are much more robust at unseen noise levels than their scale- equiieequi and ordinary counterparts.
6 Conclusion and perspectives
In this work, we presented an original approach to adapt the architecture of existing neural networks so that they become normallzation--qqiivarre applications such that image denoising. We argue that the classical pattern "conv+ReL.U" can be favorably replaced by the two proposed innovations: affine convolutions that ensure that all coefficients of the convolutional kernels sum to one; and channel-wise sort pooling nonlinearities as a substitute for all activation functions that apply element-wise, including ReLU or sigmoid functions. Despite these two important architectural changes, we show that the performance of these alternative networks is not affected in any way. On the contrary, thanks their better-conditioning. they benefit, in the context of image denoising, from an increased interpretability and especially robustness to variable noise levels both in practice and in theory.
property highly desirable and expected in many
Limitations
We would like to mention that the proposed architectural modifications for enforeing normalization- equivariance require a longer training for achieving comparable performance with its original counterparts (see Appendix B). and may be incompatible with some specific network layers such as batch-norm [19 or attention- based modules 2724 26 45). Moreover, our method has shown its potential mainly to image denoising as it stands, even though in principle catmmalizaion-eqivarriiin may be applicable and helpful in other tasks as well (see preliminary results about image classification in Appendix DD). Discovering similar advantages of normalization nornalization-qquivriin
in other computer vision tasks, possibly related to outlier robustness, is an interesting avenue of research for future work.
Acknowledgments and Disclosure of Funding
This work was supported by Bpifrance agency (funding) through the LiChlE contract. Computations were performed on the Inria Rennes computing grid facilities partly funded by Trance-Biolmaging infrastructure (French National Research Agency- ANR- ANR-10-INBS-004-00 "Investments for the future"). We would like to thank R. Fraisse (Airbus) for fruitful discussions.
10

---

References
[1] E. Agustsson R. Timofte. NTIRE 2017 Challenge on single image super-resolution: Dataset and study. In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1122-1131, 2017.
[2] C. Anil, J. Lucas, and R. Grosse. Sorting out Lipschitz function approximation. In International Conference on Machine Learning (ICML), volume 97, pages 291 1011001,20000 2019.
[3] S. Batzner, A. Musaclian, L.. Sun, M. Geiger, J. Mailoa, M. Kombluth, N. Molinari, T. Smidt, and
B. Kozinsky. -quueruuial graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1) 13(1):2453, 2022.
[4] A. Buades, B. Coll, and J.M. Morel. A non-local algorithm for image denoising. In Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 60-65, 2005.
[5] A. Buades, B. Coll, and J.-M. Morel. A review of image denoising algorithms, with a new onc. Multiscale Modeling & Simulation, 4(2))490-553) 2005.
[6] G. Bökman, F. Kahla, and A. Flinth. ZZ-Net: A universal rotation equivariant architecture for 2D point clouds. In Conference Computer Vision and Pattern Recognition (CVPR), pages 10966- 10975, 2022.
[7] L. Chen, X. Chu, X. Zhang, and J. Sun. Simple baselines for image restoration. In European Conference Computer Vision (ECCV), pages 17-33, 2022.
[8] Y. Chen and T. Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1966):1256-1222. 2017.
[9] A. Chernodub and Nowicki. Norm Norm- preserving orthogonal permutation linear unit activation functions (OPLU). arXiv preprint arXiv:1604.0233: 2016.
[10] T. Cohen and M. Welling. Group equivariant convolutional networks. International Conference on Machine Learning (ICML), volume 48, pages 2990-2999, 2016.
[11] K. Dabow, A. Foi, V. Katkovnik, and K. Egiazarian. Image demoising by sparse 3-D transform-domain collahorative filtering, IEFE Transactions on Image Processing, 1668):2080-2055, 2007.
[12] V. Duval, J.-F. Aujol, and Gousscau. viri-nariance approach for the nonlocal means. SIAM Journal on Imaging Sciences, 4(2):760-788
2011.
[13] F. Fuchs, D. Worrall, V. Fischer, and M. Welling. SE(3) Trans-Transoorme:::: 3D roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1970-1981, 2020.
[14] K. Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybermetics, 36(4):193 202, 1980.
[15] S. Gu, L. Zhang, W. Zuo, and X. Feng. Weighted nuclear norm minimization with application to image denoising. In Conference Computer Vision and Patlern Recognition (CVPR). pages 2862-2869. 2014.
[16] D. K. Gupta, D. Arya, and E. Gavves. Rotation equivariant siamese networks for tracking. In Conference Computer Vision and Pattern Recognition (CVPR), pages 12357-12366, 2021.
[17] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[18] S. Herbreteau and Kervrann. Towards unified view of unsupervised non-local methods for image denoising: NL-Ridge approach. IEEE International Conference on Image Processing (ICIP), pages 3376-3380, 2022.
[19] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), volume 37, pages 448-456, 2015.
[20] Q. Jin, I. Grama, C. Kervrann, and Q. Liu. Nonlocal means and optimal weights for noise removal. SIAM Journal on Imaging Sciences, 10(4):1878-199 2017.
[21] N. Keriven and G. Peyré. Universal invariant and equivariant graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.

---

[22] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Reprexentations (ICLR), 2015.
[23] Y. LeCun, B. Boser. J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropa- gation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989.
[24] J. Liang, J. Cao, G. Sun, K. Zhang. L. Van Gool, and R. Timofte. SwinIR: Image restoration using Swin Transformer. In International Conference Computer Vision Workshops (ICCWW), pages 1833 1844, 2021.
[25] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132-1140, 2017.
[26] D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang. Non-local recurrent network for image restoration. In Advances in Neural Information Processing Systems (NeurlPS), volume 31, 2018.
[27] C. Louchet and Moisan, Total variation as local filter. SIAM Journal Imaging Sciences, 4(2):651- 694, 2011.
[28] K. Ma, Z. Duammu, Q. Wu, Wang, H. Yong, H. Li, and L.. Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing, 26(2):1004- 1016, 2017.
[29] V. Manjón, J. Carbanell-Cahhllero,, J. Lull, G. Gareía-Marti, Mantí-Bonmati, and M. Robles. MRT denoising using non-local means. Medical image analysis, 12(4):514-523, 2008.
[30] X. Mao, C. Shen, and Y.-B. Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neural Information Processing Syxtems (NIPS), volume 29, 2016.
[31] D. Marcos, M. Volpi, N. Komodakis. and D. Tuia. Rotation equivariant vector field networks. In International Conference on Computer Vision (ICCV), pages 5058-5067 2017.
[32] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International Conference on Computer Vision (ICCV), volume 2, pages 416-44 423 vol.2, 2001.
[33] S. Mohan, Z. Kadkhodaie, E. P. Simoncelli, and C. Feraandez-Grane Robust and interpretable blind image denoising via bias free convolutional neural networks. In International Conference Learning Representations (ICLR), 2020.
[34] A. Nøkland and L. H. Eidnes. Training neural networks with local error signals. In International Conference on Machine Learning (ICML), pages 4839 4850, 2019.
[35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein. Kopf, F.. Yang. 7. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
I.. Antiga, A. Desmaison,
[36] Prakash, M. Lalit, Tomancak, A. Krul, and F. Jug. Fully unsupervised probabilistic Noise2 In International Symposium on Biomedical Imaging (ISBI), pages 154-158, 2020.
[37] L. Rudin, Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60:259-268, 1992.
[38] V. G. Satorras, F. Hoogeboom, and M. Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning (ICML), volume 139, pages 9323 9232,9933,27 2021.
[39] Shi, J. Caballero, F. Huszár, Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Conference on Computer Vision and Pattern Recognition (CVPR), pages
2016.
[40] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor field networks: Rotation- and translation-equivariaal neural networks for 3D point clouds. arXiv preprini arXiv:1802.08211 2018.
[41] B. S. Veeling, Linmans, J. Winkens, T. Cohen, and M. Welling. Rotation equivariant CNNs for digital pathology. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 210-20 21-218 2018.
12

---

[42] Y.QQ. Wang and J.-M. Morel. Can a single image denoising neural network handle all levels of Gaussian noise? IEEE Signal Processing Letters, 15001115 2014.
[43] M. Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant CNNs. In Conference Computer Vision and Pattern Recognition (CVPR), pages 849-85898 858, 2018.
[44] G. Yu and G. Sapiro. DCT imaae denoising: simple and effective image denoising algorithm. Image Processing On Line, 1:292-296, 2011.
[45] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. Yang. Restormer: Efficient transformer for high-resolution image restoration. In Conference Computer Vision and Puttern Recognition (CVPR). pages 5718-5729, 2022.
[46] K. Zhang. Y. Li, W. Zuo, L. Zhang, L.. Van Gool, and R. Timofte, Plug-and-Play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6360 6376. 2022.
[47] Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7):3142-3155, 2017.
[48] Zhang, Zuo, and Zhang. FFDNet: Toward fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, 27(9):4608-440) 4424608-422,,, 2018.
13

---

A Description of the denoising architectures and implementation
A.1 Description of models
DRUNet 46 a U-Net architecture, and such has an encoder-decoder type pathway, Spatial downsampling is performed using 2 x 2 convolutions with stride 2, while spatial upsampling leverages x 2 transposed convolutions with stride (which equivalent to a X 1 sub-pixel convolution (39] The number of channels in each layer from the first scale to the fourth scale are 64, 128, 256 and 512, respectively. Each scale is composed of successive residual blocks "3 X 3 conv + ReLU 3 x conv".
DRUNet:
with residual connections 17
FDnCNN: FDrCNN 47 is the unpublished flexible variant of the popular DnCNN 477. It consists of 20 successive 3 x 3 convolutional layers with 64 channels each and ReLU nonlinearities. As opposed to DrCNN, FDrCNN does not neither batch normalization [T9] for training, residual connections 17 and can handle an optional noisemap (concatenated with the input noisy image). Note that this architecture does use downsampling upsampling. Finally, the authors [47 recommend to train it by minimizing the & loss instead of the mean squared error (MSE).
A.2 Description of variants
The ordinary variant is built by appending additive constant ("bias") terms after each convolution of the original architecture. Note that the original FDrCNN 47 model is already in the ordinary mode.
Ordinary:
Scale-equivariant: Since both models (DRUNet and FDnCNN) use only ReLU activation functions, removing all additive constant ("bias") terms is sufficient ensure cealeequuvvariace 333 Note that the original DRUNet [46] model is already in the scale-equivariant mode.
All convolutions are replaced by the proposed affine-constriined convolutions without "bias" and with reflect padding, and the proposed channel-wise sort pooling patterns supersede ReLU nonlinearities. Moreover, classical residual connections are replaced by affine residual connections (the sum of two layers 4 and l2 is replaced by their affine combination (1 - t)4 fly where / is trainable scalar parameter).
Normalization-equiaaiiie
A.3 Practical implementation nurmaliiztionn-qquiaaiinn networks
The channel-wise sort pooling operations can be efficiently implemented by concatenating the sub- layer obtained with channel-wise mie-dimeniional max pooling with kernel size 2 and its counterpart obtained with min pooling. Note that intertwining these two sub-layers comply with the original definition not necessary in practice (although performed anyway in our implementation), since the order of the channels in CNN is arbitrary.
Regarding the implementation of affine convolutions for training, each unconstrained kernel can be in practice "telescoped" with its circular shifted version (this way, the sum of the resulting trainable coefficients cancels out) and then the inverse of the kernel size is added element- teieentwwiseee as a non- trainable offset. Despite this over parameterized form (involving extra degree of freedom), found this solution to be more easy to use in practice. Moreover, it ensures that all coefficients the affine kemells follow the same law at initialization. Another possibility is to set arbitrary coefficient of the kernel (the last one for instance) equal to one minus the sum of all the other coefficients. Note that the solution consisting in dividing each kernel coefficient by the sum of all the other coefficients does not work because it generates numerical instabilities as the divisor may be zero, or close to zero.
All our implementations pre-trained models can be downloaded here: https //tgtttgtttt//t// com/ http://hebbeeeetoooeeeeeee www.imiiiiii..... equivari ant wetivvrrinntttn. _n
written in Python and are based on the Py library 35 The code and
14

---

Table 3: Training parameters. indicates that it is divided by half every 100, 000 iterations.
Batch
Patch size
Loss
Learning
Number of
Model
size
function
rate
iterations
ordinary
16
128 x 128
1e-4*
800, 000
DRUNet 46
scale-equiv
16
128
128
&
le-4*
800, 000
norm-equiv
16
128
128
MSE
le-d
1,000 000
ordinary
128
70 70
E1
le-d
500, 000
FDrCNN
scale-equiv
70
le-4 500, 000
norm-equiv 128
70
70
MSE
le-4
900, 000
Table 4: Execution time (in seconds) comparison training batch of size 16 128 x 128 for different variants of the same DRUNet architecture [46] (GPU: Quadro RTX 6000, CPU: 2,3 GHz Intel Core 17). The difference in speed between variants is more pronounced on GPU than CPU. Inference pass +
Affine
SortPool
Backward pass
(scale-equiv)
0.229
0.067
GPU
0.268
0.102
x
0.344
0.083
(norm-equiv)
0.386
0.122
Variant
Backward pass +
Inference pass +
CPU
ordinary
38.4
12.5
scale-equiv
37.4
12.2
norm-equiv
45.3
14.3
Affine: affine-constrained convolutions with reflect padding and affine residual connections. SortPool: channel-wise sort pooling nonlinearities instead of ReLU.
B Description of datasets and training details
We use the same large training set as in 46 for all the models and all the experiments, composed of
8, 694 images, including 400 images from the Berkeley Segmentation Dataset BSD400 [32. 4, 744 images from the Waterloo Exploration Database (28) 900 images from the DIV2K dataset [1, and 2. 750 images from the Flickr2K dataset (25]. This training set is augmented via random vertical and horizontal flips and random 90" rotations. The dataset BSD32 [2], composed the 32 images, is used as validation set to control training and select the best model at the end. Finally, the two datasets 2 and BSD68 (32), strictly disjoint from the training and validation sets, are used for testing.
All the models fa are optimized by minimizing the average reconstruction error between the denoised images å = fo( + +). where
(O, I²),, and ground-truths æ with Adam algorithm (22). For "non-blind" models, the noise level a is randomly chosen from [1, 50) during training. The training parameters, specific to each model and its variants, are guided by the instructions of the original papers [6 47].
to the extent possible, and are summarized in Table Note that each training iteration consists in a gradient pass on batch composed of patches randomly cropped from training images. Normulization-equuvva rate (speed improvements are however certainly possible by adapting the learning rate throughout optimization, but we did not investigated much about it). Furthermore, contrary to [46] where the E
variants need a longer training and always use constant learning
loss function recommended to achieve better performance, supposedly due to its outlier robustness properties, we obtained slightly better results with the usual mean squared error (MSE) loss when dealing with normalization- -etirirrinniiviiiant networks. Training was performed with a Quadro RTX 6000 GPU.
In Table we compare the computational costs of different variants for training and inference. Interestingly, the computational cost on GPU for training is much more sensitive to the "affine mode" (involving affine-constrained convolutions with reflect padding and affine residual connection) than to sort pooling nonlinearities, while it is the opposite for inference. All in all, for gaining normalization- equivariance, the learning and inference time is almost doubled for the DRUNet architecture [46] on GPU. Note however that we do not claim to have the most optimized implementation and there
15

---

is probably room for improvement. Surprisingly, the difference in speed between all variants is much less pronounced on CPU. In particular. the inference pass takes only about 20% longer for the normalization-eqquivae DRUNet on CPU.
C Mathematical proofs for normalisation-equiiarian neural networks
C.1 Proofs of Lemmas and Propositions
Cemmam[[Caracteerin
Proof: For each type of equivariance, both existence and uniqueness of f must be proven. Let 0,, be the zero vector of IK" and (ee) the values that f takes its characteristic set C.
Scale- equeverquivriance
Uniqueness: Let ff and two scale-equivariant functions such that Vx E f(x) = g(x). First of all, for any scale-equivariant function h, h(O,)) = h(2 0n,) 2h,,,,, hence ((,,,)) = Om Therefore, f(O,,) = g(O,)) = Om-
Let x E R {0,)) As |all E S, we have f( [ a (mm) ) = |
1 f(x) = |l] g(x)
f(x) g(x). Finally, f = g.
||>||| V
i + O, Note that Vo E S, f(x) = ys-
Existence: Let f x E R.
Om Let R. and X E R. If o O,, f(xx)
otherwise
|| y M
y (f) and
x On f(Ax) Om \z)),, hence scale-equivariant.
Shift ehifrrequivariance:
Let f and y two shift-equivariant functions such that Vr E Span(1,,) f(x) g(x). Let E RR By orthogonal decomposition of R. into Span II,) and Span (1,):
Uniqueness:
31 (21, x2) E Span(1,,)* x Span(1.), x 2 222
Then, - = + x2) = (z1) 02 = g(01) + + = g(1 + x2) = g(x).
Existence: Let f 2 IR" + yr, + 22, where x = + 22 is the unique decomposition
such that di E Span(1,1)) 1, and 22 E (1,) Note that Vo E (1,) , (xx = yz- Let x E R and μ E TR. f(r + (1) yer + x2 p (x) + l as if x orthogonally de- composes into 21 + 22 with 2ı Span(1,,) and xa Span(1..). then a orthogonally decomposes into 21 + (x2 + plm) f is then shift-equivarian.
Normaltaationeeuiiirrinne::
functions such that Vr E S Span(1,,)* f(x)) = g(x). First, as I and y are fortiori scale-equivariant, f(0,,)
Uniqueness: Let f and g two normalization-eqquvaae
g(O,)= = 0m Let æ e IR" \ (O.). By orthogonal decomposition of g" into Span(1,.)
and Span(1n):
(01, x2) E Span(1,)) x Span(1n), x = σ1 22
If On, f(x) f (0,, + 222) f(.) + 22 0m + 22 Z2. Likewise, g(x) F2. hence f(x) = g(z). Else, 21 + 0n f() = f(1 = f(x1) + x2 = || flf( ||2
21
D2 |و #1
#t E S n Span Span ||= + x22 = g(01) + x2 = g(01 + x2) = g(x). as
||ا|||
Finally, f = g.
|-||||
x + On + 32
Existence: Let f R
otherwise where I = I1 + 22
x2
is the unique decomposition such that T1 Span(1,)' and xa Span(1.). Note that Vz E S n Span(1,)) , Ve. Let x R", X R and E R. decomposes
16

---

orthogonally into 21 + 22 with E Span(1,)) ! and 22 E Span(1n), and we have +u)), where λz₁ (Aæz + p)) is the orthogonal decomposition of λæ + μ into Span(1,)* and Span(1n).
f(Az +A)++)) = f(Az, + (Azz
x On, then Arι = O, and f(Ar + (1) = Xx + p= == Af(z) +
Else, if 21 * On, then λzι o On, and f(Az + (4) = | 1l
y - + (Aaz + 1t)
X(1221 y m + +22 + μ (z(x) + μ. Finally, is cemmalliatton-equivariian
Lemma 2 (Operations preserving equivariance)
Proof: Let E R", A R and μ E R.
If f and are both scale-equivariant (f Xf (z) and if they are both shift-equivariant, (ʃ g) + - f(g(s p))
f(Ag(z)) =Xf(()))))((())) =
[((()) + f(g(a)) + k= (f (x) + n.
Let h I + (f(z) =Tx) If f and g are both scale-equivariant, h(Xx)
(f(Ax) g(xx) =
=(x=)))==((00) (Af(x)] >g(x) Xh(x)) and if they are both shift-equivariant,
+(
= =f(x)T + g(z)T + = h(z) +
Let R and : (1 (1-t)f tt. f and are both scale-equivarian, h(Az) =
(1 - (t)f(((a))) + tg((x) = (1-)))(())++ +(g(())
^((1 ft) + tg(x)) (h(x)
and if they are both shift-equivarinn
h(x +
+ R) (1 t)f(r 14) tg(r 11)
(1- (f(x)) μ) t(g(2) + +1) = -1- (1=
+ + tu h(z) +
Proposition
1
Proof: L < <bb E R, f R" H HM . IE and μ I.
We have Ta,b(Ar + 11) (b ab bcty-min(\cty)
(b a) max(z) (2(zz)-mi)) min (x)
z-min(a)
max p min()x+a)
(iee Ta,b is normalization iiniiomnnniaaiatt..... Tap denotes the inverse transformation intricately linked to the
input x of Ta.s (note that this is an improper notation as Ta,s is not bijective). Thus, if a is the input min(z).
of Ta. then Tap y > (max(x) - min(
max(Xe + 11) min(Ar 1)) ((f Ta, + 4) a) + min(Ax p))
(Tab fo Ta,) + p)
b-
max( (x) - min(s)
((f Tab)() + u) - aa) Amin(x) + n,
b b-0
max(xx) min(x)
((f Ta,))()) -a) a)+mmimg +min(s)) u,
b
A(T o (x)))() +
Finally, Tad o fo Ta. is normalization-equuvvriinn.
Proposition 2
Proof: Let id x R + be the identity function. id is coommalization-eqqiivvite function so {a + x) C NE(1). Reciprocally, let e NE(1). By scale-equivariance, f() = f(2 x 0) = 2f) hence f() = O. By shift-equivariance, Vz E R, fx) f(x + 0) f(0) xz, hence, = id. Finally, NE((1) S {ar + ). hence NF(1)= = { + 2) Note that is coherent with Lemma 1 which states that f is entirely determined by its values on S n Span(1n)+ which reduces to the empty set for n = 1.
Let F = (21, 02) + A
if S x2 else B
A. B e E stt. Al B1, 12
82
x2
and f E F. Let x E R22 A >
0 and
E R.
f(A A(xx + p)
if Aσ + S Aw + p
XAr + μ ABx +
if S x2 otherwise
\s())+
B(x + 4)
otherwise
hence f NE(2).
17

---

Reciprocally, let f E NE(2). For n = 2 and when considering the Euclidean distance, S n Span (1a) = f- n² with u = (-1/ 12111221222 Let
1 U2 - f(u)1
f(u)1 - U1
U2 + f(-u)1 -f(-u)1 U1 U2 + f( u12 -f(- U1
and B
U2 f(u)2
f(u)2 u1
1/2
U2
un
Let g (21, 02) H A
z1 5 22 else
I1 122
We have g E F since A12 = B12 = 12
22)
(in particular g is then normalizatton-equivrran) and Vo E {-u, (()g((()) f()). According to Lemma g = f. hence f E F. Finally, NE(2) C F, hence NE(2) = F.
Proposition
Proof: f is composed of three types of building blocks of the following form:
affine convolutions: ae x R" + Or with θ Rmxx subject to O1, 1m
sort pooling nonlinearities: sortpool: R" + R,"
max pooling layers: maxpool: : IR" + R. with m < n,
which are assembled using:
function compositions: comp(S.g) + fog.
skip connections: skip(f. + + (f(x) g(x)))T),
affine residual connections: ares; g) - (1 t)f + tg with & E R.
Note that the rows of θ in ae encode the convolution kernels in CNN and the trainable parameters, denoted by 0, are only composed of matrices θ and scalars t. Moreover, note that average pooling layers are nothing else than affine convolutions with fixed parameters.
Since de, sortpool and maxpool are caamallzaation-qqiiaarii functions, Lemma 2 states that the resulting function f& is also normalization cetpatttiinneiiviviiiiia.a. Moreover, since they are continuous and the assembling operators preserve continuity, JNN is continuous. Then, for given input E R", (x) = (())., where π an operator acting on matrix θ by permuting its rows (note that the permutation is both dependent on and Q). Therefore, applying a pattern "conv affine + sortpool" simply amounts locally to a linear transformation. Moreover, since applying max pooling layer amounts removing some rows from matrix O the local linear behavior is preserved. Thus, as the nonlinearities of f are exclusively brought by sort pooling patterns (and possibly max pooling layers), S is actually locally linear. In other words, SN is piecewise-linear. Moreover, there is finite number (although high) of possible permutations (and
we have (sortpool a (.)(
possibly eliminations) of the rows of all matrices θ, f has finitely many pieces. Finally, on each piece represented by the vector yr. f (y) A y. It remains to prove that 1 = 1m- But this property is easily obtained by noticing that, subject to dimensional compatibility on matrices θ:
11,- = 1m + r()),, = 1m ("conv affine + sortpool"),
01, = 1m = 1())1, = 1, ("conv affine + maxpool") where p removes some rows,
θ 1, = 1m and 0,1m = 1; = , 1n = 1, (composition),
(0)
θ 1ng 1m and O₂ 12 1 1m +
1 Inytng -m (skip connection).
(2
1 1, = Im and 121, = 1m + (1 - ()-1), 1, +122,, = 1m (affine residual connection).
Thus, the affine combinations are preserved all along the layers of JNB In the end,
(N) AgT v. with A guxn such that Ag' 1, 1m
18

---

C.2 Examples of normalization-qquivaaaan conventional denoisers
Noise-reduction filters: All linear smoothing filters can be put under the form fe : x E R Or with è E R= (the rows of è encode the convolution kernel). Obviously, fe is always scale- equivariant, whatever the filter è. As for the snfff-eqqiarrianceee simple calculation shows that:
I + Or is shiff-equivariimt + Vx E R", Vp R, 0(x + μl,) Oz + plm + 11, 1m
Since the sum of the coefficients of Gaussian kernel and averaging kernel is one, have 11, = 1m. hence these linear filters are vormalization-eenivvrria The median filter is also
normudizationneqqiivriint because median(Xæ p)) À median(a)) + J for À E R and μ R.
Patch-based denoising:
NLM [ Assuming that the smoothing parameter h. is proportional to i.e. h = oa, we have
[p(Au, -(Av +ull (2 -P
62
hence the aggregation weights
normalization-inee invariannt Then,
(N + μ. )o);
1
Σ
(m(oc) plupl ha
(Ayr + 4) with W,
[ptar) p(u
W
91 E0(H)
My (((
1
Σ
1P( 42
Ayj
+
82
W
W
By ()E(((
MyER(y))
P
Σ
Di
+
W 85 (0(
A/NLM(U,
Finally, /NLM is a watmmmaliiaaioo-qqivvviiio function.
NL-Ridge (18] The matck-n procedure at the heart of NL-Ridge is normalization-invarin as it is based on comparisons of the l norm of the difference of image patches. For each noisy patch group, a.k.a. similarity matrix, E k composed of & vectorized similar patches of size n, the optimal weights O' E & in the Ey risk sense, are computed such that Y is as close as possible to the (unknown) clean patch group X Rn The two successive minimization problems approximating θ under affine constraints C = {e E R θ 1 = 1x} can be put under the form:
Q 1x (1
θ = arg min tr BEC
O Q0 (oT@@+CO)
1 no²
1Q
with == Y or Q X + no for the first and second step. respectively (X the patch group estimate obtained after the first step), C' = no I Q and where tr denotes the trace operator. Depending on the step, we have:
| - Y12 + 2no ir (+) + const.
2tr ( (eT@@+c0)
or
|| x1 + no | + const
where ||-I|= is the Frobenius norm. But, for any Z E B and any function / O BR + R.
[I(AZ + #) (AZ + p) + m(A)) X2 (lze 211 + ho))(()))
assuming that θ 1, = 1 Therefore, the aggregation weights θ* are normalization iimaaiiiation-nnariann and (XY + #) 2 + μ. Finally, NL-Ridge with affine constraints encodes normalization- equivariant function.
TV denoising:
Lety E R X Rt and μ E R. Let = arg min | s.t.
11 x = no²
ZER"
be the solution of TV B77
fry + p. A0) arg min |æ||rV s.t. + 21B =2 n2)o)²
19

---

arg min
2 p.
s.t.
22
y
2' P
X X
X
X
sER"
TV
I
arg min zER"
s.t.
110
Ax p.
X/T + j..
Finally, frV is a normalizzation-equiivrraaa function.
D
Additional results
Table 5: Real-world image denoising on Darmstadt Noise Dataset (DND) with raw data and variance-
stabilizing transformation (VST). All "non-blind" methods were trained solely on synthetie white
Gaussian noise.
Quality metric
PSNR f
SSIM f
Quality metric
PSNR f
SSIM f
ordinary
47.58
0.9762
ordinary
47.37
0.9754
DRUNet
scale-equiv
47.59
0.9763
FDeCNN
scale-equiy
47.31
0.9754
norm-equiv
47.57
0.9762
norm-equiv
47.35
0.9753
32
Ontpur PSNR
28
24
20
20
cedinary
scale-equiv
16
nurm equivv
12
12
Meniity
16
24
12
16
220
24
25
112
12
16
201
24
Inpui
Figure 6: Comparison of the performance of our
FDnCNN [47 with its
scale-equivane equiiarauuvaii and ordinary counterparts for Gaussian denoising on the Set12 dataset. The vertical blue line indicates the unique noise level on which the networks were trained exclusively (from left to right: a = 50, a = and 10). all cases, normalization-eequivviaa networks generalize much more robustly beyond the training noise level.
20

---

Ouigut PSNR
28
24
ordinary
scale-oquii (relu)
20
norm-oquiv
scale-equaie
wale-equii (sorppool)
adcmtity
Inpui PSNR
Teput PSNR
(a) Uniform
(b) Laplace
38
16
Input PSNR
Quality factor
(c) Rayleigh
(d) JPEG
Figure 7: Comparison of the performance of our normalization- -atiiiiiiiiiiiiiiiintt alternative with its scale- equivariant (under three different forms) and ordinary counterparts for different types of additive noises the Set12 dataset with "blind" FDrCNN architecture. The vertical blue line indicates the unique noise level on which the "blind" networks were trained exclusively. Note that JPEG noise (d) (i.e. JPEG artifacts) is treated with the networks of (a), assimilating JPEG noise with uniform noise (similar outcomes can be achieved when using the networks specialized for Laplace noise (b)).
Table 6: Test enror of different variants of the same VGG8b architecture for image classification. Note that norm-equin equivaraiie with respect to the classification vector and becomes norm invariance after label selection via argmax. See (34) for details about architecture, datasets and training. KuzushijiMMNIST Fashion-MNN MNIIST
Dataset
MNIST
ordinary 0.26%
1.53%
4.53%
VGG8b
scale-equiv
0.37%
1.52%
5.01% %
norm-equiv
0.44%
2.78%
6.57%
Note that for norm-equiv variants, learning rate is initialized to 3e-5 instead of 5e-4 and dropout rate is halved.
21

---

NOISY
SCALE-CQUIVARIANT
NORMALZAATON-IQUUAAAE
Denoised
Adaptive filters
Denoised
Adaptive filters
PSNR: 27,53 DR
PSNR:
27.54 DB
Σ 1.00
I=1-
PSNR
PSNR:
3 1.04
PSNR:
32.18
32.14
25
0.99
PSNR. 34.15 DB
PSNR: 33.67 DB
0.99
PSNR: 36.63 DB
Σ' 0.99
Figure 8: Visual comparisons of the generalization capabilities of seale- cnule--quivariant FDnCNN [47 (left) and its normalization-equvarrio
counterpart (right) for Gaussian noise. Both networks were trained for Gaussian noise at noise level 25 exclusively. The adaptive filters (rows of A in Prop. 3) are indicated for two particular pixels as well as the sum of their coefficients (note that some weights negative, indicated in red). The scale-equivariant network tends to excessively smooth out the image when evaluated at a lower noise level, whereas the more adaptable and considers the underlying texture greater extent.
network
ESNE, Sn
PSNE,
PSNR: 24.45 DB
PSNR: 24.14
(a)
fe (V) (v)
35 (y/3)
NE (U)
PSNR: 22.67
PSNR: 31.69 DB
PSNR: DB
PSNR: 32.07 DB
(b)
y (x E [0.3, 1
SB (v)
1 0.3) + 0.3
NE (v)
Figure 9: Denoising results for example images of the form = + de with o 25/255 and a E [o, 1]m, by FDrCNN (47 specialized for noise level a only. Here, Iee S and S denote the ordinary, scale-equivariant and hormalizatioon-euuvvrrie variants, respectively.
22

---

Noisy
Ordinary
Scale-equiv
Normaliaatonneequiv
DRUNet FDaCNN
PSNN 35.64
PSNK 35.68 DB
PSNR 35.65 DB
DRUNet FDDCNN
PSNR: 30.09 DB
PSNR: 30. 16 DR
PSNR: 30. DB
PSNR: 29, DB
PSNR: 29 DB
PSNR: 29, DB
PSNR: 28 DB
PSNR: 28.82
PS 28. 28.85
DRUNet FDDCNN
PSNR: 27.83 DB
PSNR: 27.81/DB
PSNR: 27. DB
Figure 10: Qualitative comparison of image denoising results with synthetic white Gaussian noise for
"non "non-blind" blind" models. Regardless of the variant model, the denoising results are visually similar.
23


Reviews:
[{"review_id": "NbAxmopEc9", "reviewer": "Reviewer_atAH", "summary": "This work proposes a normalization equivariant deep convolutional neural network for image denoising. The authors provide a mathematical construction of a normalization equivariant neural network and propose sorting nonlinearity. The proposed nonlinearity shows comparable performance with the baseline and shows robustness against different noise levels.", "questions": "1. The work [1] shows a better generalization of scale equivariant neural networks on different noise scales compared to what is shown here. What might be different here that lead to this difference?\n2. Line 282, “but this is much less true for λ < 1 (stretching) for the reason that it may exceed the bounds of the interval [0, 1]. This explains why scale-equivariant functions do not generalize well to noise levels lower than their training one”: how λ < 1 related to not generalizing to lower noise levels?\n\n1.ROBUST AND INTERPRETABLE BLIND IMAGE\nDENOISING VIA BIAS-FREE CONVOLUTIONAL\nNEURAL NETWORKS", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The work provides a sound mathematical formulation for the Normalization equivariant Neural Network.\n2. The proposed model is robust across different noise level\n3. The paper is well written, nicely explained", "weaknesses": "1. The effect of newly introduced sorting nonlinearity will be better understood through an ablation study, i.e., reporting the performance of a model with only affine convolution and Relu.\n2. The effect of shift-equivariance is not well described and demonstrated. Even though it is addressed in the last section, its’ benefit over the scale-equivariant model is not well demonstrated and motivated. Given the novelty of the work is providing shift equivariance (in addition to scale-equivariance), it deserves better motivation and justification.\n3. The proposed non-linearity performs point-wise swapping of the output of different filters at some points. So, it distorts the learned features. It has a significant possibility of affecting the performance of other computer vision tasks (e.g., segmentation), limiting its use only to denoising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. First of all, we would like to stress that our aim is not to introduce new layers that might perform better than traditional ones, but only to observe that it is possible to use the proposed substitutes as a viable alternative that allows normalization-equivariance, without degrading performance on already observed noise levels, which is demonstrated in Table 2 of the paper. In fact, our point is simply that affine convolutions and ordinary convolutions are interchangeable, as are sort pooling nonlinearities with ReLU. However, the benefits in terms on performance of the proposed architectural modifications are revealed at unseen noise levels, as demonstrated in Figure 4 of the paper. If, however, your question was: “Are affine convolutions alone or channel-wise sort-pooling nonlinearities alone sufficient to generalize to unseen noise levels?”, the answer is negative (see Figure 1 of attached pdf). The reason is because only the strict association of affine convolutions with sort-pooling nonlinearities allows normalization-equivariance and it is this property that allows increased robustness to unseen noise levels as theoretically proven (see line 268). \n\nW2-W3. Regarding the limitations of the paper, please refer to the global response to reviewers. Note that premilinary results suggest that normalization-equivariant networks (implemented using channel-wise sort pooling layers) are applicable to image classification (see Table 2 of attached pdf).\n\nQ1. “What might be different here that lead to this difference?”\n\nIn our experiments, we trained all “blind” models exclusively on a single noise level (either $\\sigma=10, 25$ or $50$), contrary to [1] that considers whole ranges of noise levels, which may explain this difference. Moreover, in their experiments they mainly train the “blind” models on ranges of the form $[0, \\sigma_{\\operatorname{max}}]$ ($\\textit{e.g.}$, the noise ranges are $[0, 10]$, $[0, 30]$, $[0, 55]$ and $[0, 100]$ in the code they provided). Therefore, the drop of performance of scale-equivariant networks at lower noise levels cannot be observed ($\\textit{e.g.}$, see Figure 11 of [1]). Finally, the drop of performance for scale-equivariant networks is even more pronounced with the DRUNet architecture for some reason, but this architecture appeared after the publication of [1].\n\nQ2. “How $\\lambda < 1$ related to not generalizing to lower noise levels?”\n\nIn the case of a scale-equivariant network $f_\\theta$, we have, adopting the same notations as in subsection 5.2:   $f_\\theta(x + \\lambda \\varepsilon) = \\lambda f_\\theta(x/\\lambda + \\varepsilon)$. Hence, denoising the image patch $x + \\lambda \\varepsilon$ amounts to denoising $x/\\lambda + \\varepsilon$. Interestingly, $x/\\lambda + \\varepsilon$ is a patch corrupted by the exact type of noise on which $f_\\theta$ has been trained for. Therefore, as long as $x/\\lambda$ is in $\\mathcal{D} \\subset [0, 1]^n$, efficient denoising can be guaranteed. In general, $x \\in \\mathcal{D} \\Rightarrow x/\\lambda \\in \\mathcal{D}$ for $\\lambda > 1$ (think of a flat patch $x$ for example) but this is much less true for $\\lambda < 1$ for the reason that it may exceed the bounds of the training interval $[0, 1]$. Indeed, if $x=1_n$ for example (flat white patch) and $\\lambda=0.5$ (half the amount of noise for which $f_\\theta$ has been trained), then $x/\\lambda = 2_n$ which is not in  $[0, 1]^n$, therefore there is no guarantee as to the result of the denoising of patch $(x/\\lambda + \\varepsilon)$ by $f_\\theta$. But if $f_\\theta$ is in addition shift-equivariant (i.e., normalization-equivariant), then $f_\\theta(x/\\lambda + \\varepsilon) = f_\\theta((x/\\lambda - 1_n) + \\varepsilon) + 1_n$, which amounts to denoising the clean patch $(x/\\lambda - 1_n) = 2_n - 1_n = 1_n$ corrupted by the exact type of noise on which $f_\\theta$ has been trained for. This time $(x/\\lambda - 1_n)$ is in $\\mathcal{D} \\subset [0, 1]^n$ and we have then a guarantee as to the result of the denoising. This behavior on flat patches was confirmed experimentally.\n"}, {"review_id": "tjKyTMFzaZ", "reviewer": "Reviewer_dbvT", "summary": "In this paper, the authors propose an architectural design which guarantees shift and scale equivariance. The proposed method is called normalization equivariance in which scale equivariance is inherited from prior work by removing layers biases and shift equivariance is achieved through affine-constrained convolutions along with channel-wise sort pool (instead of ReLUs). They adopt two popular architectures to become normalization equivariance, and show that their variant outperforms both unconstrained, and scale-equivariance models in generalizing outside of the training range.  ", "questions": "It would be useful to see comparison of computational cost for training. Although sort pooling is restricted to n=2, I assume it is still most costly than using ReLUs. A numeral comparison can show the cost for gaining normalization-equivariance. \n\nIt wasn't clear from the text how the affine convolution constraint was enforced during training. It would be useful to explain that in the main text since it is one of the two main design contributions. ", "limitations": "No major limitations or potential negative societal impacts. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is written very clearly and has excellent flow. It is technically strong and propositions and claims are backed up both theoretically and empirically. The idea of enforcing shift equivariance by design is original and the authors nicely elucidate its necessity and the positive impact it has. In my opinion, this work offers a significant contribution in the area of useful inductive biases and also understanding neural networks trained for denoising. ", "weaknesses": "Generally the paper is written very well and all sections are carefully and throughly explained. Here are a few minor issues: \n\nI believe rotation needs to generally be an invariance (as opposed to equivariance) with respect to labels in image classification (line 52)\n\nProof for proposition 2 can be included in the appendix. \n\nThe derivation of eq 6 (from line 178 onward) is not clear. It would be useful if the authors re-write that paragraph. \n\nNumber of possible states is still $2^S$ if ReLU activations are used instead of sorting (line 222)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Minor issues:\n\n- “I believe rotation needs to generally be an invariance (as opposed to equivariance) with respect to labels in image classification (line 52)”. \n\nThank you for pointing this out. We agree with you, this was an inaccuracy and we removed it from the manuscript.\n\n- “Proof for proposition 2 can be included in the appendix.” \n\nThe proof can be found from line 112 to line 122 in the supplementary material, even though it was not mentioned in the manuscript. For the sake of clarity, we have added “(see proof in Supplementary Material)” to line 78 of the manuscript.\n\n- “The derivation of eq 6 (from line 178 onward) is not clear. It would be useful if the authors re-write that paragraph.” \n\nWe assume you are referring to the transition from the arbitrarily setting $\\varphi(-u) = \\varphi(u) = u$ to the deduction that $\\varphi$ is the sorting function of $\\mathbb{R}^2$. For greater clarity, we have added the following explanation to line 185: “Indeed, the sorting function (6) is a normalization-equivariant function by concatenation (see Prop. 2) of two normalization-equivariant functions from $\\mathbb{R}^2$ to $\\mathbb{R}$, namely the functions min and max. Moreover, it does verify $\\varphi(-u) = \\varphi(u) = u$ and this is the only one according to Prop. 1. Note that another choice of arbitrary setting would have entirely identified a different normalization-equivariant function of $\\mathbb{R}^2$, possibly leading to an unintended linear function or to a more complicated nonlinear function.” To be perfectly exact, from the proof of the existence (line 97 of the Supplementary Material), one can show that all normalization-equivariant functions of $\\mathbb{R}^2$ to $\\mathbb{R}^2$ are under the form: $\\varphi: (x_1, x_2) \\mapsto \\frac{1}{\\sqrt{2}} | x_1 - x_2 | y_{\\operatorname{sign}(x_1 - x_2)  u} + \\frac{1}{2}  (x_1+x_2, x_1+x_2)$  where $u=(-1/ \\sqrt{2}, 1/ \\sqrt{2})$ and $y_{-u} \\in \\mathbb{R}^2$ and $y_u \\in \\mathbb{R}^2$ are to be chosen. Clearly, the sorting function is the simplest nonlinear function of this form, obtained for $y_{-u}=y_u=u$.\n\n- “Number of possible states is still $2^{S}$ if ReLU activations are used instead of sorting (line 222)”\n\nUnless we are mistaken, using ReLU activation functions instead of sort pooling nonlinearities generally allows for a greater number of pieces in the resulting piecewise-linear functions. It can be seen by considering the following rudimentary feedforward neural network $f_\\theta : (x, y) \\in \\mathbb{R}^2 \\mapsto  \\Theta_2 \\varphi\\left(\\Theta_1 (x \\quad y)^\\top \\right)$ with $\\varphi$ the sorting function (6), $\\Theta_1\\in\\mathbb{R}^{2\\times2}$ and $\\Theta_2\\in\\mathbb{R}^{1\\times2}$. In this case, $S=1$. An upper bound for the total number of pieces is then $2^{S}=2$ which is attained for $\\Theta_1=I_2$ and $\\Theta_2=(-1\\quad1)$ but not for $\\Theta_2=(0\\quad0)$, in which case $f_\\theta$ is the zero function (only one piece). On the other hand, if $\\varphi$ is the ReLU function, then an upper bound is $2^{2S}=4$ which is attained for the same parameters $\\Theta_1=I_2$ and $\\Theta_2=(-1\\quad1)$. (a software can help for visualization) \n\nQuestions:\n\n- “It would be useful to see comparison of computational cost for training. Although sort pooling is restricted to $n=2$, I assume it is still most costly than using ReLUs. A numeral comparison can show the cost for gaining normalization-equivariance.”\n\nYou are right, it is more costly to gain normalization-equivariance. We have compared the computational costs of different variants for training and inference (see Table 3 of attached pdf). Interestingly, the computational cost for training is much more sensitive to the “affine mode” (involving affine-constrained convolutions with reflect padding and affine residual connection) than to sort pooling nonlinearities, while it is the opposite for inference. All in all, for gaining normalization-equivariance, the learning and inference time is almost doubled for the DRUnet architecture on a Quadro RTX 6000 GPU. Note however that we do not claim to have the most optimized implementation and there is probably room for improvement.\n \n- “It wasn't clear from the text how the affine convolution constraint was enforced during training. It would be useful to explain that in the main text since it is one of the two main design contributions.”\n\nWe understand your point. Due to space limitation, we preferred to defer the description of the practical implementation in the supplementary material. However, if the article is accepted and if any space remains, we plan to raise the issue in the main text."}, {"review_id": "QlK9wge2QY", "reviewer": "Reviewer_6aGn", "summary": "The authors of this paper propose an architectural change in neural networks, in a way that enforces the normalization equivariance property. The authors demonstrate how networks that satisfy this property can be used in the setting of image denoising.", "questions": "I would be interested to know if the authors have considered applying their architectural change not when training the models from scratch, but by altering select layers of a pretrained model (for example, by including the sorting layer as part of a pretrained architecture). If something like this is possible, then it would alleviate some of the issues I mentioned above, allowing the method to be more readily applied to a larger set of problems.", "limitations": "I believe the authors should better clarify in the paper how currently their method is applicable mainly to image denoising as it stands (even though in principle normalization equivariance may be helpful in other tasks as well).\n\nRegarding negative societal impact, I do not see any immediate issues arising from this work. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written overall. The intended goal of the architectural changes proposed is clear, and how these changes achieve this goal is also made explicit in the paper. The proposed architecture is also novel, as far as I am aware.\n\n- There is a clear benefit derived from the architectural changes proposed by the authors. The intended effect of normalization equivariance is to make the output of the network disregard potential alterations in the scale of the noise added to the model. This is made clear via Figure 4, which precisely shows that the alterations to the architecture achieve the intended effect. The authors also adequately explain in the paper how this equivariance with respect to this normalization makes their models robust to the amount of noise added to the input, namely in that adding different amounts of noise only corresponds to renormalization.\n\n- The idea that the authors propose regarding sort pool between channels is an interesting and novel way of pooling information from different channels in CNNs. This seems to be a novel inclusion in the architecture of a neural network, and I am interested in its potential applications beyond normalization equivariance.\n", "weaknesses": "My main issues with this paper lie with the somewhat limited setting of the experiments performed to validate the abilities of the proposed architecture.\n\n- While interesting, the target problem for this paper is somewhat limited. The only application considered in this paper is that of image denoising, which is arguably a very limited setting. I believe that demonstrating the effects of enforcing normalization equivariance in other problems (such as image classification or other types of reconstruction e.g. superresolution) would improve the paper.\n\n- Even in the context of denoising, the authors of this paper only consider the setting of Gaussian noise being applied to an image. While generally accepted as a valid setting for denoising, It would be interesting to see if the normalization equivariance was able to help in the case where different types of noise are applied (e.g. salt-pepper noise, random masking of pixels, and so on).\n\n- Regarding baselines, the authors compare against techniques which use regular feedforward neural networks. While the comparisons support the points made by the authors, I believe that it would be interesting to compare against techniques that employ generative networks to solve the denoising problems e.g. via DPS [A].\n\nReference:\n\n[A] Chung et al., “Diffusion Posterior Sampling for General Noisy Inverse Problems”, ICLR 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Regarding your reservations about paper limitations, please refer to the global response to reviewers.\n\n- “the authors compare against techniques which use regular feedforward neural networks. While the comparisons support the points made by the authors, I believe that it would be interesting to compare against techniques that employ generative networks to solve the denoising problems e.g. via DPS [A].” \n\nTo the best of our knowledge, the methods that employ generative networks such as [A] in order to solve inverse problems are superior to regular feedforward neural networks only in the case of severe degradation models, for which the usual principle of empirical risk minimization is helpless to recover realistic attributes (e.g., full face inpainting, extreme superresolution, or very high noise...). Posterior sampling is then an interesting alternative. As far as we are aware, the SOTA methods for non-extreme noise removal are all regular feedforward neural networks.\n\n- “I would be interested to know if the authors have considered applying their architectural change not when training the models from scratch, but by altering select layers of a pretrained model (for example, by including the sorting layer as part of a pretrained architecture). If something like this is possible, then it would alleviate some of the issues I mentioned above, allowing the method to be more readily applied to a larger set of problems.” \n\nThis is indeed an interesting line of research. Preliminary results suggest that no gain in terms of training time can be achieved by retaining all the pre-trained weights. However, further investigations are needed to find out if it is possible to keep only carefully selected weights and it could be an interesting line of research for future work.\n"}, {"review_id": "RBazcP0AKr", "reviewer": "Reviewer_RNG3", "summary": "The paper proposes normalization-equivariant CNNs, in which the convolutional kernels are forced to sum to 1 and the activation sorts pairs of neurons. This is shown to perform similarly to other models on image denoising when trained with the same noise levels, but generalizes much better to unseen noise levels. \n", "questions": "Minor:\nTypically in image processing, for an image f: Z^2 -> R, \"scale\" refers to resizing (f'(x) = f(ax)) and \"shift\" refers to moving pixels (f'(x) = f(x+a), in the sense that convolution is shift-equivariant). In the submission, it seems that \"scale\" means multiplying all pixel values by a constant and shift means adding a constant to all pixels. I recommend clarifying the difference to avoid confusion.\n", "limitations": "W1) I couldn't find a discussion of limitations in the paper and I think this is a major weakness. Table 1 in the supplement shows that the norm-equivariant models need to be trained for about twice the number of iterations of the baselines and with MSE instead of L1 loss. For fairness, I think the ordinary and scale-eq baselines should be also evaluated under the same conditions and this possible limitation should be reported in the main text. It seems that the method is also incompatible with other common used layers such as max-pooling and batch normalization, which should also be discussed.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1) The paper presents a simple and effective method to improve generalization in image denoising CNNs.\n\nS2) Results are strong, particularly the ones shown in Fig 4.\n\nS3) Section 5.2 presents insights about previously observed phenomena of bias-free/scale-invariant models performing well on denoising, and explains why the normalization-equivariance proposed is potentially superior.\n", "weaknesses": "W2) Table 2 reports results on the non-blind experiments but I couldn't find whether the results in Fig 4 and 5 are blind or non-blind. I assume they are blind and since the models were trained on a single noise level. If they are not and if the noise level is known at inference time, I think the significance of the results is diminished since with a known noise level the ordinary CNNs can be perform quite well as shown in Fig. 5. Please clarify.\n  \nW3) Fig 2 implies that the kernel constraint for convolutional kernels is along the channel axis, but it seems that it should be along both channel and spatial dimensions to preserve shift-equivariance. The text does not seem to specify it for the convolutional kernel since Eq 5 refers to a linear layer. Please clarify and include a precise definition of the convolutional kernel between two layers of shape H x W x C. \n\nW4) Some recent methods of denoising are missed [1, 2]. Zamir et [1] seems to report slightly better results than the baselines on denoising on the same datasets and also evaluate on different datasets. This is not a dealbreaker but the submission would be made stronger when evaluating on more datasets since the ones used are quite small.\n\nReferences:\n\n[1] Zamir et al, \"Restormer: Efficient Transformer for High-Resolution Image Restoration\", CVPR'22.\n\n[2] Chen et al, \"Simple Baselines for Image Restoration\", ECCV'22.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W2. They are “blind” indeed since the models were trained on a single noise level (and the noise level is not known at inference time). For the sake of clarity, we will explicitly mention it in the legend of Figures 4 and 5.\n\nW3. Thank you for pointing this out. The transition from a dense layer (5) to a convolutional layer is explained in the paragraph on line 157, perhaps too quickly. The key observation is that a convolution from an input layer of size $H \\times W \\times C$ to an output layer of size $H' \\times W' \\times C'$ can always be represented by a dense layer by vectorizing the input and output layers: the elements of the $C'$ convolutional kernels of size $k \\times k \\times C$ each are then stored separately along the rows of the (sparse) transition matrix $\\Theta$ of size $(H' \\times W' \\times C') \\times (H \\times W \\times C)$. Therefore, the sum of the elements of each kernel of size $k \\times k \\times C$ must sum up to $1$ for enforcing the normalization-equivariance property. We will add a clarification of that point in the manuscript.\n\nW4. The methods [1, 2] that you mention are both SOTA transformers (in the sense that they are based on the attention mechanism) in the same way as SwinIR [3] is a SOTA transformer, which we have cited in the manuscript. The performance differences between these three networks [1,2,3] and SOTA fully convolutional network DRUNet [5] are marginal, from our point of view, but, for the sake of completeness, we will add these two references to the manuscript. \n\nHowever, we want to emphasize that the point of our paper is not to introduce a new architecture outperforming its predecessors on some datasets but only to propose architectural modifications to existing neural networks that enforce normalization-equivariance by design, which can be beneficial for robustness against unseen noise levels, in addition to their better conditioning. For the sake of completeness, we have evaluated the three “non-blind” network variants on a supplementary dataset composed of real-world noisy images [4] on which a variance-stabilizing transformation (VST) was applied beforehand and draw the same conclusion: the performance remains the same at already seen noise levels between two different variants (see Table 1 of attached pdf). Please note that the drop of performance on this dataset compared to the methods you mention [1, 2] is mainly due to the fact that the three network variants were trained solely on synthetic white Gaussian noise.\n\nQ1. Thank you for pointing this out. In line 63 of the manuscript, we have added: “Please note that our definition of \"scale\" and \"shift\" may differ from the definition given by some authors in the image processing literature”.\n\nW1. Regarding the limitations of the paper, please refer to the global response to reviewers. Note that our method is compatible with max pooling since the max function is normalization-equivariant (by the way, the proposed sort pooling layer can be implemented using one-dimensional max pooling and min pooling). As for the difference in loss functions for training (MSE / L1), we decided to take the best-performing loss function for each method, as explained in the Supplementary Material. As it happens, normalization-equivariant networks obtain slightly better results with the usual MSE loss (but the L1 loss is also possible). On the other hand, the authors  [5, 6] recommend the L1 loss to achieve better performance for the original networks DRUNet and FDnCNN and we followed strictly their instructions for training. Finally, we trained all models to convergence, as in [5, 6], and it turns out that normalization-equivariant networks need longer training to reach convergence. This specificity will be reported in the Limitation section we plan to write.\n\n[3]  Liang et al., “SwinIR: Image Restoration Using Swin Transformer,” ICCVW'21.\n\n[4] Plötz and Roth, “Benchmarking Denoising Algorithms with Real Photographs,” CVPR'17.\n\n[5] Zhang et al., “Plug-and-play image restoration with deep denoiser prior,” IEEE Trans PAMI 2021. \n\n[6] Zhang et al., “Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising,” IEEE Trans IP 2017."}]

Author rebuttal:
First of all, we want to thank the reviewers for dedicating their valuable time to the evaluation of our paper and for providing us with constructive feedback, which will improve the quality and clarity of the manuscript. We have revised the manuscript, taking into account all the comments received.

Criticism from some reviewers focused on the possibility of applying the proposed normalization-equivariant neural networks to computer vision tasks other than denoising, such as classification, segmentation or super-resolution. Preliminary results on image classification indicate that it is indeed possible (see Table 2 of attached pdf). However, we focused in this paper on the applications of normalization-equivariant networks on image denoising because, beyond their better conditioning that avoids a major source of confusion and misinterpretation linked to normalization, such networks proved to be exceptionally robust against unseen noise levels, which is a remarkable benefit specific to image denoising. Clearly, discovering similar advantages of normalization-equivariance in other computer vision tasks, possibly related to outlier robustness, is an interesting avenue of research for future work.

In the context of image denoising, several reviewers point out that the paper only considers the setting of Gaussian noise for the experimental results, which is the most widely studied image noise model. In fact, it turns out that our theoretical demonstration (line 268 of the paper) of the increased robustness across noise levels is still valid for a wide range of noise types. Indeed, our demonstration remains valid for any additive noise $\varepsilon$ that possesses the scaling property: $\forall \lambda > 0, \lambda \varepsilon$ belongs to the same family of probability distributions as $\varepsilon$ (e.g., Gaussian noise: $\varepsilon \sim \mathcal{N}(0, \sigma^2) \Rightarrow \lambda \varepsilon \sim \mathcal{N}(0, (\lambda\sigma)^2)$, uniform noise $\varepsilon \sim \mathcal{U}(-a, a) \Rightarrow \lambda \varepsilon \sim \mathcal{U}(-\lambda a, \lambda a)$, Laplace noise $\varepsilon \sim \mathcal{L}(0, b) \Rightarrow \lambda \varepsilon \sim \mathcal{N}(0, \lambda b)$ or even Rayleigh noise which is not zero-mean: $\varepsilon \sim \mathcal{R}(\sigma) \Rightarrow \lambda \varepsilon \sim \mathcal{R}(\lambda\sigma)$). By the way, the authors of [1] had already verified the noise generalization capabilities of scale-equivariant networks for uniform noise in addition to Gaussian noise, without fully elucidating why it works. In the attached pdf, we checked experimentally that “blind” normalization-equivariant networks trained on additive uniform, Laplace or Rayleigh noise at a single noise level are much more robust at unseen noise levels than their scale-equivariant and ordinary counterparts. Moreover, we also observed the robustness capabilities of normalization-equivariant networks when testing on a type of noise that is more difficult to characterize, namely JPEG noise, or JPEG artifacts (see Figure 1d of attached pdf).

Finally, we agree with the reviewers that a Limitation section is necessary. If the paper is accepted, we would like to mention that the proposed architectural modifications for enforcing normalization-equivariance do require a longer training for achieving comparable performance with its original counterpart (see Table 3 of attached pdf) , and may be incompatible with some specific network layers such as batch-norm (but compatible with pooling layers, dropout, ...). Moreover, our method has shown its potential mainly to image denoising as it stands (even though in principle normalization-equivariance may be applicable and helpful in other tasks as well).

For further details, please refer to the specific response to each reviewer comment.

[1] S. Mohan et al., “Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks,” ICLR'19.


Meta-review: The paper presents an effective way to bring normalization-equivariance to NN for image denoising, with a sound mathematical formulation. Before rebuttal, several concerns were raised, e.g., the target problem can be limited, as only image denoising was considered; for denoising, only the setting of Gaussian noise is discussed, etc.  The authors did an excellent job during the discussion phase to address most concerns. All reviewers vote for acceptance.  AC recommends accepting the paper.
Decision: Accept (poster)