Title: Normalization-Equivariant Neural Networks with Application to Image Denoising
Venue: NeurIPS 2023
Accepted: yes
Keywords: equivariance

normalization

image denoising

activation functions

ReLU

interpretability

robustness

deep learning

analysis of neural networks
Abstract: In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or  scaling, results in a corresponding change in the system response.  While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element-wise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.

Normalization-Equivariant Neural Networks with Application to Image Denoising

Abstract

In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or scaling, results in a corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied elementwise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalizationequivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.

1 Introduction

Sometimes wrongly confused with the invariance property which designates the characteristic of a function f not to be affected by a specific transformation T applied beforehand, the equivariance property, on the other hand, means that f reacts in accordance with T . Formally, invariance is f ◦ T = f whereas equivariance reads f ◦ T = T ◦ f , where ◦ denotes the function composition operator. Both invariance and equivariance play a crucial role in many areas of study, including physics, computer vision, signal processing and have recently been studied in various settings for deep learning-based models [3, 6, 10, 13, 14, 16, 21, 23, 31, 38, 40, 41, 43].

In this paper, we focus on the equivariance of neural networks f θ to a specific transformation T , namely normalization. Although highly desirable in many applications and in spite of its omnipresence in machine learning, current neural network architectures do not equivary to normalization. With application to image denoising, for which normalization-equivariance is generally guaranteed for a lot of conventional methods [5, 18, 20, 37], we propose a methodology for adapting existing neural networks, and in particular denoising CNNs [8, 30, 46-48], so that normalization-equivariance holds by design. In short, the proposed adaptation is based on two innovations:

affine convolutions: the weights from one layer to each neuron from the next layer, i.e. the convolution kernels in a CNN, are constrained to encode affine combinations of neurons (the sum of the weights is equal to 1 ).

channel-wise sort pooling: all activation functions that apply element-wise, such as the ReLU, are substituted with higher-dimensional nonlinearities, namely two by two sorting along channels that constitutes a fast and efficient normalization-equivariant alternative.

Despite strong architectural constraints, we show that these simple modifications do not degrade performance and, even better, increase robustness to noise levels in image denoising both in practice and in theory.

2 Related Work

A non-exhaustive list of application fields where equivariant neural networks were studied includes graph theory, point cloud analysis and image processing. Indeed, graph neural networks are usually expected to equivary, in the sense that a permutation of the nodes of the input graph should permute the output nodes accordingly. Several specific architectures were investigated to guarantee such a property [3, 21, 38]. In parallel, rotation and translation-equivariant networks for dealing with point cloud data were proposed in a recent line of research [6, 13, 40]. A typical application is the ability for these networks to produce direction vectors consistent with the arbitrary orientation of the input point clouds, thus eliminating the need for data augmentation. Finally, in the domain of image processing, it may be desirable that neural networks produce outputs that equivary with regard to rotations of the input image, whether these outputs are vector fields [31], segmentation maps [41, 43], or even bounding boxes for object tracking [16].

In addition to their better conditioning, equivariant neural networks by design are expected to be more robust to outliers. A spectacular example has been revealed by S. Mohan et al. [33] in the field of image denoising. By simply removing the additive constant ('bias') terms in neural networks with ReLU activation functions, they showed that a much better generalization at noise levels outside the training range was ensured. Although they do not fully elucidate why biases prevent generalization, and their removal allows it, the authors establish some clues that the answer is probably linked to the scale-equivariant property of the resulting encoded function: rescaling the input image by a positive constant value rescales the output by the same amount.

3 Overview of normalization-equivariance

3.1 Definitions and properties of three types of fundamental equivariances

We start with formal definitions of the different types of equivariances studied in this paper. Please note that our definition of 'scale' and 'shift' may differ from the definitions given by some authors in the image processing literature.

Definition 1 A function f : R n ↦→ R m is said to be:

scale-equivariant if ∀ x ∈ R n , ∀ λ ∈ R + ∗ , f ( λx ) = λf ( x ) ,

shift-equivariant if ∀ x ∈ R n , ∀ µ ∈ R , f ( x + µ ) = f ( x ) + µ,

normalization-equivariant if it is both scale-equivariant and shift-equivariant:

where addition with the scalar shift µ is applied element-wise.

Note that the scale-equivariance property is more often referred to as positive homogeneity in pure mathematics. Like linear maps that are completely determined by their values on a basis, the above described equivariant functions are actually entirely characterized by the values their take on specific subsets of R n , as stated by the following lemma (see proof in Appendix C.1).

Lemma 1 (Characterizations) f : R n ↦→ R m is entirely determined by its values on the:

unit sphere S of R n if it is scale-equivariant,

orthogonal complement of Span( 1 n ) , i.e. Span( 1 n ) ⊥ , if it is shift-equivariant,

intersection S ∩ Span( 1 n ) ⊥ if it is normalization-equivariant,

where 1 n denotes the all-ones vector of R n .

Finally, Lemma 2 highlights three basic equivariance-preserving mathematical operations that can be used as building blocks for designing neural network architectures (see proof in Appendix C.1).

Lemma 2 (Operations preserving equivariance) Let f and g be two equivariant functions of the same type (either in scale, shift or normalization). Then, subject to dimensional compatibility, all of the following functions are still equivariant:

f ◦ g ( f composed with g ),

x ↦→ ( f ( x ) ⊤ g ( x ) ⊤ ) ⊤ (concatenation of f and g ),

(1 -t ) f + tg for all t ∈ R (affine combination of f and g ).

3.2 Examples of normalization-equivariant conventional denoisers

A ('blind') denoiser is basically a function f : R n ↦→ R n which, given a noisy image y ∈ R n , tries to map the corresponding noise-free image x ∈ R n . Since scaling up an image by a positive factor λ or adding it up a constant shift µ does not change its contents, it is natural to expect scale and shift equivariance , i.e. normalization equivariance , from the denoising procedure emulated by f . In image denoising, a majority of methods usually assume an additive white Gaussian noise model with variance σ 2 . The corruption model then reads y ∼ N ( x, σ 2 I n ) , where I n denotes the identity matrix of size n , and the noise standard deviation σ > 0 is generally passed as an additional argument to the denoiser ('non-blind' denoising). In this case, the augmented function f : ( y, σ ) ∈ R n × R + ∗ ↦→ R n is said normalization-equivariant if:

as, according to the laws of statistics, λy + µ ∼ N ( λx + µ, ( λσ ) 2 I n ) . In what follows, we give some well-known examples of traditional denoisers that are normalization-equivariant (see proofs in Appendix C.2).

Noise-reduction filters: The most rudimentary methods for image denoising are the smoothing filters, among which we can mention the averaging filter or the Gaussian filter for the linear filters and the median filter which is nonlinear. These elementary 'blind' denoisers all implement a normalization-equivariant function. More generally, one can prove that a linear filter is normalizationequivariant if and only if its coefficients add up to 1 . In others words, normalization-equivariant linear filters process images by affine combinations of pixels.

Patch-based denoising: The popular N(on)-L(ocal) M(eans) algorithm [5] and its variants [12, 20, 27] consist in computing, for each pixel, an average of its neighboring noisy pixels, weighted by the degree of similarity of the patches to which they belong. In other words, they process images by convex combinations of pixels. More precisely, NLM can be defined as:

where y i denotes the i th component of vector y , Ω( y i ) is the set of its neighboring pixels, p ( y i ) represents the vectorized patch centered at y i , and the smoothing parameter h is proportional to σ as proposed by several authors [4, 12, 29]. Defined as such, f NLM is a normalization-equivariant function. More recently, N(on)-L(ocal) Ridge [18] proposes to process images by linear combinations of similar patches and achieves state-of-the-art performance in unsupervised denoising. When restricting the coefficients of the combinations to sum to 1 , that is imposing affine combination constraints, the resulting algorithm encodes a normalization-equivariant function as well.

TV denoising: Total variation (TV) denoising [37] is finally one of the most famous image denoising algorithm, appreciated for its edge-preserving properties. In its original form [37], a TV denoiser is defined as a function f : R n × R + ∗ ↦→ R n that solves the following equality-constrained problem:

where ∥ x ∥ TV := ∥∇ x ∥ 2 is the total variation of x ∈ R n . Defined as such, f TV is a normalizationequivariant function.

Denoising with 'blind' DnCNN

PSNR: 36.89 DB

PSNR: 31.47 DB

Ground truth

(a)

(b)

PSNR: 32.04 DB

PSNR: 35.22 DB

PSNR: 37.93 DB

Noisy

(c)

(d)

1000

2000

3000

4000

5000

6000

3.3 The case of neural networks

Deep learning hides a subtlety about normalization equivariance that deserves to be highlighted. Usually, the weights of neural networks are learned on a training set containing data all normalized to the same arbitrary interval [ a 0 , b 0 ] . This training procedure improves the performance and allows for more stable optimization of the model. At inference, unseen data are processed within the interval [ a 0 , b 0 ] via a a -b linear normalization with a 0 ≤ a < b ≤ b 0 denoted T a,b and defined by:

Note that this transform is actually the unique linear one with positive slope that exactly bounds the output to [ a, b ] . The data is then passed to the trained network and its response is finally returned to the original range via the inverse operator T -1 a,b . This proven pipeline is actually relevant in light of the following proposition.

Proposition 1 ∀ a < b ∈ R , ∀ f : R n ↦→ R m , T -1 a,b ◦ f ◦T a,b is a normalization-equivariant function.

While normalization-equivariance appears to be solved, a question is still remaining: how to choose the hyperparameters a and b for a given function f ? Obviously, a natural choice for neural networks is to take the same parameters a and b as in the learning phase whatever the input image is, i.e. a = a 0 and b = b 0 , but are they really optimal? The answer to this question is generally negative. Figure 1 depicts an example of the phenomenon in image denoising, taken from a real-world application. In this example, the straightforward choice is largely sub-optimal. This suggests that there are always inherent performance leaks for deep neural networks due to the two degrees of freedom induced by the normalization ( i.e. , choice of a and choice of b ). In addition, this poor conditioning can be a source of confusion and misinterpretation in critical applications.

3.4 Categorizing image denoisers

Table 1 summarizes the equivariance properties of several popular denoisers, either conventional [5, 11, 15, 18, 37, 44] or deep learning-based [24, 26, 45-47]. Interestingly, if scale-equivariance is generally guaranteed for traditional denoisers, not all of them are equivariant to shifts. In particular, the widely used algorithms DCT [44] and BM3D [11] are sensitive to offsets, mainly because the hard thresholding function at their core is not shift-equivariant . Regarding the deep-learning-based networks, only DRUNet [46] is insensitive to scale because it is a bias-free convolutional neural

TV NLM NLR DCT BM3D WNNM DnCNN NLRN SwinIR Restormer DRUNet

Scale

✓

✓

✓

✓

✓

✓

✗

✗

✗

✗

✓

Shift

✓

✓

✓

✗

✗

✗

✗

✗

✗

✗

✗

network with only ReLU activation functions [33]. In particular, all transformer models [7, 24, 26, 45], even bias-free, are not scale-equivariant due to their inherent attention-based modules. In the next section, we show how to adapt existing neural architectures to guarantee normalization-equivariance without loss of performance and study the resulting class of parameterized functions ( f θ ) .

4 Design of Normalization-Equivariant Networks

4.1 Affine convolutions

To justify the introduction of a new type of convolutional layers, let us study one of the most basic neural network, namely the linear (parameterized) function f Θ : x ∈ R n ↦→ Θ x , where parameters Θ are a matrix of R m × n . Indeed, f Θ can be interpreted as a dense neural network with no bias, no hidden layer and no activation function. Obviously, f Θ is always scale-equivariant , whatever the weights Θ . As for the shift-equivariance , a simple calculation shows that:

Therefore, f Θ is normalization-equivariant if and only if each row of matrix Θ sums to 1 . In other words, for the normalization-equivariance to hold, the rows of Θ must encode weights of affine combinations. Transposing the demonstration to any convolutional neural network follows from the observation that a convolution from an input layer of size H × W × C to an output layer of size H ′ × W ′ × C ′ can always be represented with a dense connection by vectorizing the input and output layers. The elements of the C ′ convolutional kernels of size k × k × C each are then stored separately along the rows of the (sparse) transition matrix Θ of size ( H ′ × W ′ × C ′ ) × ( H × W × C ) . Therefore, a convolutional layer preserves the normalization-equivariance if and only if the weights of the each convolutional kernel sums to 1 . In the following, we call such convolutional layers 'affine convolutions'.

In order to guarantee the affine constraint on each convolutional kernel throughout the training phase, one possibility is to 'telescope' the circular shifted version of an unconstrained kernel to itself (this way, the sum of the resulting trainable coefficients cancels out) and then add the inverse of the kernel size element-wise as a non-trainable offset. Despite this over-parameterized form (involving an extra degree of freedom), we found this solution to be easier to use in practice. Moreover, it ensures that all coefficients of the affine kernels follow the same law at initialization.

Since normalization-equivariance is preserved through function composition, concatenation and affine combination (see Lemma 2), a (linear) convolutional neural network composed of only affine convolutions with no bias and possibly skip or affine residual connections (trainable affine combination of two layers), is guaranteed to be normalization-equivariant , provided that padding is performed with existing features (reflect, replicate or circular padding for example). Obviously, in their current state, these neural networks are of little interest, as linear functions do not encode best-performing functions for many applications, image denoising being no exception. Nevertheless, based on such networks, we show in the next subsection how to introduce nonlinearities without breaking the normalization-equivariance .

4.2 Channel-wise sort pooling as a normalization-equivariant alternative to ReLU

The first idea that comes to mind is to apply a nonlinear activation function φ : R ↦→ R preserving normalization-equivariance after each affine convolution. In other words, we look for a nonlinear solution φ of the characteristic functional equation of normalization-equivariant functions (see Def. 1) for n = 1 . Unfortunately, according to Prop. 2 (see proof in Appendix C.1 which is based on Lemma 1), the unique solution is the identity function which is linear. Therefore, activation functions that apply element-wise are to be excluded.

Channel-wise

convolution

sort pooling

k

H

Two by two sorting

k

in all non-overlapping pools:

M(

M

1

Proposition 2 Let NE( n ) be the set of normalization-equivariant functions from R n to R n .

NE(1) = { x ↦→ x } and

̸

To find interesting nonlinear functions, one needs to examine multi-dimensional activation functions, i.e. ones of the form φ : R n ↦→ R m with n ≥ 2 . In order to preserve the dimensions of the neural layers and to limit the computational costs, we focus on the case n = m = 2 , meaning that φ processes pre-activated neurons by pairs. According to Prop. 2, the normalization-equivariant functions from R 2 to R 2 are parameterized by two matrices A,B ∈ R 2 × 2 such that A 1 2 = B 1 2 = 1 2 and apply a different (affine-constrained) linear mapping depending on whether or not the input is in ascending order. As long as A = B , the resulting function is nonlinear and makes it de facto a candidate to replace the conventional one-dimensional activation functions such as the popular ReLU (rectified linear unit) function. Interestingly, when arbitrarily choosing A and B to be the permutation matrices of R 2 × 2 , the resulting normalization-equivariant function simply reads:

which is nothing else than the sorting function in R 2 . Clearly, it is among the simplest normalizationequivariant nonlinear function from R 2 to R 2 and it is the one we consider as surrogate for the one-dimensional activation functions (choosing other functions, that is considering other choices for A and B , does not bring improvements in terms of performance in our experiments). More generally, it is easy to show that all the sorting functions of R n are normalization-equivariant and are nonlinear as soon as n ≥ 2 . Note that such sorting operators have been promoted by [2, 9] in totally different contexts for their norm-preserving properties of the backpropagated gradients.

Since the sorting function (6) is to be applied on non-overlapping pairs of neurons, the partitioning of layers needs to be determined. In order not to mix unrelated neurons, we propose to apply this two-dimensional activation function channel-wisely across layers and call this operation 'sort pooling' in reference to the max pooling operation, widely used for downsampling, and from which it can be effectively implemented. Figure 2 illustrates the sequence of the two proposed innovations, namely affine convolution followed by channel-wise sort pooling, to replace the traditional scheme 'conv+ReLU', while guaranteeing normalization-equivariance .

4.3 Encoding adaptive affine filters

Based on Lemma 2, we can formulate the following proposition which tells more about the class of parameterized functions ( f θ ) encoded by the proposed networks.

一

...... NOISY

SCALE-EQUIVARIANT

NORMALIZATION-EQUIVARIANT

Denoised

Adaptive filters

Denoised

Adaptive filters

σ

= 25

Pixel 1

Pixel 2

PSNR:

20.17 DB

PSNR:

32.38 DB

Σ = 0

.

99

Σ = 1

.

00

PSNR:

32.36 DB

Σ = 1

Σ = 1

σ

= 5

PSNR:

34.15 DB

PSNR:

30.56 DB

Σ = 0

.

94

Σ = 0

.

99

PSNR:

36.72 DB

Σ = 1

Σ = 1

Proposition 3 Let f NE θ : R n ↦→ R m be a CNN composed of only:

affine convolution kernels with no bias and where padding is made of existing features,

sort pooling nonlinearities,

possibly skip or affine residual connections, and max or average pooling layers.

Then, f NE θ is a normalization-equivariant continuous piecewise-linear function with finitely many pieces. Moreover, on each piece represented by the vector y r ,

In Prop. 3, the subscripts on A y r θ serve as a reminder that this matrix depends on the sort pooling activation patterns, which in turn depend on both the input vector y and the weights θ . As already revealed for bias-free networks with ReLU [33], A y r θ is the Jacobian matrix of f NE θ taken at any point y in the interior of the piece represented by vector y r . Moreover, as A y r θ 1 n = 1 m , the output vector of such networks are locally made of fixed affine combinations of the entries of the input vector. And since a CNN has a limited receptive field centered on each pixel, f NE θ can be thought of as an adaptive filter that produces an estimate of each pixel through a custom affine combination of pixels. By examining these filters in the case of image denoising (see Fig. 3), it becomes apparent that they vary in their characteristics and are intricately linked to the contents of the underlying images. Indeed, these filters are specifically designed to cater to the specific local features of the noisy image: averaging is done over uniform areas without affecting the sharpness of edges. Note that this behavior has already been extensively studied by [33] for unconstrained filters.

The total number of fixed adaptive affine filters depends on the weights θ of the network f NE θ and is bounded by 2 S where S represents the total number of sort pooling patterns traversed to get from the receptive filed to its final pixel (assuming no max pooling layers). Obviously, this upper bound grows exponentially with S , suggesting that a limited number of sort pooling operations may generate an extremely large number of filters. Interestingly, if ReLU activation functions where used instead, the upper bound would reach 2 2 S .

5 Experimental results

We demonstrate the effectiveness and versatility of the proposed methodology in the case of image denoising. To this end, we modify two well-established neural network architectures for image denoising, chosen for both their simplicity and efficiency, namely DRUNet [46]: a state-of-the-art

8

12

16

20

24

28

32

8

12

16

20

24

28

32

Input PSNR

Output PSNR

ordinary

scale-equiv

norm-equiv

identity

8

12

16

20

24

28

32

8

12

16

20

24

28

32

36

Input PSNR

8

12

16

20

24

28

32

8

12

16

20

24

28

32

36

Input PSNR

U-Net with residual connections [17]; and FDnCNN, the unpublished flexible variant of the popular DnCNN [47]: a simple feedforward CNN that chains 'conv+ReLU' layers with no downsampling, no residual connections and no batch normalization during training [19], and with a tunable noise level map as additional input [48]. We show that adapting these networks to become normalizationequivariant does not adversely affect performance and, better yet, increases their generalization capabilities. For each scenario, we train three variants of the original Gaussian denoising network for grayscale images: ordinary (original network with additive bias), scale-equivariant (bias-free variation with ReLU [33]) and our normalization-equivariant architecture (see Fig. 2). Details about training and implementations can be found in Appendix A and Appendix B. Unless otherwise noted, all results presented in this paper are obtained with DRUNet [46]; similar outcomes can be achieved with FDnCNN [47] architecture (see Appendix D).

Finally, note that both DRUNet [46] and FDnCNN [47] can be trained as 'blind' but also as 'nonblind' denoisers and thus achieve increased performance, by passing an additional noisemap as input. In the case of additive white Gaussian noise of variance σ 2 , the noisemap is constant equal to σ 1 n and the resulting parameterized functions can then be put mathematically under the form f θ : ( y, σ ) ∈ R n × R + ∗ ↦→ R n . In order to integrate this feature to normalization-equivariant networks as well, a slight modification of the first affine convolutional layer must be made. Indeed, by adapting the proof (5) to the case (1), we can show that the first convolutional layer must be affine with respect to the input image y only - the coefficients of the kernels acting on the image pixels add up to 1 - while the other coefficients of the kernels need not be constrained.

5.1 The proposed architectural modifications do not degrade performance

The performance, assessed in terms of PSNR values, of our normalization-equivariant alternative (see Fig. 2) and of its scale-equivariant and ordinary counterparts is compared in Table 2 for 'non-blind' architectures on two popular datasets [32]. We can notice that the performance gap between two different variants is less than 0.05 dB at most for all noise levels, which is not significant. This result suggests that the class of parameterized functions ( f θ ) currently used in image denoising can drastically be reduced at no cost. Moreover, it shows that it is possible to dispense with activation functions, such as the popular ReLU: nonlinearities can simply be brought by sort pooling patterns. In terms of subjective visual evaluation, we can draw the same conclusion since images produced by two architectural variants inside the training range are hardly distinguishable (see Fig. 3 at σ = 25 ).

5.2 Increased robustness across noise levels

S. Mohan et al. [33] revealed that bias-free neural networks with ReLU, which are scale-equivariant , could much better generalize when evaluated at new noise levels beyond their training range, than their counterparts with bias that systematically overfit. Even if they do not fully elucidate how such networks achieve this remarkable generalization, they suggest that scale-equivariance certainly plays a major role. What about normalization-equivariance then? We have compared the robustness faculties of the three variants of networks when trained at a fixed noise level σ for Gaussian noise. Figure

Dataset

Set12

BSD68

Noise level

σ

15

/

25

/

50

15

/

25

/

50

DRUNet [46]

ordinary

33.23 / 30.92 / 27.87

31.89 / 29.44 / 26.54

scale-equiv

33.25 / 30.94 / 27.90

31.91 / 29.48 / 26.59

norm-equiv

33.20 / 30.90 / 27.85

31.88 / 29.45 / 26.55

FDnCNN [47]

ordinary

32.87 / 30.49 / 27.28

31.69 / 29.22 / 26.27

scale-equiv

32.85 / 30.49 / 27.29

31.67 / 29.20 / 26.25

norm-equiv

32.85 / 30.50 / 27.27

31.69 / 29.22 / 26.25

PSNR: 10.63 DB

(

λ

= 3)

(a)

PSNR: 12.32 DB

PSNR: 25.28 DB

PSNR: 25.20 DB

input

y

f

∅

θ

(

y

)

3

f

∅

θ

(

y/

3)

f

NE

θ

(

y

)

PSNR: 22.67 DB

(

λ

= 0

.

75)

(b)

PSNR: 31.68 DB

PSNR: 32.46 DB

PSNR: 32.45 DB

input

y

(

x

∈

[0

.

3

,

1]

n

)

f

SE

θ

(

y

)

f

SE

θ

(

y

-

0

.

3) + 0

.

3

f

NE

θ

(

y

)

4 summarizes the explicit results obtained: normalization-equivariance pushes generalization capabilities of neural networks one step further. While performance is identical to their scale-equivariant counterparts when evaluated at higher noise levels, the normalization-equivariant networks are, however, much more robust at lower noise levels. This phenomenon is also illustrated in Fig. 3.

Demystifying robustness Let x be a clean patch of size n , representative of the training set on which a CNN f θ was optimized to denoise its noisy realizations y = x + ε with ε ∼ N (0 , σ 2 I n ) (denoising at a fixed noise level σ exclusively). Formally, we note x ∈ D ⊂ R n , where D is the space of representative clean patches of size n on which f θ was trained. We are interested in the output of f θ when it is evaluated at x + λε (denoising at noise level λσ ) with λ > 0 . Assuming that f θ encodes a normalization-equivariant function, we have:

The above equality shows how such networks can deal with noise levels λσ different from σ : normalization-equivariance simply brings the problem back to the denoising of an implicitly renormalized image patch with fixed noise level σ . Note that this artificial change of noise level does not make this problem any easier to solve as the signal-to-noise ratio is preserved by normalization.

Obviously, the denoising result of x + λε will be all the more accurate as ( x -µ ) /λ is a representative patch of the training set. In other words, if ( x -µ ) /λ can still be considered to be in D , then f θ should output a consistent denoised image patch. For a majority of methods [46-48], training is performed within the interval [0 , 1] and therefore x/λ still belongs generally to D for 1 < λ < 10 (contraction), but this is much less true for λ < 1 (stretching) for the reason that it may exceed the bounds of the interval [0 , 1] . This explains why scale-equivariant functions do not generalize well to noise levels lower than their training one. In contrast, normalization-equivariant functions can benefit from the implicit extra adjustment parameter µ . Indeed, there exists some cases where the stretched patch x/λ is not in D but ( x -µ ) /λ is (see Fig. 5b). This is why normalization-equivariant networks are more able to generalize at low noise levels. Note that, based on this argument, ordinary neural networks trained at a fixed noise level σ can also be used to denoise images at noise level λσ , provided that a correct normalization is done beforehand [42]. However, this time the normalization is explicit: the exact scale factor λ , and possibly the shift µ , must be known (see Fig. 5a).

It turns out that this theoretical argument is valid for a wide range of noise types, not only Gaussian noise. Indeed, the same argument holds for any additive noise ε that possesses the scaling property: λε belongs to the same family of probability distributions as ε (e.g., Gaussian, uniform, Laplace or even Rayleigh noise which is not zero-mean). By the way, the authors of [33] had already verified the noise generalization capabilities of scale-equivariant networks for uniform noise in addition to Gaussian noise, without fully elucidating why it works. In Appendix D, we checked experimentally that 'blind' normalization-equivariant networks trained on additive uniform, Laplace or Rayleigh noise at a single noise level are much more robust at unseen noise levels than their scale-equivariant and ordinary counterparts.

6 Conclusion and perspectives

In this work, we presented an original approach to adapt the architecture of existing neural networks so that they become normalization-equivariant , a property highly desirable and expected in many applications such that image denoising. We argue that the classical pattern 'conv+ReLU' can be favorably replaced by the two proposed innovations: affine convolutions that ensure that all coefficients of the convolutional kernels sum to one; and channel-wise sort pooling nonlinearities as a substitute for all activation functions that apply element-wise, including ReLU or sigmoid functions. Despite these two important architectural changes, we show that the performance of these alternative networks is not affected in any way. On the contrary, thanks to their better-conditioning, they benefit, in the context of image denoising, from an increased interpretability and especially robustness to variable noise levels both in practice and in theory.

Limitations

We would like to mention that the proposed architectural modifications for enforcing normalizationequivariance require a longer training for achieving comparable performance with its original counterparts (see Appendix B), and may be incompatible with some specific network layers such as batch-norm [19] or attention-based modules [7, 24, 26, 45]. Moreover, our method has shown its potential mainly to image denoising as it stands, even though in principle normalization-equivariance may be applicable and helpful in other tasks as well (see preliminary results about image classification in Appendix D). Discovering similar advantages of normalization-equivariance in other computer vision tasks, possibly related to outlier robustness, is an interesting avenue of research for future work.

Acknowledgments and Disclosure of Funding

This work was supported by Bpifrance agency (funding) through the LiChIE contract. Computations were performed on the Inria Rennes computing grid facilities partly funded by France-BioImaging infrastructure (French National Research Agency - ANR-10-INBS-04-07, 'Investments for the future'). We would like to thank R. Fraisse (Airbus) for fruitful discussions.

References

E. Agustsson and R. Timofte. NTIRE 2017 Challenge on single image super-resolution: Dataset and study. In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 1122-1131, 2017.

C. Anil, J. Lucas, and R. Grosse. Sorting out Lipschitz function approximation. In International Conference on Machine Learning (ICML) , volume 97, pages 291-301, 2019.

S. Batzner, A. Musaelian, L. Sun, M. Geiger, J. P. Mailoa, M. Kornbluth, N. Molinari, T. E. Smidt, and B. Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications , 13(1):2453, 2022.

A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. In Conference on Computer Vision and Pattern Recognition (CVPR) , volume 2, pages 60-65, 2005.

A. Buades, B. Coll, and J.-M. Morel. A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation , 4(2):490-530, 2005.

G. Bökman, F. Kahla, and A. Flinth. ZZ-Net: A universal rotation equivariant architecture for 2D point clouds. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10966-10975, 2022.

L. Chen, X. Chu, X. Zhang, and J. Sun. Simple baselines for image restoration. In European Conference on Computer Vision (ECCV) , pages 17-33, 2022.

Y. Chen and T. Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(6):1256-1272, 2017.

A. Chernodub and D. Nowicki. Norm-preserving orthogonal permutation linear unit activation functions (OPLU). arXiv preprint arXiv:1604.02313 , 2016.

T. Cohen and M. Welling. Group equivariant convolutional networks. In International Conference on Machine Learning (ICML) , volume 48, pages 2990-2999, 2016.

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing , 16(8):2080-2095, 2007.

V. Duval, J.-F. Aujol, and Y. Gousseau. A bias-variance approach for the nonlocal means. SIAM Journal on Imaging Sciences , 4(2):760-788, 2011.

F. Fuchs, D. Worrall, V. Fischer, and M. Welling. SE(3)-Transformers: 3D roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 1970-1981, 2020.

K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics , 36(4):193-202, 1980.

S. Gu, L. Zhang, W. Zuo, and X. Feng. Weighted nuclear norm minimization with application to image denoising. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2862-2869, 2014.

D. K. Gupta, D. Arya, and E. Gavves. Rotation equivariant siamese networks for tracking. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12357-12366, 2021.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

S. Herbreteau and C. Kervrann. Towards a unified view of unsupervised non-local methods for image denoising: the NL-Ridge approach. In IEEE International Conference on Image Processing (ICIP) , pages 3376-3380, 2022.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML) , volume 37, pages 448-456, 2015.

Q. Jin, I. Grama, C. Kervrann, and Q. Liu. Nonlocal means and optimal weights for noise removal. SIAM Journal on Imaging Sciences , 10(4):1878-1920, 2017.

N. Keriven and G. Peyré. Universal invariant and equivariant graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation , 1(4):541-551, 1989.

J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. SwinIR: Image restoration using Swin Transformer. In International Conference on Computer Vision Workshops (ICCVW) , pages 1833-1844, 2021.

B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 1132-1140, 2017.

D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang. Non-local recurrent network for image restoration. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

C. Louchet and L. Moisan. Total variation as a local filter. SIAM Journal on Imaging Sciences , 4(2):651694, 2011.

K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing , 26(2):10041016, 2017.

J. V. Manjón, J. Carbonell-Caballero, J. J. Lull, G. García-Martí, L. Martí-Bonmatí, and M. Robles. MRI denoising using non-local means. Medical image analysis , 12(4):514-523, 2008.

X. Mao, C. Shen, and Y.-B. Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neural Information Processing Systems (NIPS) , volume 29, 2016.

D. Marcos, M. Volpi, N. Komodakis, and D. Tuia. Rotation equivariant vector field networks. In International Conference on Computer Vision (ICCV) , pages 5058-5067, 2017.

D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International Conference on Computer Vision (ICCV) , volume 2, pages 416-423 vol.2, 2001.

S. Mohan, Z. Kadkhodaie, E. P. Simoncelli, and C. Fernandez-Granda. Robust and interpretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations (ICLR) , 2020.

A. Nøkland and L. H. Eidnes. Training neural networks with local error signals. In International Conference on Machine Learning (ICML) , pages 4839-4850, 2019.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

M. Prakash, M. Lalit, P. Tomancak, A. Krul, and F. Jug. Fully unsupervised probabilistic Noise2Void. In International Symposium on Biomedical Imaging (ISBI) , pages 154-158, 2020.

L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena , 60:259-268, 1992.

V. G. Satorras, E. Hoogeboom, and M. Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning (ICML) , volume 139, pages 9323-9332, 2021.

W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1874-1883, 2016.

N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley. Tensor field networks: Rotationand translation-equivariant neural networks for 3D point clouds. arXiv preprint arXiv:1802.08219 , 2018.

B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling. Rotation equivariant CNNs for digital pathology. In Medical Image Computing and Computer Assisted Intervention (MICCAI) , pages 210-218, 2018.

Y.-Q. Wang and J.-M. Morel. Can a single image denoising neural network handle all levels of Gaussian noise? IEEE Signal Processing Letters , 21(9):1150-1153, 2014.

M. Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant CNNs. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 849-858, 2018.

G. Yu and G. Sapiro. DCT image denoising: a simple and effective image denoising algorithm. Image Processing On Line , 1:292-296, 2011.

S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. Yang. Restormer: Efficient transformer for high-resolution image restoration. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5718-5729, 2022.

K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-Play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):6360-6376, 2022.

K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing , 26(7):3142-3155, 2017.

K. Zhang, W. Zuo, and L. Zhang. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing , 27(9):4608-4622, 2018.

A Description of the denoising architectures and implementation

A.1 Description of models

DRUNet: DRUNet [46] is a U-Net architecture, and as such has an encoder-decoder type pathway, with residual connections [17]. Spatial downsampling is performed using 2 × 2 convolutions with stride 2 , while spatial upsampling leverages 2 × 2 transposed convolutions with stride 2 (which is equivalent to a 1 × 1 sub-pixel convolution [39]). The number of channels in each layer from the first scale to the fourth scale are 64 , 128 , 256 and 512 , respectively. Each scale is composed of 4 successive residual blocks ' 3 × 3 conv + ReLU + 3 × 3 conv'.

FDnCNN: FDnCNN [47] is the unpublished flexible variant of the popular DnCNN [47]. It consists of 20 successive 3 × 3 convolutional layers with 64 channels each and ReLU nonlinearities. As opposed to DnCNN, FDnCNN does not use neither batch normalization [19] for training, nor residual connections [17] and can handle an optional noisemap (concatenated with the input noisy image). Note that this architecture does not use downsampling or upsampling. Finally, the authors [47] recommend to train it by minimizing the ℓ 1 loss instead of the mean squared error (MSE).

A.2 Description of variants

Ordinary: The ordinary variant is built by appending additive constant ('bias') terms after each convolution of the original architecture. Note that the original FDnCNN [47] model is already in the ordinary mode.

Scale-equivariant: Since both models (DRUNet and FDnCNN) use only ReLU activation functions, removing all additive constant ('bias') terms is sufficient to ensure scale-equivariance [33]. Note that the original DRUNet [46] model is already in the scale-equivariant mode.

Normalization-equivariant: All convolutions are replaced by the proposed affine-constrained convolutions without 'bias' and with reflect padding, and the proposed channel-wise sort pooling patterns supersede ReLU nonlinearities. Moreover, classical residual connections are replaced by affine residual connections (the sum of two layers l 1 and l 2 is replaced by their affine combination (1 -t ) l 1 + tl 2 where t is a trainable scalar parameter).

A.3 Practical implementation of normalization-equivariant networks

The channel-wise sort pooling operations can be efficiently implemented by concatenating the sublayer obtained with channel-wise one-dimensional max pooling with kernel size 2 and its counterpart obtained with min pooling. Note that intertwining these two sub-layers to comply with the original definition is not necessary in practice (although performed anyway in our implementation), since the order of the channels in a CNN is arbitrary.

Regarding the implementation of affine convolutions for training, each unconstrained kernel can be in practice 'telescoped' with its circular shifted version (this way, the sum of the resulting trainable coefficients cancels out) and then the inverse of the kernel size is added element-wise as a nontrainable offset. Despite this over-parameterized form (involving an extra degree of freedom), we found this solution to be more easy to use in practice. Moreover, it ensures that all coefficients of the affine kernels follow the same law at initialization. Another possibility is to set an arbitrary coefficient of the kernel (the last one for instance) equal to one minus the sum of all the other coefficients. Note that the solution consisting in dividing each kernel coefficient by the sum of all the other coefficients does not work because it generates numerical instabilities as the divisor may be zero, or close to zero.

All our implementations are written in Python and are based on the PyTorch library [35]. The code and pre-trained models can be downloaded here: https://github.com/sherbret/normalization_ equivariant_nn/ .

GPU

(scale-equiv)

(norm-equiv)

✗

✗

0.229

0.067

✗

✓

0.268

0.102

✓

✗

0.344

0.083

✓

✓

0.386

0.122

CPU

Variant

Backward pass

↓

Inference pass

↓

ordinary

38.4

12.5

scale-equiv

37.4

12.2

norm-equiv

45.3

14.3

SortPool

Backward pass

↓

Inference pass

↓

Affine: affine-constrained convolutions with reflect padding and affine residual connections. SortPool: channel-wise sort pooling nonlinearities instead of ReLU.

B Description of datasets and training details

We use the same large training set as in [46] for all the models and all the experiments, composed of 8 , 694 images, including 400 images from the Berkeley Segmentation Dataset BSD400 [32], 4 , 744 images from the Waterloo Exploration Database [28], 900 images from the DIV2K dataset [1], and 2 , 750 images from the Flickr2K dataset [25]. This training set is augmented via random vertical and horizontal flips and random 90 ◦ rotations. The dataset BSD32 [32], composed of the 32 images, is used as validation set to control training and select the best model at the end. Finally, the two datasets Set12 and BSD68 [32], strictly disjoint from the training and validation sets, are used for testing.

All the models f θ are optimized by minimizing the average reconstruction error between the denoised images ˆ x = f θ ( x + ε ) , where ε ∼ N (0 , σ 2 I n ) , and ground-truths x with Adam algorithm [22]. For 'non-blind' models, the noise level σ is randomly chosen from [1 , 50] during training. The training parameters, specific to each model and its variants, are guided by the instructions of the original papers [46, 47], to the extent possible, and are summarized in Table 3. Note that each training iteration consists in a gradient pass on a batch composed of patches randomly cropped from training images. Normalization-equivariant variants need a longer training and always use a constant learning rate (speed improvements are however certainly possible by adapting the learning rate throughout optimization, but we did not investigated much about it). Furthermore, contrary to [46] where the ℓ 1 loss function is recommended to achieve better performance, supposedly due to its outlier robustness properties, we obtained slightly better results with the usual mean squared error (MSE) loss when dealing with normalization-equivariant networks. Training was performed with a Quadro RTX 6000 GPU.

In Table B, we compare the computational costs of different variants for training and inference. Interestingly, the computational cost on GPU for training is much more sensitive to the 'affine mode' (involving affine-constrained convolutions with reflect padding and affine residual connection) than to sort pooling nonlinearities, while it is the opposite for inference. All in all, for gaining normalizationequivariance , the learning and inference time is almost doubled for the DRUNet architecture [46] on GPU. Note however that we do not claim to have the most optimized implementation and there

is probably room for improvement. Surprisingly, the difference in speed between all variants is much less pronounced on CPU. In particular, the inference pass takes only about 20% longer for the normalization-equivariant DRUNet on CPU.

C Mathematical proofs for normalization-equivariant neural networks

C.1 Proofs of Lemmas and Propositions

Lemma 1 (Characterizations)

Proof: For each type of equivariance, both existence and uniqueness of f must be proven. Let 0 n be the zero vector of R n and ( y x ) x ∈C the values that f takes on its characteristic set C .

Scale-equivariance:

Uniqueness: Let f and g two scale-equivariant functions such that ∀ x ∈ S , f ( x ) = g ( x ) . First of all, for any scale-equivariant function h , h ( 0 n ) = h (2 · 0 n ) = 2 h ( 0 n ) , hence h ( 0 n ) = 0 m . Therefore, f ( 0 n ) = g ( 0 n ) = 0 m .

̸

Existence: Let f : x ∈ R n ↦→ { ∥ x ∥ · y x ∥ x ∥ if x = 0 n 0 m otherwise . Note that ∀ x ∈ S , f ( x ) = y x . Let x ∈ R n and λ ∈ R + ∗ . If x = 0 n , f ( λx ) = ∥ λx ∥ · y λx ∥ λx ∥ = λ ∥ x ∥ · y x ∥ x ∥ = λf ( x ) and if x = 0 n f ( λx ) = 0 m = λf ( x ) , hence f is scale-equivariant .

Shift-equivariance:

Uniqueness: Let f and g two shift-equivariant functions such that ∀ x ∈ Span( 1 n ) ⊥ , f ( x ) = g ( x ) . Let x ∈ R n . By orthogonal decomposition of R n into Span( 1 n ) ⊥ and Span( 1 n ) :

Existence: Let f : x ∈ R n ↦→ y x 1 + x 2 , where x = x 1 + x 2 is the unique decomposition such that x 1 ∈ Span( 1 n ) ⊥ and x 2 ∈ Span( 1 n ) . Note that ∀ x ∈ Span( 1 n ) ⊥ , f ( x ) = y x . Let x ∈ R n and µ ∈ R . f ( x + µ ) = y x 1 + x 2 + µ 1 m = f ( x ) + µ as if x orthogonally decomposes into x 1 + x 2 with x 1 ∈ Span( 1 n ) ⊥ and x 2 ∈ Span( 1 n ) , then x + µ orthogonally decomposes into x 1 +( x 2 + µ 1 m ) . f is then shift-equivariant .

Normalization-equivariance:

Uniqueness: Let f and g two normalization-equivariant functions such that ∀ x ∈ S ∩ Span( 1 n ) ⊥ , f ( x ) = g ( x ) . First, as f and g are a fortiori scale-equivariant , f ( 0 n ) = g ( 0 n ) = 0 m . Let x ∈ R n \ { 0 n } . By orthogonal decomposition of R n into Span( 1 n ) ⊥ and Span( 1 n ) :

̸

If x 1 = 0 n , f ( x ) = f ( 0 n + x 2 ) = f ( 0 n ) + x 2 = 0 m + x 2 = x 2 . Likewise, g ( x ) = x 2 , hence f ( x ) = g ( x ) . Else, if x 1 = 0 n , f ( x ) = f ( x 1 + x 2 ) = f ( x 1 ) + x 2 = ∥ x 1 ∥ f ( x 1 ∥ x 1 ∥ ) + x 2 = ∥ x 1 ∥ g ( x 1 ∥ x 1 ∥ ) + x 2 = g ( x 1 ) + x 2 = g ( x 1 + x 2 ) = g ( x ) , as x 1 ∥ x 1 ∥ ∈ S ∩ Span( 1 n ) ⊥ . Finally, f = g .

̸

Existence: Let f : x ∈ R n ↦→ { ∥ x 1 ∥ · y x 1 ∥ x 1 ∥ + x 2 if x 1 = 0 n x 2 otherwise , where x = x 1 + x 2 is the unique decomposition such that x 1 ∈ Span( 1 n ) ⊥ and x 2 ∈ Span( 1 n ) . Note that ∀ x ∈ S ∩ Span( 1 n ) ⊥ , f ( x ) = y x . Let x ∈ R n , λ ∈ R + ∗ and µ ∈ R . x decomposes

̸

orthogonally into x 1 + x 2 with x 1 ∈ Span( 1 n ) ⊥ and x 2 ∈ Span( 1 n ) , and we have f ( λx + µ ) = f ( λx 1 +( λx 2 + µ )) , where λx 1 +( λx 2 + µ ) is the orthogonal decomposition of λx + µ into Span( 1 n ) ⊥ and Span( 1 n ) .

̸

̸

Lemma 2 (Operations preserving equivariance)

Proof: Let x ∈ R n , λ ∈ R + ∗ and µ ∈ R .

If f and g are both scale-equivariant , ( f ◦ g )( λx ) = f ( g ( λx )) = f ( λg ( x )) = λf ( g ( x )) = λ ( f ◦ g )( x ) and if they are both shift-equivariant , ( f ◦ g )( x + µ ) = f ( g ( x + µ )) = f ( g ( x ) + µ ) = f ( g ( x )) + µ = ( f ◦ g )( x ) + µ .

Let h : x ↦→ ( f ( x ) ⊤ g ( x ) ⊤ ) ⊤ . If f and g are both scale-equivariant , h ( λx ) = ( f ( λx ) ⊤ g ( λx ) ⊤ ) ⊤ = ( λf ( x ) ⊤ λg ( x ) ⊤ ) ⊤ = λh ( x ) and if they are both shift-equivariant , h ( x + µ ) = ( f ( x + µ ) ⊤ g ( x + µ ) ⊤ ) ⊤ = ( f ( x ) ⊤ + µ g ( x ) ⊤ + µ ) ⊤ = h ( x ) + µ .

Let t ∈ R and h : x ↦→ (1 -t ) f + tg . If f and g are both scale-equivariant , h ( λx ) = (1 -t ) f ( λx ) + tg ( λx ) = (1 -t ) λf ( x ) + tλg ( x ) = λ ((1 -t ) f ( x ) + tg ( x )) = λh ( x ) and if they are both shift-equivariant , h ( x + µ ) = (1 -t ) f ( x + µ ) + tg ( x + µ ) = (1 -t )( f ( x ) + µ ) + t ( g ( x ) + µ ) = (1 -t ) f ( x ) + tg ( x ) + (1 -t ) µ + tµ = h ( x ) + µ . □

Proposition 1

Proof: Let a < b ∈ R , f : R n ↦→ R m , x ∈ R n , λ ∈ R + ∗ and µ ∈ R .

We have T a,b ( λx + µ ) = ( b -a ) λx + µ -min( λx + µ ) max( λx + µ ) -min( λx + µ ) + a = ( b -a ) x -min( x ) max( x ) -min( x ) + a = T a,b ( x ) ( i.e. T a,b is normalization-invariant ). T -1 a,b denotes the inverse transformation intricately linked to the input x of T a,b (note that this is an improper notation as T a,b is not bijective). Thus, if x is the input of T a,b , then T -1 a,b : y ↦→ (max( x ) -min( x )) y -a b -a +min( x ) .

Finally, T -1 a,b ◦ f ◦ T a,b is normalization-equivariant .

Proposition 2

Proof: Let id : x ∈ R ↦→ x be the identity function. id is a normalization-equivariant function so { x ↦→ x } ⊆ NE(1) . Reciprocally, let f ∈ NE(1) . By scale-equivariance , f (0) = f (2 × 0) = 2 f (0) , hence f (0) = 0 . By shift-equivariance , ∀ x ∈ R , f ( x ) = f ( x +0) = f (0) + x = x , hence, f = id . Finally, NE(1) ⊆ { x ↦→ x } , hence NE(1) = { x ↦→ x } . Note that it is coherent with Lemma 1 which states that f is entirely determined by its values on S ∩ Span( 1 n ) ⊥ , which reduces to the empty set for n = 1 .

□

Reciprocally, let f ∈ NE(2) . For n = 2 and when considering the Euclidean distance, S ∩ Span( 1 n ) ⊥ = {-u, u } with u = ( -1 / √ 2 , 1 / √ 2) . Let

(in particular g is then normalization-equivariant ) and ∀ x ∈ {-u, u } , g ( x ) = f ( x ) . According to Lemma 1, g = f , hence f ∈ F . Finally, NE(2) ⊆ F , hence NE(2) = F . □

Proposition 3

Proof: f NE θ is composed of three types of building blocks of the following form:

affine convolutions: a Θ : x ∈ R n ↦→ Θ x with Θ ∈ R m × n subject to Θ 1 n = 1 m ,

sort pooling nonlinearities: sortpool : R n ↦→ R n ,

max pooling layers: maxpool : R n ↦→ R m with m<n ,

which are assembled using:

function compositions: comp( f, g ) ↦→ f ◦ g ,

skip connections: skip( f, g ) ↦→ ( x ↦→ ( f ( x ) ⊤ g ( x ) ⊤ ) ⊤ ) ,

affine residual connections: ares t ( f, g ) ↦→ (1 -t ) f + tg with t ∈ R .

Note that the rows of Θ in a Θ encode the convolution kernels in a CNN and the trainable parameters, denoted by θ , are only composed of matrices Θ and scalars t . Moreover, note that average pooling layers are nothing else than affine convolutions with fixed parameters.

Since a Θ , sortpool and maxpool are normalization-equivariant functions, Lemma 2 states that the resulting function f NE θ is also normalization-equivariant . Moreover, since they are continuous and the assembling operators preserve continuity, f NE θ is continuous. Then, for a given input x ∈ R n , we have (sortpool ◦ a Θ )( x ) = a π (Θ) ( x ) = π (Θ) x , where π an operator acting on matrix Θ by permuting its rows (note that the permutation π is both dependent on x and Θ ). Therefore, applying a pattern 'conv affine + sortpool' simply amounts locally to a linear transformation. Moreover, since applying a max pooling layer amounts to removing some rows from matrix Θ , the local linear behavior is preserved. Thus, as the nonlinearities of f NE θ are exclusively brought by sort pooling patterns (and possibly max pooling layers), f NE θ is actually locally linear. In other words, f NE θ is piecewise-linear. Moreover, as there is a finite number (although high) of possible permutations (and possibly eliminations) of the rows of all matrices Θ , f NE θ has finitely many pieces. Finally, on each piece represented by the vector y r , f NE θ ( y ) = A y r θ y . It remains to prove that A y r θ 1 n = 1 m . But this property is easily obtained by noticing that, subject to dimensional compatibility on matrices Θ :

Θ 1 n = 1 m ⇒ π (Θ) 1 n = 1 m ('conv affine + sortpool') ,

Θ 1 n = 1 m ⇒ ρ (Θ) 1 n = 1 l ('conv affine + maxpool') where ρ removes some rows ,

Θ 1 1 n = 1 m and Θ 2 1 m = 1 l ⇒ Θ 2 Θ 1 1 n = 1 l (composition) ,

Θ 1 1 n = 1 m and Θ 2 1 n = 1 m ⇒ (1 -t )Θ 1 1 n + t Θ 2 1 n = 1 m (affine residual connection) .

Thus, the affine combinations are preserved all along the layers of f NE θ . In the end,

C.2 Examples of normalization-equivariant conventional denoisers

Noise-reduction filters: All linear smoothing filters can be put under the form f Θ : x ∈ R n ↦→ Θ x with Θ ∈ R n × n (the rows of Θ encode the convolution kernel). Obviously, f Θ is always scaleequivariant , whatever the filter Θ . As for the shift-equivariance , a simple calculation shows that:

Since the sum of the coefficients of a Gaussian kernel and an averaging kernel is one, we have Θ 1 n = 1 m , hence these linear filters are normalization-equivariant . The median filter is also normalization-equivariant because median( λx + µ ) = λ median( x ) + µ for λ ∈ R + ∗ and µ ∈ R .

Patch-based denoising:

-

-NLM [5]: Assuming that the smoothing parameter h is proportional to σ , i.e. h = ασ , we have e -∥ p ( λy i + µ ) -p ( λy j + µ ) ∥ 2 2 ( αλσ ) 2 = e -λ 2 ∥ p ( y i ) -p ( y j ) ∥ 2 2 λ 2 ( ασ ) 2 = e -∥ p ( y i ) -p ( y j ) ∥ 2 2 h 2 , hence the aggregation weights are normalization-invariant . Then,

Finally, f NLM is a normalization-equivariant function.

-NL-Ridge [18]: The block-matching procedure at the heart of NL-Ridge is normalization-invariant as it is based on comparisons of the ℓ 2 norm of the difference of image patches. For each noisy patch group, a.k.a. similarity matrix, Y ∈ R n × k composed of k vectorized similar patches of size n , the optimal weights Θ ∗ ∈ R k × k , in the ℓ 2 risk sense, are computed such that Y Θ ∗ is as close as possible to the (unknown) clean patch group X ∈ R n × k . The two successive minimization problems approximating Θ under affine constraints C = { Θ ∈ R k × k , Θ ⊤ 1 k = 1 k } can be put under the form:

with Q = Y ⊤ Y or Q = ˆ X ⊤ ˆ X + nσ 2 I k for the first and second step, respectively ( ˆ X is the patch group estimate obtained after the first step), C = nσ 2 I k -Q and where tr denotes the trace operator. Depending on the step, we have:

where ∥ . ∥ F is the Frobenius norm. But, for any Z ∈ R n × k and any function h : Θ ∈ R k × k ↦→ R ,

assuming that Θ ⊤ 1 k = 1 k . Therefore, the aggregation weights Θ ∗ are normalization-invariant and ( λY + µ )Θ ∗ = λY Θ ∗ + µ . Finally, NL-Ridge with affine constraints encodes a normalizationequivariant function.

TVdenoising: Let y ∈ R n , λ ∈ R + ∗ and µ ∈ R . Let x ∗ = arg min x ∈ R n ∥ x ∥ TV s.t. ∥ y -x ∥ 2 2 = nσ 2 be the solution of TV [37].

Finally, f TV is a normalization-equivariant function.

D Additional results

8

12

16

20

24

28

32

8

12

16

20

24

28

32

Input PSNR

Output PSNR

ordinary

scale-equiv

norm-equiv

identity

8

12

16

20

24

28

32

8

12

16

20

24

28

32

36

Input PSNR

8

12

16

20

24

28

32

8

12

16

20

24

28

32

36

Input PSNR

8

12

16

20

24

28

32

8

12

16

20

24

28

32

Input PSNR

Output PSNR

ordinary

scale-equiv (relu)

norm-equiv

scale-equiv (affine)

scale-equiv (sortpool)

identity

8

12

16

20

24

28

32

8

12

16

20

24

28

32

Input PSNR

(a) Uniform

(b) Laplace

8

12

16

20

24

28

32

8

12

16

20

24

28

32

Input PSNR

5

15

25

35

45

55

65

75

85

26

28

30

32

34

36

38

Quality factor

(c) Rayleigh

(d) JPEG

Note that for norm-equiv variants, learning rate is initialized to 3 e -5 instead of 5 e -4 and dropout rate is halved.

...... NOISY

SCALE-EQUIVARIANT

NORMALIZATION-EQUIVARIANT

Denoised

Adaptive filters

Denoised

Adaptive filters

σ

= 75

PSNR:

10.63 DB

PSNR:

27.53 DB

Σ = 0

.

98

Σ = 1

.

00

PSNR:

27.54 DB

Σ = 1

Σ = 1

σ

= 25

Pixel 1

Pixel 2

PSNR:

20.17 DB

PSNR:

32.18 DB

Σ = 1

.

04

Σ = 0

.

99

PSNR:

32.14 DB

Σ = 1

Σ = 1

σ

= 5

PSNR:

34.15 DB

PSNR:

33.67 DB

Σ = 0

.

99

Σ = 0

.

99

PSNR:

36.63 DB

Σ = 1

Σ = 1

PSNR: 10.63 DB

(

λ

= 3)

(a)

PSNR: 12.34 DB

PSNR: 24.45 DB

PSNR: 24.34 DB

input

y

f

∅

θ

(

y

)

3

f

∅

θ

(

y/

3)

f

NE

θ

(

y

)

PSNR: 22.67 DB

(

λ

= 0

.

75)

(b)

PSNR: 31.69 DB

PSNR: 32.14 DB

PSNR: 32.07 DB

input

y

(

x

∈

[0

.

3

,

1]

n

)

f

SE

θ

(

y

)

f

SE

θ

(

y

-

0

.

3) + 0

.

3

f

NE

θ

(

y

)

Noisy

Ordinary

Scale-equiv

Normalization-equiv

σ

= 10

PSNR: 36.11 DB

PSNR: 36.20 DB

PSNR: 36.12 DB

PSNR: 35.64 DB

PSNR: 35.68 DB

PSNR: 35.65 DB

DRUNet

FDnCNN

σ

= 20

PSNR: 30.09 DB

PSNR: 30.16 DB

PSNR: 30.13 DB

PSNR: 29.57 DB

PSNR: 29.45 DB

PSNR: 29.65 DB

DRUNet

FDnCNN

σ

= 30

PSNR: 28.79 DB

PSNR: 28.82 DB

PSNR: 28.85 DB

PSNR: 27.83 DB

PSNR: 27.81 DB

PSNR: 27.96 DB

DRUNet

FDnCNN

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes normalization-equivariant CNNs, in which the convolutional kernels are forced to sum to 1 and the activation sorts pairs of neurons. This is shown to perform similarly to other models on image denoising when trained with the same noise levels, but generalizes much better to unseen noise levels.\n\nSTRENGTHS:\nS1) The paper presents a simple and effective method to improve generalization in image denoising CNNs.\n\nS2) Results are strong, particularly the ones shown in Fig 4.\n\nS3) Section 5.2 presents insights about previously observed phenomena of bias-free/scale-invariant models performing well on denoising, and explains why the normalization-equivariance proposed is potentially superior.\n\nWEAKNESSES:\nW2) Table 2 reports results on the non-blind experiments but I couldn't find whether the results in Fig 4 and 5 are blind or non-blind. I assume they are blind and since the models were trained on a single noise level. If they are not and if the noise level is known at inference time, I think the significance of the results is diminished since with a known noise level the ordinary CNNs can be perform quite well as shown in Fig. 5. Please clarify.\n  \nW3) Fig 2 implies that the kernel constraint for convolutional kernels is along the channel axis, but it seems that it should be along both channel and spatial dimensions to preserve shift-equivariance. The text does not seem to specify it for the convolutional kernel since Eq 5 refers to a linear layer. Please clarify and include a precise definition of the convolutional kernel between two layers of shape H x W x C. \n\nW4) Some recent methods of denoising are missed [1, 2]. Zamir et [1] seems to report slightly better results than the baselines on denoising on the same datasets and also evaluate on different datasets. This is not a dealbreaker but the submission would be made stronger when evaluating on more datasets since the ones used are quite small.\n\nReferences:\n\n[1] Zamir et al, \"Restormer: Efficient Transformer for High-Resolution Image Restoration\", CVPR'22.\n\n[2] Chen et al, \"Simple Baselines for Image Restoration\", ECCV'22.\n\nQUESTIONS:\nMinor:\nTypically in image processing, for an image f: Z^2 -> R, \"scale\" refers to resizing (f'(x) = f(ax)) and \"shift\" refers to moving pixels (f'(x) = f(x+a), in the sense that convolution is shift-equivariant). In the submission, it seems that \"scale\" means multiplying all pixel values by a constant and shift means adding a constant to all pixels. I recommend clarifying the difference to avoid confusion.\n\nLIMITATIONS:\nW1) I couldn't find a discussion of limitations in the paper and I think this is a major weakness. Table 1 in the supplement shows that the norm-equivariant models need to be trained for about twice the number of iterations of the baselines and with MSE instead of L1 loss. For fairness, I think the ordinary and scale-eq baselines should be also evaluated under the same conditions and this possible limitation should be reported in the main text. It seems that the method is also incompatible with other common used layers such as max-pooling and batch normalization, which should also be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper propose an architectural change in neural networks, in a way that enforces the normalization equivariance property. The authors demonstrate how networks that satisfy this property can be used in the setting of image denoising.\n\nSTRENGTHS:\n- The paper is clearly written overall. The intended goal of the architectural changes proposed is clear, and how these changes achieve this goal is also made explicit in the paper. The proposed architecture is also novel, as far as I am aware.\n\n- There is a clear benefit derived from the architectural changes proposed by the authors. The intended effect of normalization equivariance is to make the output of the network disregard potential alterations in the scale of the noise added to the model. This is made clear via Figure 4, which precisely shows that the alterations to the architecture achieve the intended effect. The authors also adequately explain in the paper how this equivariance with respect to this normalization makes their models robust to the amount of noise added to the input, namely in that adding different amounts of noise only corresponds to renormalization.\n\n- The idea that the authors propose regarding sort pool between channels is an interesting and novel way of pooling information from different channels in CNNs. This seems to be a novel inclusion in the architecture of a neural network, and I am interested in its potential applications beyond normalization equivariance.\n\nWEAKNESSES:\nMy main issues with this paper lie with the somewhat limited setting of the experiments performed to validate the abilities of the proposed architecture.\n\n- While interesting, the target problem for this paper is somewhat limited. The only application considered in this paper is that of image denoising, which is arguably a very limited setting. I believe that demonstrating the effects of enforcing normalization equivariance in other problems (such as image classification or other types of reconstruction e.g. superresolution) would improve the paper.\n\n- Even in the context of denoising, the authors of this paper only consider the setting of Gaussian noise being applied to an image. While generally accepted as a valid setting for denoising, It would be interesting to see if the normalization equivariance was able to help in the case where different types of noise are applied (e.g. salt-pepper noise, random masking of pixels, and so on).\n\n- Regarding baselines, the authors compare against techniques which use regular feedforward neural networks. While the comparisons support the points made by the authors, I believe that it would be interesting to compare against techniques that employ generative networks to solve the denoising problems e.g. via DPS [A].\n\nReference:\n\n[A] Chung et al., “Diffusion Posterior Sampling for General Noisy Inverse Problems”, ICLR 2023.\n\nQUESTIONS:\nI would be interested to know if the authors have considered applying their architectural change not when training the models from scratch, but by altering select layers of a pretrained model (for example, by including the sorting layer as part of a pretrained architecture). If something like this is possible, then it would alleviate some of the issues I mentioned above, allowing the method to be more readily applied to a larger set of problems.\n\nLIMITATIONS:\nI believe the authors should better clarify in the paper how currently their method is applicable mainly to image denoising as it stands (even though in principle normalization equivariance may be helpful in other tasks as well).\n\nRegarding negative societal impact, I do not see any immediate issues arising from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an architectural design which guarantees shift and scale equivariance. The proposed method is called normalization equivariance in which scale equivariance is inherited from prior work by removing layers biases and shift equivariance is achieved through affine-constrained convolutions along with channel-wise sort pool (instead of ReLUs). They adopt two popular architectures to become normalization equivariance, and show that their variant outperforms both unconstrained, and scale-equivariance models in generalizing outside of the training range.\n\nSTRENGTHS:\nThe paper is written very clearly and has excellent flow. It is technically strong and propositions and claims are backed up both theoretically and empirically. The idea of enforcing shift equivariance by design is original and the authors nicely elucidate its necessity and the positive impact it has. In my opinion, this work offers a significant contribution in the area of useful inductive biases and also understanding neural networks trained for denoising.\n\nWEAKNESSES:\nGenerally the paper is written very well and all sections are carefully and throughly explained. Here are a few minor issues: \n\nI believe rotation needs to generally be an invariance (as opposed to equivariance) with respect to labels in image classification (line 52)\n\nProof for proposition 2 can be included in the appendix. \n\nThe derivation of eq 6 (from line 178 onward) is not clear. It would be useful if the authors re-write that paragraph. \n\nNumber of possible states is still $2^S$ if ReLU activations are used instead of sorting (line 222)\n\nQUESTIONS:\nIt would be useful to see comparison of computational cost for training. Although sort pooling is restricted to n=2, I assume it is still most costly than using ReLUs. A numeral comparison can show the cost for gaining normalization-equivariance. \n\nIt wasn't clear from the text how the affine convolution constraint was enforced during training. It would be useful to explain that in the main text since it is one of the two main design contributions.\n\nLIMITATIONS:\nNo major limitations or potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a normalization equivariant deep convolutional neural network for image denoising. The authors provide a mathematical construction of a normalization equivariant neural network and propose sorting nonlinearity. The proposed nonlinearity shows comparable performance with the baseline and shows robustness against different noise levels.\n\nSTRENGTHS:\n1. The work provides a sound mathematical formulation for the Normalization equivariant Neural Network.\n2. The proposed model is robust across different noise level\n3. The paper is well written, nicely explained\n\nWEAKNESSES:\n1. The effect of newly introduced sorting nonlinearity will be better understood through an ablation study, i.e., reporting the performance of a model with only affine convolution and Relu.\n2. The effect of shift-equivariance is not well described and demonstrated. Even though it is addressed in the last section, its’ benefit over the scale-equivariant model is not well demonstrated and motivated. Given the novelty of the work is providing shift equivariance (in addition to scale-equivariance), it deserves better motivation and justification.\n3. The proposed non-linearity performs point-wise swapping of the output of different filters at some points. So, it distorts the learned features. It has a significant possibility of affecting the performance of other computer vision tasks (e.g., segmentation), limiting its use only to denoising.\n\nQUESTIONS:\n1. The work [1] shows a better generalization of scale equivariant neural networks on different noise scales compared to what is shown here. What might be different here that lead to this difference?\n2. Line 282, “but this is much less true for λ < 1 (stretching) for the reason that it may exceed the bounds of the interval [0, 1]. This explains why scale-equivariant functions do not generalize well to noise levels lower than their training one”: how λ < 1 related to not generalizing to lower noise levels?\n\n1.ROBUST AND INTERPRETABLE BLIND IMAGE\nDENOISING VIA BIAS-FREE CONVOLUTIONAL\nNEURAL NETWORKS\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}