SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sebastien Herbreteau
Emmanuel Moebel
Charles Kervrann
Centre Inria de l'Université de Rennes, France
{sebastien.herbreteau, emmanuel.moebel, charles.kervrann}@inria.fr
In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or scaling, results in a corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied elementwise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.
Sometimes wrongly confused with the invariance property which designates the characteristic of a function  $f$  not to be affected by a specific transformation  $\mathcal{T}$  applied beforehand, the equivariance property, on the other hand, means that  $f$  reacts in accordance with  $\mathcal{T}$ . Formally, invariance is  $f \circ \mathcal{T} = f$  whereas equivariance reads  $f \circ \mathcal{T} = \mathcal{T} \circ f$ , where  $\circ$  denotes the function composition operator. Both invariance and equivariance play a crucial role in many areas of study, including physics, computer vision, signal processing and have recently been studied in various settings for deep learning-based models [3, 6, 10, 13, 14, 16, 21, 23, 31, 38, 40, 41, 43].
In this paper, we focus on the equivariance of neural networks  $f_{\theta}$  to a specific transformation  $\mathcal{T}$ , namely normalization. Although highly desirable in many applications and in spite of its omnipresence in machine learning, current neural network architectures do not equivary to normalization. With application to image denoising, for which normalization-equivariance is generally guaranteed for a lot of conventional methods [5, 18, 20, 37], we propose a methodology for adapting existing neural networks, and in particular denoising CNNs [8, 30, 46-48], so that normalization-equivariance holds by design. In short, the proposed adaptation is based on two innovations:
1. affine convolutions: the weights from one layer to each neuron from the next layer, i.e. the convolution kernels in a CNN, are constrained to encode affine combinations of neurons (the sum of the weights is equal to 1).
2. channel-wise sort pooling: all activation functions that apply element-wise, such as the ReLU, are substituted with higher-dimensional nonlinearities, namely two by two sorting along channels that constitutes a fast and efficient normalization-equivariant alternative.
Despite strong architectural constraints, we show that these simple modifications do not degrade performance and, even better, increase robustness to noise levels in image denoising both in practice and in theory.
A non-exhaustive list of application fields where equivariant neural networks were studied includes graph theory, point cloud analysis and image processing. Indeed, graph neural networks are usually expected to equivary, in the sense that a permutation of the nodes of the input graph should permute the output nodes accordingly. Several specific architectures were investigated to guarantee such a property [3, 21, 38]. In parallel, rotation and translation-equivariant networks for dealing with point cloud data were proposed in a recent line of research [6, 13, 40]. A typical application is the ability for these networks to produce direction vectors consistent with the arbitrary orientation of the input point clouds, thus eliminating the need for data augmentation. Finally, in the domain of image processing, it may be desirable that neural networks produce outputs that equivary with regard to rotations of the input image, whether these outputs are vector fields [31], segmentation maps [41, 43], or even bounding boxes for object tracking [16].
In addition to their better conditioning, equivariant neural networks by design are expected to be more robust to outliers. A spectacular example has been revealed by S. Mohan et al. [33] in the field of image denoising. By simply removing the additive constant ("bias") terms in neural networks with ReLU activation functions, they showed that a much better generalization at noise levels outside the training range was ensured. Although they do not fully elucidate why biases prevent generalization, and their removal allows it, the authors establish some clues that the answer is probably linked to the scale-equivariant property of the resulting encoded function: rescaling the input image by a positive constant value rescales the output by the same amount.
3 Overview of normalization-equivariance
3.1 Definitions and properties of three types of fundamental equivariances
We start with formal definitions of the different types of equivariances studied in this paper. Please note that our definition of "scale" and "shift" may differ from the definitions given by some authors in the image processing literature.
Definition 1 A function  $f: \mathbb{R}^n \mapsto \mathbb{R}^m$  is said to be:
- scale-equivariant if  $\forall x\in \mathbb{R}^n,\forall \lambda \in \mathbb{R}_*^+$ $f(\lambda x) = \lambda f(x)$
- shift-equivariant if  $\forall x\in \mathbb{R}^n,\forall \mu \in \mathbb{R},f(x + \mu) = f(x) + \mu$
- normalization-equivariant if it is both scale-equivariant and shift-equivariant:
$$
\forall x \in \mathbb {R} ^ {n}, \forall \lambda \in \mathbb {R} _ {*} ^ {+}, \forall \mu \in \mathbb {R}, f (\lambda x + \mu) = \lambda f (x) + \mu ,
$$
where addition with the scalar shift  $\mu$  is applied element-wise.
Note that the scale-equivariance property is more often referred to as positive homogeneity in pure mathematics. Like linear maps that are completely determined by their values on a basis, the above described equivariant functions are actually entirely characterized by the values their take on specific subsets of  $\mathbb{R}^n$ , as stated by the following lemma (see proof in Appendix C.1).
Lemma 1 (Characterizations)  $f:\mathbb{R}^n\mapsto \mathbb{R}^m$  is entirely determined by its values on the:
- unit sphere  $\mathcal{S}$  of  $\mathbb{R}^n$  if it is scale-equivariant,
orthogonal complement of  $\operatorname{Span}(\mathbf{1}_n)$ , i.e.  $\operatorname{Span}(\mathbf{1}_n)^\perp$ , if it is shift-equivariant,
- intersection  $S \cap \operatorname{Span}(\mathbf{1}_n)^\perp$  if it is normalization-equivariant,
where  $\mathbf{1}_n$  denotes the all-ones vector of  $\mathbb{R}^n$ .
Finally, Lemma 2 highlights three basic equivariance-preserving mathematical operations that can be used as building blocks for designing neural network architectures (see proof in Appendix C.1).
Lemma 2 (Operations preserving equivariance) Let  $f$  and  $g$  be two equivariant functions of the same type (either in scale, shift or normalization). Then, subject to dimensional compatibility, all of the following functions are still equivariant:
$f\circ g$  (f composed with  $g$
-  $x \mapsto (f(x)^\top g(x)^\top)^\top$  (concatenation of  $f$  and  $g$ ),
-  $(1 - t)f + tg$  for all  $t \in \mathbb{R}$  (affine combination of  $f$  and  $g$ ).
3.2 Examples of normalization-equivariant conventional denoisers
A ("blind") denoiser is basically a function  $f: \mathbb{R}^n \mapsto \mathbb{R}^n$  which, given a noisy image  $y \in \mathbb{R}^n$ , tries to map the corresponding noise-free image  $x \in \mathbb{R}^n$ . Since scaling up an image by a positive factor  $\lambda$  or adding it up a constant shift  $\mu$  does not change its contents, it is natural to expect scale and shift equivariance, i.e. normalization equivariance, from the denoising procedure emulated by  $f$ . In image denoising, a majority of methods usually assume an additive white Gaussian noise model with variance  $\sigma^2$ . The corruption model then reads  $y \sim \mathcal{N}(x, \sigma^2 I_n)$ , where  $I_n$  denotes the identity matrix of size  $n$ , and the noise standard deviation  $\sigma > 0$  is generally passed as an additional argument to the denoiser ("non-blind" denoising). In this case, the augmented function  $f: (y, \sigma) \in \mathbb{R}^n \times \mathbb{R}_*^+ \mapsto \mathbb{R}^n$  is said normalization-equivariant if:
$$
\forall (y, \sigma) \in \mathbb {R} ^ {n} \times \mathbb {R} _ {*} ^ {+}, \forall \lambda \in \mathbb {R} _ {*} ^ {+}, \forall \mu \in \mathbb {R}, f (\lambda y + \mu , \lambda \sigma) = \lambda f (y, \sigma) + \mu , \tag {1}
$$
as, according to the laws of statistics,  $\lambda y + \mu \sim \mathcal{N}(\lambda x + \mu, (\lambda \sigma)^2 I_n)$ . In what follows, we give some well-known examples of traditional denoisers that are normalization-equivariant (see proofs in Appendix C.2).
Noise-reduction filters: The most rudimentary methods for image denoising are the smoothing filters, among which we can mention the averaging filter or the Gaussian filter for the linear filters and the median filter which is nonlinear. These elementary "blind" denoisers all implement a normalization-equivariant function. More generally, one can prove that a linear filter is normalization-equivariant if and only if its coefficients add up to 1. In others words, normalization-equivariant linear filters process images by affine combinations of pixels.
Patch-based denoising: The popular N(on)-L(ocal) M(eans) algorithm [5] and its variants [12, 20, 27] consist in computing, for each pixel, an average of its neighboring noisy pixels, weighted by the degree of similarity of the patches to which they belong. In other words, they process images by convex combinations of pixels. More precisely, NLM can be defined as:
$$
f _ {\mathrm {N L M}} (y, \sigma) _ {i} = \frac {1}{W _ {i}} \sum_ {y _ {j} \in \Omega \left(y _ {i}\right)} e ^ {- \frac {\| p \left(y _ {i}\right) - p \left(y _ {j}\right) \| _ {2} ^ {2}}{h ^ {2}}} y _ {j} \quad \text {w i t h} \quad W _ {i} = \sum_ {y _ {j} \in \Omega \left(y _ {i}\right)} e ^ {- \frac {\| p \left(y _ {i}\right) - p \left(y _ {j}\right) \| _ {2} ^ {2}}{h ^ {2}}} \tag {2}
$$
where  $y_{i}$  denotes the  $i^{th}$  component of vector  $y$ ,  $\Omega(y_{i})$  is the set of its neighboring pixels,  $p(y_{i})$  represents the vectorized patch centered at  $y_{i}$ , and the smoothing parameter  $h$  is proportional to  $\sigma$  as proposed by several authors [4, 12, 29]. Defined as such,  $f_{\mathrm{NLM}}$  is a normalization-equivariant function. More recently, N(on)-L(ocal) Ridge [18] proposes to process images by linear combinations of similar patches and achieves state-of-the-art performance in unsupervised denoising. When restricting the coefficients of the combinations to sum to 1, that is imposing affine combination constraints, the resulting algorithm encodes a normalization-equivariant function as well.
TV denoising: Total variation (TV) denoising [37] is finally one of the most famous image denoising algorithm, appreciated for its edge-preserving properties. In its original form [37], a TV denoiser is defined as a function  $f:\mathbb{R}^n\times \mathbb{R}_*^+ \mapsto \mathbb{R}^n$  that solves the following equality-constrained problem:
$$
f _ {\mathrm {T V}} (y, \sigma) = \underset {x \in \mathbb {R} ^ {n}} {\arg \min } \| x \| _ {\mathrm {T V}} \quad \text {s . t .} \quad \| y - x \| _ {2} ^ {2} = n \sigma^ {2} \tag {3}
$$
where  $\| x\|_{\mathrm{TV}}\coloneqq \| \nabla x\| _2$  is the total variation of  $x\in \mathbb{R}^n$ . Defined as such,  $f_{\mathrm{TV}}$  is a normalization-equivariant function.
Figure 1: Influence of normalization for deep-learning-based image denoising. The raw input data is a publicly available real-world noisy image of the Convallaria dataset [36]. "Blind" DnCNN [47] with official pre-trained weights is used for denoising and is applied on four different normalization intervals displayed in red, each of which being included in  $[0,1]$  over which it was learned. PSNR is calculated with the average of 100 independent noisy static acquisitions of the same sample (called ground truth). Interestingly, the straightforward interval  $[0,1]$  does not give the best results. Normalization intervals are (a)  $[0,1]$ , (b)  $[0.08,0.12]$ , (c)  $[0.48,0,52]$  and (d)  $[0.64,0.96]$ . In the light of the denoising results  $(b)-(c)$  and  $(b)-(d)$ , DnCNN is neither shift-equivariant, nor scale-equivariant.
Denoising with "blind" DnCNN
3.3 The case of neural networks
Deep learning hides a subtlety about normalization equivariance that deserves to be highlighted. Usually, the weights of neural networks are learned on a training set containing data all normalized to the same arbitrary interval  $[a_0, b_0]$ . This training procedure improves the performance and allows for more stable optimization of the model. At inference, unseen data are processed within the interval  $[a_0, b_0]$  via a  $a-b$  linear normalization with  $a_0 \leq a < b \leq b_0$  denoted  $\mathcal{T}_{a,b}$  and defined by:
$$
\mathcal {T} _ {a, b}: y \mapsto (b - a) \frac {y - \operatorname* {m i n} (y)}{\operatorname* {m a x} (y) - \operatorname* {m i n} (y)} + a. \tag {4}
$$
Note that this transform is actually the unique linear one with positive slope that exactly bounds the output to  $[a, b]$ . The data is then passed to the trained network and its response is finally returned to the original range via the inverse operator  $\mathcal{T}_{a,b}^{-1}$ . This proven pipeline is actually relevant in light of the following proposition.
Proposition 1  $\forall a < b \in \mathbb{R}, \forall f: \mathbb{R}^n \mapsto \mathbb{R}^m, \mathcal{T}_{a,b}^{-1} \circ f \circ \mathcal{T}_{a,b}$  is a normalization-equivariant function.
While normalization-equivariance appears to be solved, a question is still remaining: how to choose the hyperparameters  $a$  and  $b$  for a given function  $f$ ? Obviously, a natural choice for neural networks is to take the same parameters  $a$  and  $b$  as in the learning phase whatever the input image is, i.e.  $a = a_0$  and  $b = b_0$ , but are they really optimal? The answer to this question is generally negative. Figure 1 depicts an example of the phenomenon in image denoising, taken from a real-world application. In this example, the straightforward choice is largely sub-optimal. This suggests that there are always inherent performance leaks for deep neural networks due to the two degrees of freedom induced by the normalization (i.e., choice of  $a$  and choice of  $b$ ). In addition, this poor conditioning can be a source of confusion and misinterpretation in critical applications.
3.4 Categorizing image denoisers
Table 1 summarizes the equivariance properties of several popular denoisers, either conventional [5, 11, 15, 18, 37, 44] or deep learning-based [24, 26, 45-47]. Interestingly, if scale-equivariance is generally guaranteed for traditional denoisers, not all of them are equivariant to shifts. In particular, the widely used algorithms DCT [44] and BM3D [11] are sensitive to offsets, mainly because the hard thresholding function at their core is not shift-equivariant. Regarding the deep-learning-based networks, only DRUNet [46] is insensitive to scale because it is a bias-free convolutional neural
Table 1: Equivariance properties of several image denoisers (left: traditional, right: learning-based)
TV	NLM	NLR	DCT	BM3D	WNNM	DnCNN	NLRN	SwinIR	Restormer	DRUNet
Scale	✓	✓	✓	✓	✓	X	X	X	X	✓
Shift	✓	✓	✓	X	X	X	X	X	X	X
network with only ReLU activation functions [33]. In particular, all transformer models [7, 24, 26, 45], even bias-free, are not scale-equivariant due to their inherent attention-based modules. In the next section, we show how to adapt existing neural architectures to guarantee normalization-equivariance without loss of performance and study the resulting class of parameterized functions  $(f_{\theta})$ .
4 Design of Normalization-Equivariant Networks
4.1 Affine convolutions
To justify the introduction of a new type of convolutional layers, let us study one of the most basic neural network, namely the linear (parameterized) function  $f_{\Theta}:x\in \mathbb{R}^n\mapsto \Theta x$ , where parameters  $\Theta$  are a matrix of  $\mathbb{R}^{m\times n}$ . Indeed,  $f_{\Theta}$  can be interpreted as a dense neural network with no bias, no hidden layer and no activation function. Obviously,  $f_{\Theta}$  is always scale-equivariant, whatever the weights  $\Theta$ . As for the shift-equivariance, a simple calculation shows that:
$$
x \mapsto \Theta x \text {i s} \Leftrightarrow \forall x \in \mathbb {R} ^ {n}, \forall \mu \in \mathbb {R}, \Theta (x + \mu \mathbf {1} _ {n}) = \Theta x + \mu \mathbf {1} _ {m} \Leftrightarrow \Theta \mathbf {1} _ {n} = \mathbf {1} _ {m}. \tag {5}
$$
Therefore,  $f_{\Theta}$  is normalization-equivariant if and only if each row of matrix  $\Theta$  sums to 1. In other words, for the normalization-equivariance to hold, the rows of  $\Theta$  must encode weights of affine combinations. Transposing the demonstration to any convolutional neural network follows from the observation that a convolution from an input layer of size  $H \times W \times C$  to an output layer of size  $H' \times W' \times C'$  can always be represented with a dense connection by vectorizing the input and output layers. The elements of the  $C'$  convolutional kernels of size  $k \times k \times C$  each are then stored separately along the rows of the (sparse) transition matrix  $\Theta$  of size  $(H' \times W' \times C') \times (H \times W \times C)$ . Therefore, a convolutional layer preserves the normalization-equivariance if and only if the weights of the each convolutional kernel sums to 1. In the following, we call such convolutional layers "affine convolutions".
In order to guarantee the affine constraint on each convolutional kernel throughout the training phase, one possibility is to "telescope" the circular shifted version of an unconstrained kernel to itself (this way, the sum of the resulting trainable coefficients cancels out) and then add the inverse of the kernel size element-wise as a non-trainable offset. Despite this over-parameterized form (involving an extra degree of freedom), we found this solution to be easier to use in practice. Moreover, it ensures that all coefficients of the affine kernels follow the same law at initialization.
Since normalization-equivariance is preserved through function composition, concatenation and affine combination (see Lemma 2), a (linear) convolutional neural network composed of only affine convolutions with no bias and possibly skip or affine residual connections (trainable affine combination of two layers), is guaranteed to be normalization-equivariant, provided that padding is performed with existing features (reflect, replicate or circular padding for example). Obviously, in their current state, these neural networks are of little interest, as linear functions do not encode best-performing functions for many applications, image denoising being no exception. Nevertheless, based on such networks, we show in the next subsection how to introduce nonlinearities without breaking the normalization-equivariance.
4.2 Channel-wise sort pooling as a normalization-equivariant alternative to ReLU
The first idea that comes to mind is to apply a nonlinear activation function  $\varphi : \mathbb{R} \mapsto \mathbb{R}$  preserving normalization-equivariance after each affine convolution. In other words, we look for a nonlinear solution  $\varphi$  of the characteristic functional equation of normalization-equivariant functions (see Def. 1) for  $n = 1$ . Unfortunately, according to Prop. 2 (see proof in Appendix C.1 which is based on Lemma 1), the unique solution is the identity function which is linear. Therefore, activation functions that apply element-wise are to be excluded.
Figure 2: Illustration of the proposed alternative for replacing the traditional scheme "convolution + element-wise activation function" in convolutional neural networks: affine convolutions supersede ordinary ones by restricting the coefficients of each kernel to sum to one and the proposed sort pooling patterns introduce nonlinearities by sorting two by two the pre-activated neurons along the channels.
Proposition 2 Let  $\mathrm{NE}(n)$  be the set of normalization-equivariant functions from  $\mathbb{R}^n$  to  $\mathbb{R}^n$ .
$\mathrm{NE}(1) = \{x\mapsto x\}$  and
$$
\mathrm {N E} (2) = \left\{(x _ {1}, x _ {2}) \mapsto A \left( \begin{array}{c} x _ {1} \\ x _ {2} \end{array} \right) i f x _ {1} \leq x _ {2} e l s e B \left( \begin{array}{c} x _ {1} \\ x _ {2} \end{array} \right) \Big | A, B \in \mathbb {R} ^ {2 \times 2} s. t. A \mathbf {1} _ {2} = B \mathbf {1} _ {2} = \mathbf {1} _ {2} \right\}.
$$
To find interesting nonlinear functions, one needs to examine multi-dimensional activation functions, i.e. ones of the form  $\varphi : \mathbb{R}^n \mapsto \mathbb{R}^m$  with  $n \geq 2$ . In order to preserve the dimensions of the neural layers and to limit the computational costs, we focus on the case  $n = m = 2$ , meaning that  $\varphi$  processes pre-activated neurons by pairs. According to Prop. 2, the normalization-equivariant functions from  $\mathbb{R}^2$  to  $\mathbb{R}^2$  are parameterized by two matrices  $A, B \in \mathbb{R}^{2 \times 2}$  such that  $A\mathbf{1}_2 = B\mathbf{1}_2 = \mathbf{1}_2$  and apply a different (affine-constrained) linear mapping depending on whether or not the input is in ascending order. As long as  $A \neq B$ , the resulting function is nonlinear and makes it de facto a candidate to replace the conventional one-dimensional activation functions such as the popular ReLU (rectified linear unit) function. Interestingly, when arbitrarily choosing  $A$  and  $B$  to be the permutation matrices of  $\mathbb{R}^{2 \times 2}$ , the resulting normalization-equivariant function simply reads:
$$
\varphi : \left(x _ {1}, x _ {2}\right) \in \mathbb {R} ^ {2} \mapsto \binom {\min  \left(x _ {1}, x _ {2}\right)} {\max  \left(x _ {1}, x _ {2}\right)}, \tag {6}
$$
which is nothing else than the sorting function in  $\mathbb{R}^2$ . Clearly, it is among the simplest normalization-equivariant nonlinear function from  $\mathbb{R}^2$  to  $\mathbb{R}^2$  and it is the one we consider as surrogate for the one-dimensional activation functions (choosing other functions, that is considering other choices for  $A$  and  $B$ , does not bring improvements in terms of performance in our experiments). More generally, it is easy to show that all the sorting functions of  $\mathbb{R}^n$  are normalization-equivariant and are nonlinear as soon as  $n \geq 2$ . Note that such sorting operators have been promoted by [2, 9] in totally different contexts for their norm-preserving properties of the backpropagated gradients.
Since the sorting function (6) is to be applied on non-overlapping pairs of neurons, the partitioning of layers needs to be determined. In order not to mix unrelated neurons, we propose to apply this two-dimensional activation function channel-wisely across layers and call this operation "sort pooling" in reference to the max pooling operation, widely used for downsampling, and from which it can be effectively implemented. Figure 2 illustrates the sequence of the two proposed innovations, namely affine convolution followed by channel-wise sort pooling, to replace the traditional scheme "conv+ReLU", while guaranteeing normalization-equivariance.
4.3 Encoding adaptive affine filters
Based on Lemma 2, we can formulate the following proposition which tells more about the class of parameterized functions  $(f_{\theta})$  encoded by the proposed networks.