SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is probably room for improvement. Surprisingly, the difference in speed between all variants is much less pronounced on CPU. In particular, the inference pass takes only about  $20\%$  longer for the normalization-equivariant DRUNet on CPU.
C Mathematical proofs for normalization-equivariant neural networks
C.1 Proofs of Lemmas and Propositions
Lemma 1(Characterizations)
Proof: For each type of equivariance, both existence and uniqueness of  $f$  must be proven. Let  $\mathbf{0}_n$  be the zero vector of  $\mathbb{R}^n$  and  $(y_x)_{x \in \mathcal{C}}$  the values that  $f$  takes on its characteristic set  $\mathcal{C}$ .
Scale-equivariance:
- Uniqueness: Let  $f$  and  $g$  two scale-equivariant functions such that  $\forall x \in S, f(x) = g(x)$ . First of all, for any scale-equivariant function  $h$ ,  $h(\mathbf{0}_n) = h(2 \cdot \mathbf{0}_n) = 2h(\mathbf{0}_n)$ , hence  $h(\mathbf{0}_n) = \mathbf{0}_m$ . Therefore,  $f(\mathbf{0}_n) = g(\mathbf{0}_n) = \mathbf{0}_m$ .
Let  $x \in \mathbb{R}^n \setminus \{\mathbf{0}_n\}$ . As  $\frac{x}{\|x\|} \in S$ , we have  $f(\frac{x}{\|x\|}) = g(\frac{x}{\|x\|}) \Rightarrow \frac{1}{\|x\|} f(x) = \frac{1}{\|x\|} g(x) \Rightarrow f(x) = g(x)$ . Finally,  $f = g$ .
- Existence: Let  $f: x \in \mathbb{R}^n \mapsto \left\{ \begin{array}{ll} \|x\| \cdot y_{\frac{x}{\|\boldsymbol{x}\|}} & \text{if } x \neq \mathbf{0}_n \\ \mathbf{0}_m & \text{otherwise} \end{array} \right.$ . Note that  $\forall x \in S, f(x) = y_x$ .
Let  $x \in \mathbb{R}^n$  and  $\lambda \in \mathbb{R}_*^+$ . If  $x \neq \mathbf{0}_n$ ,  $f(\lambda x) = \| \lambda x\| \cdot y_{\frac{\lambda x}{\|\lambda x\|}} = \lambda \| x\| \cdot y_{\frac{x}{\|x\|}} = \lambda f(x)$  and if  $x = \mathbf{0}_n$ ,  $f(\lambda x) = \mathbf{0}_m = \lambda f(x)$ , hence  $f$  is scale-equivariant.
Shift-equivariance:
- Uniqueness: Let  $f$  and  $g$  two shift-equivariant functions such that  $\forall x \in \operatorname{Span}(\mathbf{1}_n)^\perp$ ,  $f(x) = g(x)$ . Let  $x \in \mathbb{R}^n$ . By orthogonal decomposition of  $\mathbb{R}^n$  into  $\operatorname{Span}(\mathbf{1}_n)^\perp$  and  $\operatorname{Span}(\mathbf{1}_n)$ :
$$
\exists ! \left(x _ {1}, x _ {2}\right) \in \operatorname {S p a n} \left(\mathbf {1} _ {n}\right) ^ {\perp} \times \operatorname {S p a n} \left(\mathbf {1} _ {n}\right), x = x _ {1} + x _ {2}.
$$
Then,  $f(x) = f(x_{1} + x_{2}) = f(x_{1}) + x_{2} = g(x_{1}) + x_{2} = g(x_{1} + x_{2}) = g(x)$ .
- Existence: Let  $f : x \in  {\mathbb{R}}^{n} \mapsto  {y}_{{x}_{1}} + {x}_{2}$  ,where  $x = {x}_{1} + {x}_{2}$  is the unique decomposition such that  ${x}_{1} \in  \operatorname{Span}\left( {\mathbf{1}}_{n}\right)  \bot$  and  ${x}_{2} \in  \operatorname{Span}\left( {\mathbf{1}}_{n}\right)$  . Note that  $\forall x \in  \operatorname{Span}\left( {\mathbf{1}}_{n}\right)  \bot  ,f\left( x\right)  = {y}_{x}$  . Let  $x \in  {\mathbb{R}}^{n}$  and  $\mu  \in  \mathbb{R}.f\left( {x + \mu }\right)  = {y}_{{x}_{1}} + {x}_{2} + \mu {\mathbf{1}}_{m} = f\left( x\right)  + \mu$  as if  $x$  orthogonally decomposes into  ${x}_{1} + {x}_{2}$  with  ${x}_{1} \in  \operatorname{Span}\left( {\mathbf{1}}_{n}\right)  \bot$  and  ${x}_{2} \in  \operatorname{Span}\left( {\mathbf{1}}_{n}\right)$  ,then  $x + \mu$  orthogonally decomposes into  ${x}_{1} + \left( {{x}_{2} + \mu {\mathbf{1}}_{m}}\right) .f$  is then shift-equivariant.
Normalization-equivariance:
- Uniqueness: Let  $f$  and  $g$  two normalization-equivariant functions such that  $\forall x \in S \cap \operatorname{Span}(\mathbf{1}_n)^\perp, f(x) = g(x)$ . First, as  $f$  and  $g$  are a fortiori scale-equivariant,  $f(\mathbf{0}_n) = g(\mathbf{0}_n) = \mathbf{0}_m$ . Let  $x \in \mathbb{R}^n \setminus \{\mathbf{0}_n\}$ . By orthogonal decomposition of  $\mathbb{R}^n$  into  $\operatorname{Span}(\mathbf{1}_n)^\perp$  and  $\operatorname{Span}(\mathbf{1}_n)$ :
If  $x_{1} = \mathbf{0}_{n}$ ,  $f(x) = f(\mathbf{0}_{n} + x_{2}) = f(\mathbf{0}_{n}) + x_{2} = \mathbf{0}_{m} + x_{2} = x_{2}$ . Likewise,  $g(x) = x_{2}$ , hence  $f(x) = g(x)$ . Else, if  $x_{1} \neq \mathbf{0}_{n}$ ,  $f(x) = f(x_{1} + x_{2}) = f(x_{1}) + x_{2} = \|x_{1}\|f\left(\frac{x_{1}}{\|x_{1}\|}\right) + x_{2} = \|x_{1}\|g\left(\frac{x_{1}}{\|x_{1}\|}\right) + x_{2} = g(x_{1}) + x_{2} = g(x_{1} + x_{2}) = g(x)$ , as  $\frac{x_{1}}{\|x_{1}\|} \in S \cap \operatorname{Span}(\mathbf{1}_{n})^{\perp}$ . Finally,  $f = g$ .
- Existence: Let  $f : x \in  {\mathbb{R}}^{n} \mapsto  \left\{  \begin{array}{ll} {\begin{Vmatrix}{x}_{1}\end{Vmatrix}}^{2} \cdot  {y}_{\frac{{x}_{1}}{{\begin{Vmatrix}{x}_{1}\end{Vmatrix}}} + {x}_{2} & \text{ if }{x}_{1} \neq  {\mathbf{0}}_{n} \\  {x}_{2} & \text{ otherwise } \end{Vmatrix}$  ,where  $x = {x}_{1} + {x}_{2}$
is the unique decomposition such that  $x_{1} \in \operatorname{Span}(\mathbf{1}_{n})^{\perp}$  and  $x_{2} \in \operatorname{Span}(\mathbf{1}_{n})$ . Note that  $\forall x \in S \cap \operatorname{Span}(\mathbf{1}_{n})^{\perp}, f(x) = y_{x}$ . Let  $x \in \mathbb{R}^{n}$ ,  $\lambda \in \mathbb{R}_{*}^{+}$  and  $\mu \in \mathbb{R}$ .  $x$  decomposes
orthogonally into  $x_{1} + x_{2}$  with  $x_{1} \in \operatorname{Span}(\mathbf{1}_{n})^{\perp}$  and  $x_{2} \in \operatorname{Span}(\mathbf{1}_{n})$ , and we have  $f(\lambda x + \mu) = f(\lambda x_{1} + (\lambda x_{2} + \mu))$ , where  $\lambda x_{1} + (\lambda x_{2} + \mu)$  is the orthogonal decomposition of  $\lambda x + \mu$  into  $\operatorname{Span}(\mathbf{1}_{n})^{\perp}$  and  $\operatorname{Span}(\mathbf{1}_{n})$ .
If  $x_{1} = \mathbf{0}_{n}$ , then  $\lambda x_{1} = \mathbf{0}_{n}$  and  $f(\lambda x + \mu) = \lambda x_{2} + \mu = \lambda f(x) + \mu$ .
Else, if  $x_{1} \neq \mathbf{0}_{n}$ , then  $\lambda x_{1} \neq \mathbf{0}_{n}$ , and  $f(\lambda x + \mu) = \| \lambda x_{1} \| \cdot y_{\frac{\lambda x_{1}}{\|\lambda x_{1}\|}} + (\lambda x_{2} + \mu) = \lambda (\| x_{1} \| \cdot y_{\frac{x_{1}}{\|\lambda x_{1}\|}} + x_{2}) + \mu = \lambda f(x) + \mu$ . Finally,  $f$  is normalization-equivariant.
Lemma 2 (Operations preserving equivariance)
Proof: Let  $x\in \mathbb{R}^n$ $\lambda \in \mathbb{R}_*^+$  and  $\mu \in \mathbb{R}$
- If  $f$  and  $g$  are both scale-equivariant,  $(f \circ g)(\lambda x) = f(g(\lambda x)) = f(\lambda g(x)) = \lambda f(g(x)) = \lambda (f \circ g)(x)$  and if they are both shift-equivariant,  $(f \circ g)(x + \mu) = f(g(x + \mu)) = f(g(x) + \mu) = f(g(x)) + \mu = (f \circ g)(x) + \mu$ .
- Let  $h: x \mapsto (f(x)^\top g(x)^\top)^\top$ . If  $f$  and  $g$  are both scale-equivariant,  $h(\lambda x) = (f(\lambda x)^\top g(\lambda x)^\top)^\top = (\lambda f(x)^\top \lambda g(x)^\top)^\top = \lambda h(x)$  and if they are both shift-equivariant,  $h(x + \mu) = (f(x + \mu)^\top g(x + \mu)^\top)^\top = (f(x)^\top + \mu \quad g(x)^\top + \mu)^\top = h(x) + \mu$ .
- Let  $t \in \mathbb{R}$  and  $h: x \mapsto (1 - t)f + tg$ . If  $f$  and  $g$  are both scale-equivariant,  $h(\lambda x) = (1 - t)f(\lambda x) + tg(\lambda x) = (1 - t)\lambda f(x) + t\lambda g(x) = \lambda ((1 - t)f(x) + tg(x)) = \lambda h(x)$  and if they are both shift-equivariant,  $h(x + \mu) = (1 - t)f(x + \mu) + tg(x + \mu) = (1 - t)(f(x) + \mu) + t(g(x) + \mu) = (1 - t)f(x) + tg(x) + (1 - t)\mu + t\mu = h(x) + \mu$ .
Proposition 1
Proof: Let  $a < b \in \mathbb{R}$ ,  $f: \mathbb{R}^n \mapsto \mathbb{R}^m$ ,  $x \in \mathbb{R}^n$ ,  $\lambda \in \mathbb{R}_*^+$  and  $\mu \in \mathbb{R}$ .
We have  $\mathcal{T}_{a,b}(\lambda x + \mu) = (b - a)\frac{\lambda x + \mu - \min(\lambda x + \mu)}{\max(\lambda x + \mu) - \min(\lambda x + \mu)} + a = (b - a)\frac{x - \min(x)}{\max(x) - \min(x)} + a = \mathcal{T}_{a,b}(x)$  (i.e.  $\mathcal{T}_{a,b}$  is normalization-invariant).  $\mathcal{T}_{a,b}^{-1}$  denotes the inverse transformation intricately linked to the input  $x$  of  $\mathcal{T}_{a,b}$  (note that this is an improper notation as  $\mathcal{T}_{a,b}$  is not bijective). Thus, if  $x$  is the input of  $\mathcal{T}_{a,b}$ , then  $\mathcal{T}_{a,b}^{-1}: y \mapsto (\max(x) - \min(x))\frac{y - a}{b - a} + \min(x)$ .
$$
\begin{array}{l} \left(\mathcal {T} _ {a, b} ^ {- 1} \circ f \circ \mathcal {T} _ {a, b}\right) (\lambda x + \mu) = \frac {\operatorname* {m a x} (\lambda x + \mu) - \operatorname* {m i n} (\lambda x + \mu)}{b - a} \left((f \circ \mathcal {T} _ {a, b}) (\lambda x + \mu) - a\right) + \operatorname * {m i n} (\lambda x + \mu), \\ = \lambda \frac {\operatorname* {m a x} (x) - \operatorname* {m i n} (x)}{b - a} \left(\left(f \circ \mathcal {T} _ {a, b}\right) (\lambda x + \mu) - a\right) + \lambda \operatorname * {m i n} (x) + \mu , \\ = \lambda \left(\frac {\operatorname* {m a x} (x) - \operatorname* {m i n} (x)}{b - a} \left(\left(f \circ \mathcal {T} _ {a, b}\right) (x) - a\right) + \operatorname * {m i n} (x)\right) + \mu , \\ = \lambda \left(\mathcal {T} _ {a, b} ^ {- 1} \circ f \circ \mathcal {T} _ {a, b}\right) (x) + \mu . \\ \end{array}
$$
Finally,  $\mathcal{T}_{a,b}^{-1}\circ f\circ \mathcal{T}_{a,b}$  is normalization-equivariant.
Proposition 2
Proof: Let  $\mathrm{id}: x \in \mathbb{R} \mapsto x$  be the identity function.  $\mathrm{id}$  is a normalization-equivariant function so  $\{x \mapsto x\} \subseteq \mathrm{NE}(1)$ . Reciprocally, let  $f \in \mathrm{NE}(1)$ . By scale-equivariance,  $f(0) = f(2 \times 0) = 2f(0)$ , hence  $f(0) = 0$ . By shift-equivariance,  $\forall x \in \mathbb{R}$ ,  $f(x) = f(x + 0) = f(0) + x = x$ , hence,  $f = \mathrm{id}$ . Finally,  $\mathrm{NE}(1) \subseteq \{x \mapsto x\}$ , hence  $\mathrm{NE}(1) = \{x \mapsto x\}$ . Note that it is coherent with Lemma 1 which states that  $f$  is entirely determined by its values on  $S \cap \operatorname{Span}(\mathbf{1}_n)^{\perp}$ , which reduces to the empty set for  $n = 1$ .
Let  $F = \left\{(x_{1},x_{2})\mapsto A\binom{x_{1}}{x_{2}}\text{if} x_{1}\leq x_{2}\text{else} B\binom{x_{1}}{x_{2}}\Bigg{|}A,B\in \mathbb{R}^{2\times 2}\text{s.t.} A\mathbf{1}_{2} = B\mathbf{1}_{2} = \mathbf{1}_{2}\right\}$
and  $f\in F$  . Let  $x\in \mathbb{R}^2$ $\lambda >0$  and  $\mu \in \mathbb{R}$
$$
f (\lambda x + \mu) = \left\{ \begin{array}{l l} A (\lambda x + \mu) & \text {i f} \lambda x _ {1} + \mu \leq \lambda x _ {2} + \mu \\ B (\lambda x + \mu) & \text {o t h e r w i s e} \end{array} \right. = \left\{ \begin{array}{l l} \lambda A x + \mu & \text {i f} x _ {1} \leq x _ {2} \\ \lambda B x + \mu & \text {o t h e r w i s e} \end{array} \right. = \lambda f (x) + \mu ,
$$
hence  $f\in \mathrm{NE}(2)$
Reciprocally, let  $f \in \mathrm{NE}(2)$ . For  $n = 2$  and when considering the Euclidean distance,  $S \cap \operatorname{Span}(\mathbf{1}_n)^\perp = \{-u, u\}$  with  $u = (-1/\sqrt{2}, 1/\sqrt{2})$ . Let
$$
A = \frac {1}{u _ {2} - u _ {1}} \left( \begin{array}{c c} u _ {2} - f (u) _ {1} & f (u) _ {1} - u _ {1} \\ u _ {2} - f (u) _ {2} & f (u) _ {2} - u _ {1} \end{array} \right) \text {a n d} B = \frac {1}{u _ {2} - u _ {1}} \left( \begin{array}{c c} u _ {2} + f (- u) _ {1} & - f (- u) _ {1} - u _ {1} \\ u _ {2} + f (- u) _ {2} & - f (- u) _ {2} - u _ {1} \end{array} \right).
$$
Let  $g:(x_1,x_2)\mapsto A\left( \begin{array}{c}x_1\\ x_2 \end{array} \right)$  if  $x_{1}\leq x_{2}$  else  $B\left( \begin{array}{c}x_{1}\\ x_{2} \end{array} \right)$ . We have  $g\in F$  since  $A\mathbf{1}_2 = B\mathbf{1}_2 = \mathbf{1}_2$  (in particular  $g$  is then normalization-equivariant) and  $\forall x\in \{-u,u\} ,g(x) = f(x)$ . According to Lemma 1,  $g = f$ , hence  $f\in F$ . Finally,  $\mathrm{NE}(2)\subseteq F$ , hence  $\mathrm{NE}(2) = F$ .
Proposition 3
Proof:  $f_{\theta}^{\mathrm{NE}}$  is composed of three types of building blocks of the following form:
- affine convolutions:  $a_{\Theta}:x\in \mathbb{R}^n\mapsto \Theta x$  with  $\Theta \in \mathbb{R}^{m\times n}$  subject to  $\Theta \mathbf{1}_n = \mathbf{1}_m$
- sort pooling nonlinearities: sortpool:  $\mathbb{R}^n\mapsto \mathbb{R}^n$
- max pooling layers: maxpool:  $\mathbb{R}^n\mapsto \mathbb{R}^m$  with  $m < n$
which are assembled using:
- function compositions:  $\mathrm{comp}(f,g)\mapsto f\circ g$
- skip connections:  $\operatorname{skip}(f, g) \mapsto (x \mapsto (f(x)^{\top} g(x)^{\top})^{\top})$ ,
- affine residual connections:  $\operatorname{ares}_t(f, g) \mapsto (1 - t)f + tg$  with  $t \in \mathbb{R}$ .
Note that the rows of  $\Theta$  in  $a_{\Theta}$  encode the convolution kernels in a CNN and the trainable parameters, denoted by  $\theta$ , are only composed of matrices  $\Theta$  and scalars  $t$ . Moreover, note that average pooling layers are nothing else than affine convolutions with fixed parameters.
Since  $a_{\Theta}$ , sortpool and maxpool are normalization-equivariant functions, Lemma 2 states that the resulting function  $f_{\theta}^{\mathrm{NE}}$  is also normalization-equivariant. Moreover, since they are continuous and the assembling operators preserve continuity,  $f_{\theta}^{\mathrm{NE}}$  is continuous. Then, for a given input  $x \in \mathbb{R}^n$ , we have (sortpool  $\circ a_{\Theta}(x) = a_{\pi(\Theta)}(x) = \pi(\Theta)x$ , where  $\pi$  an operator acting on matrix  $\Theta$  by permuting its rows (note that the permutation  $\pi$  is both dependent on  $x$  and  $\Theta$ ). Therefore, applying a pattern "conv affine + sortpool" simply amounts locally to a linear transformation. Moreover, since applying a max pooling layer amounts to removing some rows from matrix  $\Theta$ , the local linear behavior is preserved. Thus, as the nonlinearities of  $f_{\theta}^{\mathrm{NE}}$  are exclusively brought by sort pooling patterns (and possibly max pooling layers),  $f_{\theta}^{\mathrm{NE}}$  is actually locally linear. In other words,  $f_{\theta}^{\mathrm{NE}}$  is piecewise-linear. Moreover, as there is a finite number (although high) of possible permutations (and possibly eliminations) of the rows of all matrices  $\Theta$ ,  $f_{\theta}^{\mathrm{NE}}$  has finitely many pieces. Finally, on each piece represented by the vector  $y_r$ ,  $f_{\theta}^{\mathrm{NE}}(y) = A_{\theta}^{y_r}y$ . It remains to prove that  $A_{\theta}^{y_r}\mathbf{1}_n = \mathbf{1}_m$ . But this property is easily obtained by noticing that, subject to dimensional compatibility on matrices  $\Theta$ :
-  $\Theta \mathbf{1}_n = \mathbf{1}_m \Rightarrow \pi(\Theta) \mathbf{1}_n = \mathbf{1}_m$  ("conv affine + sortpool"),
-  $\Theta \mathbf{1}_n = \mathbf{1}_m \Rightarrow \rho (\Theta) \mathbf{1}_n = \mathbf{1}_l$  ("conv affine + maxpool") where  $\rho$  removes some rows,
-  $\Theta_{1}\mathbf{1}_{n} = \mathbf{1}_{m}$  and  $\Theta_{2}\mathbf{1}_{m} = \mathbf{1}_{l}\Rightarrow \Theta_{2}\Theta_{1}\mathbf{1}_{n} = \mathbf{1}_{l}$  (composition),
-  $\Theta_{1}\mathbf{1}_{n_{1}} = \mathbf{1}_{m_{1}}$  and  $\Theta_{2}\mathbf{1}_{n_{2}} = \mathbf{1}_{m_{2}}\Rightarrow \binom{\Theta_{1}}{\Theta_{2}}\mathbf{1}_{n_{1} + n_{2}} = \mathbf{1}_{m_{1} + m_{2}}$  (skip connection),
-  $\Theta_{1}\mathbf{1}_{n} = \mathbf{1}_{m}$  and  $\Theta_{2}\mathbf{1}_{n} = \mathbf{1}_{m} \Rightarrow (1 - t)\Theta_{1}\mathbf{1}_{n} + t\Theta_{2}\mathbf{1}_{n} = \mathbf{1}_{m}$  (affine residual connection).
Thus, the affine combinations are preserved all along the layers of  $f_{\theta}^{\mathrm{NE}}$ . In the end,
$$
f _ {\theta} ^ {\mathrm {N E}} (y) = A _ {\theta} ^ {y _ {r}} y, \text {w i t h} A _ {\theta} ^ {y _ {r}} \in \mathbb {R} ^ {m \times n} \text {s u c h t h a t} A _ {\theta} ^ {y _ {r}} \mathbf {1} _ {n} = \mathbf {1} _ {m}.
$$
C.2 Examples of normalization-equivariant conventional denoisers
Noise-reduction filters: All linear smoothing filters can be put under the form  $f_{\Theta}: x \in \mathbb{R}^n \mapsto \Theta x$  with  $\Theta \in \mathbb{R}^{n \times n}$  (the rows of  $\Theta$  encode the convolution kernel). Obviously,  $f_{\Theta}$  is always scale-equivariant, whatever the filter  $\Theta$ . As for the shift-equivariance, a simple calculation shows that:
$x\mapsto \Theta x$  is shift-equivariant  $\Leftrightarrow \forall x\in \mathbb{R}^n,\forall \mu \in \mathbb{R},\Theta (x + \mu \mathbf{1}_n) = \Theta x + \mu \mathbf{1}_m\Leftrightarrow \Theta \mathbf{1}_n = \mathbf{1}_m$
Since the sum of the coefficients of a Gaussian kernel and an averaging kernel is one, we have  $\Theta \mathbf{1}_n = \mathbf{1}_m$ , hence these linear filters are normalization-equivariant. The median filter is also normalization-equivariant because  $\mathrm{median}(\lambda x + \mu) = \lambda \mathrm{median}(x) + \mu$  for  $\lambda \in \mathbb{R}_{*}^{+}$  and  $\mu \in \mathbb{R}$ .
Patch-based denoising:
- NLM [5]: Assuming that the smoothing parameter  $h$  is proportional to  $\sigma$ , i.e.  $h = \alpha \sigma$ , we have  $e^{-\frac{\|p(\lambda y_i + \mu) - p(\lambda y_j + \mu)\|_2^2}{(\alpha\lambda\sigma)^2}} = e^{-\frac{\lambda^2\|p(y_i) - p(y_j)\|_2^2}{\lambda^2(\alpha\sigma)^2}} = e^{-\frac{\|p(y_i) - p(y_j)\|_2^2}{h^2}}$ , hence the aggregation weights are normalization-invariant. Then,
$$
\begin{array}{l} f _ {\mathrm {N L M}} (\lambda y + \mu , \lambda \sigma) _ {i} = \frac {1}{W _ {i}} \sum_ {y _ {j} \in \Omega (y _ {i})} e ^ {- \frac {\| p (y _ {i}) - p (y _ {j}) \| _ {2} ^ {2}}{h ^ {2}}} (\lambda y _ {j} + \mu) \text {w i t h} W _ {i} = \sum_ {y _ {j} \in \Omega (y _ {i})} e ^ {- \frac {\| p (y _ {i}) - p (y _ {j}) \| _ {2} ^ {2}}{h ^ {2}}}, \\ = \frac {1}{W _ {i}} \sum_ {y _ {j} \in \Omega (y _ {i})} e ^ {- \frac {\| p (y _ {i}) - p (y _ {j}) \| _ {2} ^ {2}}{h ^ {2}}} \lambda y _ {j} + \frac {1}{W _ {i}} \sum_ {y _ {j} \in \Omega (y _ {i})} e ^ {- \frac {\| p (y _ {i}) - p (y _ {j}) \| _ {2} ^ {2}}{h ^ {2}}} \mu , \\ = \lambda \left(\frac {1}{W _ {i}} \sum_ {y _ {j} \in \Omega (y _ {i})} e ^ {- \frac {\| p (y _ {i}) - p (y _ {j}) \| _ {2} ^ {2}}{h ^ {2}}} y _ {j}\right) + \mu , \\ = \lambda f _ {\mathrm {N L M}} (y, \sigma) _ {i} + \mu . \\ \end{array}
$$
Finally,  $f_{\mathrm{NLM}}$  is a normalization-equivariant function.
- NL-Ridge [18]: The block-matching procedure at the heart of NL-Ridge is normalization-invariant as it is based on comparisons of the  $\ell_2$  norm of the difference of image patches. For each noisy patch group, a.k.a. similarity matrix,  $Y\in \mathbb{R}^{n\times k}$  composed of  $k$  vectorized similar patches of size  $n$ , the optimal weights  $\Theta^{*}\in \mathbb{R}^{k\times k}$ , in the  $\ell_2$  risk sense, are computed such that  $Y\Theta^{*}$  is as close as possible to the (unknown) clean patch group  $X\in \mathbb{R}^{n\times k}$ . The two successive minimization problems approximating  $\Theta$  under affine constraints  $\mathcal{C} = \{\Theta \in \mathbb{R}^{k\times k},\Theta^{\top}\mathbf{1}_k = \mathbf{1}_k\}$  can be put under the form:
$$
\Theta^ {*} = \arg \min _ {\Theta \in \mathcal {C}} \mathrm {t r} \left(\frac {1}{2} \Theta^ {\top} Q \Theta + C \Theta\right) = I _ {k} - n \sigma^ {2} \left[ Q ^ {- 1} - \frac {Q ^ {- 1} \mathbf {1} _ {k} (Q ^ {- 1} \mathbf {1} _ {k} ^ {\top})}{\mathbf {1} _ {k} ^ {\top} Q ^ {- 1} \mathbf {1} _ {k}} \right].
$$
with  $Q = Y^{\top}Y$  or  $Q = \hat{X}^{\top}\hat{X} +n\sigma^{2}I_{k}$  for the first and second step, respectively ( $\hat{X}$  is the patch group estimate obtained after the first step),  $C = n\sigma^{2}I_{k} - Q$  and where  $\operatorname{tr}$  denotes the trace operator. Depending on the step, we have:
$$
2 \operatorname {t r} \left(\frac {1}{2} \Theta^ {\top} Q \Theta + C \Theta\right) = \left\{ \begin{array}{l l} \| Y \Theta - Y \| _ {F} ^ {2} + 2 n \sigma^ {2} \operatorname {t r} (\Theta) + \mathrm {c o n s t} \\ \mathrm {o r} \\ \| \hat {X} \Theta - \hat {X} \| _ {F} ^ {2} + n \sigma^ {2} \| \Theta \| _ {F} ^ {2} + \mathrm {c o n s t} \end{array} \right.
$$
where  $\| .\| _F$  is the Frobenius norm. But, for any  $Z\in \mathbb{R}^{n\times k}$  and any function  $h:\Theta \in \mathbb{R}^{k\times k}\mapsto \mathbb{R}$
$$
\| (\lambda Z + \mu) \Theta - (\lambda Z + \mu) \| _ {F} ^ {2} + n (\lambda \sigma) ^ {2} h (\Theta) = \lambda^ {2} \left(\| Z \Theta - Z \| _ {F} ^ {2} + n \sigma^ {2} h (\Theta)\right),
$$
assuming that  $\Theta^{\top}\mathbf{1}_k = \mathbf{1}_k$ . Therefore, the aggregation weights  $\Theta^{*}$  are normalization-invariant and  $(\lambda Y + \mu)\Theta^{*} = \lambda Y\Theta^{*} + \mu$ . Finally, NL-Ridge with affine constraints encodes a normalization-equivariant function.
TV denoising: Let  $y \in \mathbb{R}^n$ ,  $\lambda \in \mathbb{R}_*^+$  and  $\mu \in \mathbb{R}$ . Let  $x^* = \operatorname*{arg  min}_{x \in \mathbb{R}^n} \| x\|_{\mathrm{TV}}$  s.t.  $\| y - x\|_2^2 = n\sigma^2$  be the solution of TV [37].
$$
f_{\mathrm{TV}}(\lambda y + \mu ,\lambda \sigma) = \operatorname *{arg  min}_{x\in \mathbb{R}^{n}}\| x\|_{\mathrm{TV}}\quad \text{s.t.}\quad \| \lambda y + \mu -x\|_{2}^{2} = n(\lambda \sigma)^{2},
$$
$$
\begin{array}{l} = \operatorname * {a r g   m i n} _ {x \in \mathbb {R} ^ {n}} \lambda \left\| \frac {x - \mu}{\lambda} \right\| _ {\mathrm {T V}} \quad \text {s . t .} \quad \lambda^ {2} \left\| y - \frac {x - \mu}{\lambda} \right\| _ {2} ^ {2} = \lambda^ {2} n \sigma^ {2}, \\ = \operatorname * {a r g   m i n} _ {x \in \mathbb {R} ^ {n}} \left\| \frac {x - \mu}{\lambda} \right\| _ {\mathrm {T V}} \quad \text {s . t .} \quad \left\| y - \frac {x - \mu}{\lambda} \right\| _ {2} ^ {2} = n \sigma^ {2}, \\ = \lambda x ^ {*} + \mu , \\ = \lambda f _ {\mathrm {T V}} (y, \sigma) + \mu . \\ \end{array}
$$
Finally,  $f_{\mathrm{TV}}$  is a normalization-equivariant function.
Table 5: Real-world image denoising on Darmstadt Noise Dataset (DND) with raw data and variance-stabilizing transformation (VST). All "non-blind" methods were trained solely on synthetic white Gaussian noise.
Quality metric	PSNR ↑	SSIM ↑	Quality metric	PSNR ↑	SSIM ↑
DRUNet	ordinary	47.58	0.9762	FDnCNN	ordinary	47.37	0.9754
scale-equiv	47.59	0.9763	scale-equiv	47.31	0.9754
norm-equiv	47.57	0.9762	norm-equiv	47.35	0.9753
Figure 6: Comparison of the performance of our normalization-equivariant FDnCNN [47] with its scale-equivariant and ordinary counterparts for Gaussian denoising on the Set12 dataset. The vertical blue line indicates the unique noise level on which the networks were trained exclusively (from left to right:  $\sigma = 50$ ,  $\sigma = 25$  and  $\sigma = 10$ ). In all cases, normalization-equivariant networks generalize much more robustly beyond the training noise level.
(a) Uniform
(b) Laplace
(c) Rayleigh
(d) JPEG
Figure 7: Comparison of the performance of our normalization-equivariant alternative with its scale-equivariant (under three different forms) and ordinary counterparts for different types of additive noises on the Set12 dataset with "blind" FDnCNN architecture. The vertical blue line indicates the unique noise level on which the "blind" networks were trained exclusively. Note that JPEG noise (d) (i.e. JPEG artifacts) is treated with the networks of (a), assimilating JPEG noise with uniform noise (similar outcomes can be achieved when using the networks specialized for Laplace noise (b)).
Table 6: Test error of different variants of the same VGG8b architecture for image classification. Note that norm-equivariance is with respect to the classification vector and becomes norm-invariance after label selection via argmax. See [34] for details about architecture, datasets and training.
Dataset	MNIST	Kuzushiji-MNIST	Fashion-MNIST
VGG8b	ordinary	0.26%	1.53%	4.53%
scale-equiv	0.37%	1.52%	5.01%
norm-equiv	0.44%	2.78%	6.57%
Note that for norm-equiv variants, learning rate is initialized to  $3e - 5$  instead of  $5e - 4$  and dropout rate is halved.
12 11 b
NOISY
SCALE-EQUIVARIANT
Denoised
Adaptive filters
NORMALIZATION-EQUIVARIANT
Denoised
1 b
Figure 8: Visual comparisons of the generalization capabilities of a scale-equivariant FDnCNN [47] (left) and its normalization-equivariant counterpart (right) for Gaussian noise. Both networks were trained for Gaussian noise at noise level  $\sigma = 25$  exclusively. The adaptive filters (rows of  $A_{\theta}^{y_r}$  in Prop. 3) are indicated for two particular pixels as well as the sum of their coefficients (note that some weights are negative, indicated in red). The scale-equivariant network tends to excessively smooth out the image when evaluated at a lower noise level, whereas the normalization-equivariant network is more adaptable and considers the underlying texture to a greater extent.