SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} G (\boldsymbol {\theta}) = \nabla_ {\boldsymbol {\theta}} \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {y} ^ {\prime} \| _ {2} ^ {2} \\ = \nabla_ {\pmb {\theta}} \| f _ {\pmb {\theta}} (\mathbf {y}) - (\mathbf {x} + \mathbf {e}) \| _ {2} ^ {2} \\ = \nabla_ {\boldsymbol {\theta}} \left(\| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} - 2 \langle f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x}, \mathbf {e} \rangle + \| \mathbf {e} \| _ {2} ^ {2}\right) \\ = \nabla_ {\pmb {\theta}} \frac {1}{2} \| f _ {\pmb {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} - \mathbf {J} _ {\pmb {\theta}, \mathbf {y}} ^ {T} \mathbf {e}, \\ \end{array}
$$
where  $\mathbf{J}_{\theta ,\mathbf{y}}\in \mathbb{R}^{n\times p}$  is the Jacobian of the function  $f_{\theta}$  with respect to the parameters  $\pmb{\theta}$ . Note that the function  $f_{\theta}$  has parameters  $\pmb {\theta}\in \mathbb{R}^p$  and maps a vector in  $\mathbb{R}^n$  to another vector in  $\mathbb{R}^n$ ; we consider the
Jacobian with respect to the parameters, not with respect to the input vector. The stochastic gradient is an unbiased estimate of the gradient of the risk, i.e.,
$$
\mathbb {E} [ G (\mathbf {W}) ] = \nabla_ {\boldsymbol {\theta}} \mathbb {E} \left[ \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} \right] = \nabla R (\boldsymbol {\theta}). \tag {7}
$$
The variance of the stochastic gradient is
$$
\begin{array}{l} \mathbb {E} \left[ \| G (\boldsymbol {\theta}) \| _ {2} ^ {2} \right] = \mathbb {E} \left[ \left\| \nabla_ {\boldsymbol {\theta}} \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} - \mathbf {J} _ {\boldsymbol {\theta}, \mathbf {y}} ^ {T} \mathbf {e} \right\| _ {2} ^ {2} \right] \\ = \mathbb {E} \left[ \left\| \nabla_ {\boldsymbol {\theta}} \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} \right\| _ {2} ^ {2} \right] + \frac {\sigma_ {e} ^ {2}}{n} \| \mathbf {J} _ {\boldsymbol {\theta}, \mathbf {y}} \| _ {F} ^ {2}. \tag {8} \\ \end{array}
$$
We are now ready to prove Theorem 1. Recall that for the theorem, we consider a simple denoising problem where  $\mathbf{x} = \mathbf{U}\mathbf{c}$ ,  $\mathbf{c} \sim \mathcal{N}(0,1 / d\mathbf{I})$  and the measurement is  $\mathbf{y} = \mathbf{x} + \mathbf{z}$ , where  $\mathbf{z} \sim \bar{\mathbf{N}}(0,\sigma_z^2 / n\mathbf{I})$ . Also, we consider a simple linear estimator  $f(\mathbf{y}) = \mathbf{W}\mathbf{y}$  parameterized by the matrix  $\mathbf{W} \in \mathbb{R}^{n \times n}$ . For this setup, the risk becomes
$$
\begin{array}{l} R (\mathbf {W}) = \mathbb {E} \left[ \| \mathbf {W y} - \mathbf {x} \| _ {2} ^ {2} \right] \\ = \frac {1}{d} \| (\mathbf {W} - \mathbf {I}) \mathbf {U} \| _ {F} ^ {2} + \frac {\sigma_ {z} ^ {2}}{n} \| \mathbf {W} \| _ {F} ^ {2}. \\ \end{array}
$$
Thus, the risk is  $m$ -strongly convex with  $m = \frac{\sigma_z^2}{n}$ , since the risk is lower-bounded by the quadratic  $\frac{\sigma_z^2}{n}\| \mathbf{W}\| _F^2$ . Moreover, the gradient of the risk is
$$
\nabla R (\mathbf {W}) = \frac {1}{d} (\mathbf {W} - \mathbf {I}) \mathbf {U} \mathbf {U} ^ {T} + \frac {\sigma_ {z} ^ {2}}{n} \mathbf {W}, \tag {9}
$$
and setting this to zero yields the risk-minimizing parameter
$$
\mathbf {W} ^ {*} = \frac {1}{1 + \sigma_ {z} ^ {2} \frac {d}{n}} \mathbf {U U} ^ {T}.
$$
Below, we show that the second moment of the risk is  $(M,B)$ -bounded in that for all  $\mathbf{W}$ ,
$$
\mathbb {E} \left[ \| G (\mathbf {W}) \| _ {2} ^ {2} \right] \leq M ^ {2} \| \mathbf {W} - \mathbf {W} ^ {*} \| _ {F} ^ {2} + B ^ {2}, \tag {10}
$$
where  $\mathbf{W}^*$  is a minimizer of the risk  $R(\mathbf{W})$ , and where  $M = 10 / d$  and  $B = 12\sigma_z^2\frac{d}{n} +\sigma_e^2 (1 + \sigma_z^2)$ .
We proceed by applying the following standard result for the convergence of the SGM. For the sake of completeness, the result is proven below.
Lemma 1. Let  $f$  be  $m$ -strongly convex (i.e.,  $g(\pmb{\theta}) = f(\pmb{\theta}) - \frac{m}{2} \| \pmb{\theta} \|_2^2$  is convex) with minimizer  $\pmb{\theta}^*$  and let  $G$  be a stochastic gradient that is  $(M, B)$ -bounded. Assume we start the SGM iterates at  $\pmb{\theta}_0 = 0$ , with a stepsize  $\alpha_k = \frac{2}{m} \frac{1}{2 \frac{M^2}{m^2} + k}$ . Then
$$
\mathbb {E} \left[ \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \| _ {2} ^ {2} \right] \leq \frac {1}{k - 2} \frac {1}{m ^ {2}} \left(2 M ^ {2} e _ {0} + B ^ {2}\right).
$$
Below we show that for all  $\mathbf{W}$
$$
R (\mathbf {W}) - R \left(\mathbf {W} ^ {*}\right) \leq \left(1 / d + \sigma_ {z} ^ {2} / n\right) \| \mathbf {W} - \mathbf {W} ^ {*} \| _ {F} ^ {2}. \tag {11}
$$
Thus, an application of Lemma 1 to the RHS yields
$$
\begin{array}{l} R \left(\mathbf {W} _ {N}\right) - R \left(\mathbf {W} ^ {*}\right) \leq \left(1 / d + \sigma_ {z} ^ {2} / n\right) \frac {1}{k - 2} \left(2 \frac {M ^ {2}}{m ^ {2}} \| \mathbf {W} ^ {*} \| _ {F} ^ {2} + \frac {B ^ {2}}{m ^ {2}}\right) \\ \leq \frac {1 / d + \sigma_ {z} ^ {2} / n}{(\sigma_ {z} ^ {2} / n) ^ {2}} \frac {1}{k - 2} (2 + B ^ {2}), \\ \end{array}
$$
B.1 Bound on variance of stochastic gradient, proof of equation (8):
From equation (8), we have
$$
\begin{array}{l} \mathbb {E} \left[ \| G (\mathbf {W}) \| _ {F} ^ {2} \right] = \mathbb {E} \left[ \left\| \nabla_ {\mathbf {W}} \| \mathbf {W} \mathbf {y} - \mathbf {x} \| _ {2} ^ {2} \right\| _ {F} ^ {2} \right] + \frac {\sigma_ {e} ^ {2}}{n} \mathbb {E} \left[ \| [ \mathbf {x} + \mathbf {z}, \dots , \mathbf {x} + \mathbf {z} ] \| _ {F} ^ {2} \right] \\ = \mathbb {E} \left[ \left\| ((\mathbf {W} - \mathbf {I}) \mathbf {x} + \mathbf {W} \mathbf {z}) (\mathbf {x} + \mathbf {z}) ^ {T} \right\| _ {F} ^ {2} \right] + \mathbb {E} \left[ \left\| \mathbf {e} (\mathbf {x} + \mathbf {z}) ^ {T} \right\| _ {F} ^ {2} \right] \\ = R + \sigma_ {e} ^ {2} \left(1 + \sigma_ {z} ^ {2}\right). \\ \end{array}
$$
The term  $R$  can be bounded as
$$
\begin{array}{l} R = \mathbb {E} \left[ \left\| \left(\left(\mathbf {W} - \mathbf {W} ^ {*} + \mathbf {W} ^ {*} - \mathbf {I}\right) \mathbf {x} + \left(\mathbf {W} - \mathbf {W} ^ {*} + \mathbf {W} ^ {*}\right) \mathbf {z}\right) \left(\mathbf {x} + \mathbf {z}\right) ^ {T} \right\| _ {F} ^ {2} \right] \\ = \mathbb {E} \left[ \left\| (\mathbf {W} - \mathbf {W} ^ {*}) (\mathbf {x x} ^ {T} + \mathbf {x z} ^ {T} + \mathbf {z x} ^ {T} + \mathbf {z z} ^ {T}) + (\mathbf {W} ^ {*} - \mathbf {I}) (\mathbf {x x} ^ {T} + \mathbf {x z} ^ {T}) + \mathbf {W} ^ {*} (\mathbf {z x} ^ {T} + \mathbf {z z} ^ {T}) \right\| _ {F} ^ {2} \right] \\ \leq 2 \mathbb {E} \left[ \left\| (\mathbf {W} - \mathbf {W} ^ {*}) (\mathbf {x x} ^ {T} + \mathbf {x z} ^ {T} + \mathbf {z x} ^ {T} + \mathbf {z z} ^ {T}) \right\| _ {F} ^ {2} \right] \\ + 2 \mathbb {E} \left[ \left\| (\mathbf {W} ^ {*} - \mathbf {I}) (\mathbf {x x} ^ {T} + \mathbf {x z} ^ {T}) + \mathbf {W} ^ {*} (\mathbf {z x} ^ {T} + \mathbf {z z} ^ {T}) \right\| _ {F} ^ {2} \right] \\ = 2 R _ {1} + 2 R _ {2}. \\ \end{array}
$$
We next bound  $R_{1}$  and  $R_{2}$ .
Bound on  $R_{1}$ : To bound  $R_{1}$ , first note that for  $\mathbf{x} \sim \mathcal{N}(0, 1 / d\mathbf{I})$
$$
\mathbb {E} \left[ \left\| \mathbf {A x x} ^ {T} \right\| _ {F} ^ {2} \right] = \operatorname {t r a c e} \mathbb {E} \left[ \mathbf {A x x} ^ {T} \mathbf {x x} ^ {T} \mathbf {A} ^ {T} \right] \leq 2 \operatorname {t r a c e} \mathbb {E} \left[ \mathbf {A x x} ^ {T} \mathbf {A} ^ {T} \right] = 2 \mathbb {E} \left[ \| \mathbf {A x} \| _ {2} ^ {2} \right] = \frac {2}{d} \| \mathbf {A U} \| _ {F} ^ {2}.
$$
Similarly, for  $\mathbf{x} \sim \mathcal{N}(0, 1 / d\mathbf{I})$  and  $\mathbf{z} \sim \mathcal{N}(0, \sigma_z^2 / n\mathbf{I})$
$$
\mathbb {E} \left[ \left\| \mathbf {A x z} ^ {T} \right\| _ {F} ^ {2} \right] = \mathrm {t r a c e} \mathbb {E} \left[ \mathbf {A x z} ^ {T} \mathbf {z x} ^ {T} \mathbf {A} ^ {T} \right] = \sigma_ {z} ^ {2} \mathrm {t r a c e} \mathbb {E} \left[ \mathbf {A x x} ^ {T} \mathbf {A} ^ {T} \right] = \sigma_ {z} ^ {2} \mathbb {E} \left[ \| \mathbf {A x} \| _ {2} ^ {2} \right] = \frac {\sigma_ {z} ^ {2}}{d} \| \mathbf {A U} \| _ {F} ^ {2},
$$
$$
\mathbb {E} \left[ \left\| \mathbf {A z x} ^ {T} \right\| _ {F} ^ {2} \right] = \mathrm {t r a c e} \mathbb {E} \left[ \mathbf {A z x} ^ {T} \mathbf {x z} ^ {T} \mathbf {A} ^ {T} \right] = \mathrm {t r a c e} \mathbb {E} \left[ \mathbf {A z z} ^ {T} \mathbf {A} ^ {T} \right] = \mathbb {E} \left[ \| \mathbf {A z} \| _ {2} ^ {2} \right] = \frac {\sigma_ {z} ^ {2}}{n} \| \mathbf {A} \| _ {F} ^ {2},
$$
Also similarly,
$$
\mathbb {E} \left[ \left\| \mathbf {A} \mathbf {z} \mathbf {z} ^ {T} \right\| _ {F} ^ {2} \right] \leq \frac {2 \sigma_ {z} ^ {4}}{n} \| \mathbf {A} \| _ {F} ^ {2}.
$$
$$
R _ {1} \leq \| \mathbf {W} - \mathbf {W} ^ {*} \| _ {F} ^ {2} \left(\frac {2}{d} + \frac {2 \sigma_ {z} ^ {2}}{d} + \frac {2 \sigma_ {z} ^ {4}}{n}\right).
$$
Bound on  $R_{2}$ : Note that
$$
\begin{array}{l} R _ {2} \leq \| (\mathbf {W} ^ {*} - \mathbf {I}) \mathbf {U} \| _ {F} ^ {2} \frac {2 + \sigma_ {z} ^ {2}}{d} + \| \mathbf {W} ^ {*} \| _ {F} ^ {2} \frac {\sigma_ {z} ^ {2} + 2 \sigma_ {z} ^ {4}}{n} \\ \leq \left(\sigma_ {z} ^ {2} \frac {d}{n}\right) ^ {2} d \frac {2 + \sigma_ {z} ^ {2}}{d} + d \frac {\sigma_ {z} ^ {2} + 2 \sigma_ {z} ^ {4}}{n} \\ \leq 3 \left(\sigma_ {z} ^ {2} \frac {d}{n}\right) ^ {2} + d \frac {\sigma_ {z} ^ {2} + 2 \sigma_ {z} ^ {4}}{n} \\ \leq 6 \frac {d}{n} \sigma_ {z} ^ {2}. \\ \end{array}
$$
where we used that  $\| (\mathbf{W}^{*} - \mathbf{I})\mathbf{U}\|_{F}^{2}\leq \left(\sigma_{z}^{2}\frac{d}{n}\right)^{2}d$  and  $\| \mathbf{W}^{*}\|_{F}^{2}\leq d$ , and for the last inequality we used that  $\sigma_z^2\le 1$ , by assumption.
Application of the bounds on  $R_{1}$  and  $R_{2}$  above concludes the proof of Equation (8).
B.2 Proof of equation (11)
Note that the risk can be written as
$$
R (\mathbf {W}) = \mathbb {E} \left[ \| \mathbf {W y} - \mathbf {x} \| _ {2} ^ {2} \right] = \mathbb {E} \left[ \| \mathbf {Y w} - \mathbf {x} \| _ {2} ^ {2} \right], \tag {12}
$$
where  $\mathbf{w} = [\mathbf{w}_1^T, \ldots, \mathbf{w}_n^T]^T \in \mathbb{R}^{n^2}$  and  $\mathbf{Y} \in \mathbb{R}^{n \times n^2}$  is a block diagonal matrix containing the row-vector  $\mathbf{y}^T$  on the diagonal. With this notation,
$$
\begin{array}{l} R (\mathbf {W}) = \mathbb {E} \left[ \| \mathbf {Y} \mathbf {w} - \mathbf {x} \| _ {2} ^ {2} \right] \\ = \mathbb {E} \left[ \| \mathbf {Y w} - \mathbf {Y w} ^ {*} + \mathbf {Y w} ^ {*} - \mathbf {x} \| _ {2} ^ {2} \right] \\ = \mathbb {E} \left[ \| \mathbf {Y} (\mathbf {w} - \mathbf {w} ^ {*}) \| _ {2} ^ {2} \right] + 2 \mathbb {E} \left[ (\mathbf {Y} (\mathbf {w} - \mathbf {w} ^ {*}) ^ {T} (\mathbf {Y} \mathbf {w} ^ {*} - \mathbf {x}) \right] + \mathbb {E} \left[ \| \mathbf {Y} \mathbf {w} ^ {*} - \mathbf {x} ^ {*} \| _ {2} ^ {2} \right] \\ = \mathbb {E} \left[ \| \mathbf {Y} (\mathbf {w} - \mathbf {w} ^ {*}) \| _ {2} ^ {2} \right] + 2 (\mathbf {w} - \mathbf {w} ^ {*}) ^ {T} \mathbb {E} \left[ \mathbf {Y} ^ {T} (\mathbf {Y} \mathbf {w} ^ {*} - \mathbf {x}) \right] + R (\mathbf {W} ^ {*}) \\ \stackrel {\mathrm {i}} {=} \mathbb {E} \left[ \| \mathbf {Y} (\mathbf {w} - \mathbf {w} ^ {*}) \| _ {2} ^ {2} \right] + R (\mathbf {W} ^ {*}) \\ \leq \left(1 / d + \sigma_ {z} ^ {2} / n\right) \| \mathbf {W} - \mathbf {W} ^ {*} \| _ {F} ^ {2} + R (\mathbf {W} ^ {*}) \\ \end{array}
$$
where equality i follows from  $0 = \nabla R(\mathbf{w}^{*}) = \mathbb{E}\left[2\mathbf{Y}^{T}(\mathbf{Y}\mathbf{w}^{*} - \mathbf{x})\right]$  for a minimizer  $\mathbf{w}^*$ , and the last inequality follows from  $\mathbf{y} = \mathbf{U}\mathbf{c} + \mathbf{z}$  where  $\mathbf{c} \sim \mathcal{N}(0,1 / d\mathbf{I})$  and  $\mathbf{x} \sim \mathcal{N}(0,\sigma_z^2 /2\mathbf{I})$ .
B.3 Proof of Lemma 1
We start by upper bounding the expectation  $e_k = \mathbb{E}\left[\|\pmb{\theta}_k - \pmb{\theta}^*\|_2^2\right]$ . We write  $G_k(\pmb{\theta}_k) = G(\pmb{\theta}_k)$  for the stochastic gradient at iteration  $k$  to emphasize that at each iteration, the gradient is drawn independently from the gradient at other iterations. First note that
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {k + 1} - \boldsymbol {\theta} ^ {*} \right\| _ {2} ^ {2} = \left\| \boldsymbol {\theta} _ {k} - \alpha_ {k} G _ {k} \left(\boldsymbol {\theta} _ {k}\right) - \boldsymbol {\theta} ^ {*} \right\| _ {2} ^ {2} \\ = \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \| _ {2} ^ {2} - 2 \alpha_ {k} \left\langle G _ {k} (\boldsymbol {\theta} _ {k}), \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \right\rangle + \alpha_ {k} ^ {2} \| G _ {k} (\boldsymbol {\theta} _ {k}) \| _ {2} ^ {2}. \\ \end{array}
$$
The expectation of the middle term above is
$$
\mathbb {E} \left[ \left\langle G _ {k} \left(\boldsymbol {\theta} _ {k}\right), \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \right\rangle \right] = \mathbb {E} \left[ \left\langle \mathbb {E} \left[ G _ {k} \left(\boldsymbol {\theta} _ {k}\right) \mid \boldsymbol {\theta} _ {k} \right], \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \right\rangle \right] = \mathbb {E} \left[ \left\langle \nabla h \left(\boldsymbol {\theta} _ {k}\right), \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \right\rangle \right], \tag {13}
$$
where we used that the random variable  $G_{k}$  is independent of the random variable  $G_{i}, i < k$ , since each stochastic gradient is drawn independently, and is therefore independent of the iterate  $\pmb{\theta}_{k}$ . Thus, iterating the expectation allows us to replace the stochastic gradient by the gradient.
Using the assumption that the stochastic gradient  $G$  is  $(M, B)$ -bounded yields
$$
e _ {k + 1} \leq \left(1 + \alpha_ {k} ^ {2} M ^ {2}\right) e _ {k} - 2 \alpha_ {k} \mathbb {E} \left[ \left\langle \nabla \mathbf {f} \left(\boldsymbol {\theta} _ {k}\right), \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \right\rangle \right] + \alpha_ {k} ^ {2} B ^ {2}. \tag {14}
$$
With  $\nabla h(\pmb{\theta}^{*}) = 0$ , we have that
$$
\langle \nabla h (\boldsymbol {\theta} _ {k}), \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \rangle = \langle \nabla h (\boldsymbol {\theta} _ {k}) - \nabla h (\boldsymbol {\theta} ^ {*}), \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \rangle \geq m \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \| _ {2},
$$
where the inequality follows from  $h$  being  $m$ -strongly convex. Application of this inequality to (14) yields
$$
e _ {k + 1} \leq \left(1 + \alpha_ {k} ^ {2} M ^ {2} - 2 \alpha_ {k} m\right) e _ {k} + \alpha_ {k} ^ {2} B ^ {2}. \tag {15}
$$
Since we choose the stepsize as  $\alpha_{k} = \frac{2}{m}\frac{1}{2\beta + k}$  with  $\beta = \frac{M^2}{m^2}$  we have  $\alpha_{k}\leq \frac{m}{M^{2}}$  . Then Equation (15) yields
$$
e _ {k + 1} \leq (1 - \alpha_ {k} m) e _ {k} + \alpha_ {k} ^ {2} B ^ {2}.
$$
Unrolling the iterations gives
$$
\begin{array}{l} e _ {k} \leq (1 - \alpha_ {k - 1} m) e _ {k - 1} + \alpha_ {k - 1} ^ {2} B ^ {2} \\ \leq (1 - \alpha_ {k - 1} m) (1 - \alpha_ {k - 2} m) e _ {k - 2} + (1 - \alpha_ {k - 1} m) \alpha_ {k - 2} ^ {2} B ^ {2} + \alpha_ {k - 1} ^ {2} B ^ {2} \\ \leq \prod_ {i = 0} ^ {k - 1} (1 - \alpha_ {i} m) e _ {0} + B ^ {2} \sum_ {i = 0} ^ {k - 1} \alpha_ {i} ^ {2} \prod_ {j = i + 1} ^ {k - 1} (1 - \alpha_ {j} m) \\ = \prod_ {i = 0} ^ {k - 1} \left(1 - \frac {2}{2 \beta + i}\right) e _ {0} + B ^ {2} \sum_ {i = 0} ^ {k - 1} \left(\frac {1}{m} \frac {2}{2 \beta + i}\right) ^ {2} \prod_ {j = i + 1} ^ {k - 1} \left(1 - \frac {2}{2 \beta + j}\right). \\ \end{array}
$$
Figure 7: Effect of adapting the network size for denoising. For each training set size the dark blue curves show the best performing network over a range of network sizes (see Figure 8 for all results). Light blue curves show the performance for a fixed network size. Since the performance curves of supervised training  $(\sigma_{e} = 0$ , top curves) and of self-supervised training  $(\sigma_{e} \in \{25, 50\}$ , middle and bottom curves) are affected very similarly from fixing the network size, the relative comparison between supervised and self-supervised training remain valid.
Using that  $\prod_{j = i}^{k}\left(1 - \frac{2}{j}\right)^{2} = \frac{(i - 2)(i - 1)}{(k - 1)k}$ , we get
$$
\begin{array}{l} e _ {k} \leq \frac {(2 \beta - 2) (2 \beta - 1)}{(k - 2) (k - 1)} e _ {0} + \frac {B ^ {2}}{m ^ {2}} \sum_ {i = 0} ^ {k - 1} \left(\frac {2}{2 \beta + i}\right) ^ {2} \frac {(2 \beta + i - 1) (2 \beta + i)}{(k - 2) (k - 1)} \\ \leq \frac {4 \beta^ {2}}{(k - 2) ^ {2}} e _ {0} + \frac {B ^ {2}}{m ^ {2}} \frac {1}{k - 2} \\ \leq \frac {1}{k - 2} \left(2 \beta e _ {0} + \frac {B ^ {2}}{m ^ {2}}\right), \\ \end{array}
$$
C Additional material for Section 4: Self-supervised image denoising
C.1 Gaussian image denoising
The Gaussian image denoising experiments are conducted on the following datasets. From the ImageNet dataset [37] we reserve 80/230 images for validation/testing, and create training sets  $S_N$  of size  $N$  ranging from 100 to 300k images and  $S_i \subset S_j$  for  $i < j$ . For the training set we center crop images to size  $128 \times 128$ .
Further, all following experiments that rely on the U-net as the network architecture, use a U-net with two blocks in the encoder and decoder part respectively and skip connections between blocks. Each block consists of two convolutional layers with LeakyReLU activation and instance normalization [41] after every layer, where the number of channels is doubled (halved) after every block in the encoder (decoder). The downsampling in the encoder is implemented as average pooling and the upsampling in the decoder as transposed convolutions. As proposed by Zhang et al. [48] we train a residual denoiser that learns to predict  $\mathbf{y} - \mathbf{y}'$  instead of directly predicting  $\mathbf{y}'$ , which improves performance.
All experiments were conducted on NVIDIA A40, NVIDIA RTX A6000 and NVIDIA Quadro RTX 6000 GPUs. We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 700 GPU hours for the experiments on self-supervised Gaussian denoising presented in Figure 2.
C.1.1 Noise2noise self-supervised denoising with a U-net of varying size
The results in Figure 2 are for a U-net of fixed network size, which is the natural setup since we're interested in studying differences of training schemes, and therefore we have to keep the network architecture constant.
However, for different training set sizes, networks of different sizes perform best; if the training set size is larger, typically a larger network performs best. In the following, we therefore perform a comparison of self-supervised training and supervised training, where we adapt the network size along with the training set size,  $N$ .
We find that adapting the network size to the training set size improves the performance of models trained in a supervised and a self-supervised fashion similarly, thus our findings on self-supervised versus supervised training continue to hold even if we adopt the network size as we change the training set size.
To show this, we follow an approach from the scaling law literature, where the goal is to observe the isolated effect of one resource on the model performance by providing unlimited access to all other resources. In the context of image reconstruction Klug and Heckel [22], investigate the performance as a function of the training set size without limiting network size and invested compute, i.e., by only considering the best performing model per training set size over a wide range of network sizes and by training each model until the performance converges on the validation set. Similar work has been done in other fields including computer vision [47] and natural language processing [20].
Figure 8 shows our additional results for adapting the network size for noise2noise self-supervised training with target noise levels  $\sigma_{e} \in \{0, 25, 50\}$ . The panels on the right show the performance as a function of the network size for a fixed training set size. The left panels show the performance as a function of the training set size, where only the best model per training set size is considered.
Figure 7 compares the three curves with adapted network size from the left panels in Figure 8 to the performance curves for a fixed network size from Figure 2.
We chose the fixed network to have a moderate size (7.4M) compared to the smallest (0.1M) and largest networks (46.5M) considered for adapting the network size. As a consequence in Figure 7 the smallest/largest training set sizes gain most performance from adapting to a smaller/larger network size. However, the gain in performance is small, which is also indicated by the relatively flat curves in the right panels of Figure 8, which show the scaling of the reconstruction performance as a function of the network size. Further, the three training methods, supervised and self-supervised training with  $\sigma_{e} \in \{25, 50\}$ , are affected similarly from fixing the network size. Hence, our findings regarding a relative comparison of supervised and self-supervised methods are not affected.
Training details for varying the network size. The left column in Figure 8 is a summary of all models that we trained for Gaussian denoising with a U-net under different levels of noise on the training targets  $\sigma_{e} \in \{0,25,50\}$ . For different training set sizes  $N \in \{0.1,0.3,0.6,1,3,6,10,30,60,100,300\}$  thousand images we varied the number of channels in the first layer of the U-net in  $\{16,32,64,128,192,256,320\}$  corresponding to  $P = \{0.1,0.5,1.9,7.4,16.7,30.0,46.5\}$  million network parameters.
For small training set sizes the curves in Figure 8 use the best performance out of up to 5 training runs with different random network initialization, indicated by the additional points underneath the different curves. In Figure 7 only the best runs are reported for brevity. As the variation among runs decreases as the training set size increases, a single run is performed for large training set sizes, and further runs are only added in case of obvious outliers.
We tested two different batch sizes 1 and 10 for training set sizes up to 60k images and found that while models trained with small training set sizes significantly benefit from batch size 1, models trained with larger training set sizes are indifferent to the choice of the batch size. Hence, we set the batch size to 1 for training set sizes  $N \leq 6\mathrm{k}$  and to 10 otherwise.
All models are trained with the Adam optimizer [21] with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ . As we train on different combinations of network size, batch size and training set size we have to adjust the initial learning rate accordingly.
We found that the following automated scheme for learning rate annealing reliably finds an initial learning rate that lies within the range of suitable learning rates. Starting from a small learning rate  $1.25 \times 10^{-6}$  we double the learning rate after every epoch as long as the validation error improves. If training with a learning rate for 3 epochs does not yield an improvement, the annealing is terminated. We then continue the training with the model checkpoint corresponding to a learning rate 4-times smaller than the current learning rate.
(a) Supervised denoising  $(\sigma_{e} = 0)$
(b) Self-supervised denoising  $(\sigma_{e} = 25)$
(c) Self-supervised denoising  $(\sigma_{e} = 50)$
Figure 8: Complete results for adapting the network size for denoising. Each curve on the right is obtained by fixing the training set size and sweeping across different network sizes. Then, the curve on the left is created by taking the best performing network for each training set size. Colors in the plots on the left and right correspond to the same training set size.