SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Once the learning rate is set, we apply an automated learning rate decay. We halve the learning rate once the validation PSNR did not improve for eight consecutive epochs. If the learning rate was decayed twice in a row without an improvement, we terminate the training. Then, the model with the best validation PSNR is picked for evaluation on the test set. Picking the model based on the validation error was necessary since we faced overfitting during training for  $\sigma_{e} \in \{25, 50\}$ , as shown in the example in Figure 9. The overfitting is to be expected as we fix the noise per training example over all epochs on both noisy inputs  $\mathbf{y}$  and noisy targets  $\mathbf{y}'$ .
To perform early stopping we compute a self-supervised validation PSNR between measurements  $\mathbf{y}$  and  $\mathbf{y}'$  of the images  $\mathbf{x}$  in the validation set. It is well known that the self-supervised validation PSNR is proportional to the actual PSNR, see [9] for a discussion. As the self-supervised training error approximates the supervised one, so does the self-supervised validation error. Figure 9 shows an example for the correlation between self-supervised and supervised validation error.
C.1.2 Noise2noise self-supervised denoising with a U-net of fixed size
The results in Figure 2 for noise2noise self-supervised denoising were obtained with a fixed network size with 128 channels in the first layer resulting in 7.4M parameters. The training procedure for fixed size networks is similar to training with networks of varying size described in Appendix C.1.1.
Figure 9: Overfitting and supervised vs. self-supervised validation PSNR for denoising. In this example, supervised and self-supervised validation PSNR of a network trained in a noise2noise self-supervised manner (with  $\sigma_{e} = 25$ ) improves until epoch 20. At epoch 20, performance declines due to overfitting to the fixed noise realizations on the training inputs and targets. As supervised and self-supervised PSNR behave qualitatively the same we can use the self-supervised PSNR to pick the best model over all epochs. This experiment uses a training set of size 100k and a U-net with 320 channels in the first layer.
For  $N \leq 6000$  images, we reused the networks trained for the adaptive network scenario from Figure 8 since we already have 128-channeled networks for them. The training procedure for those networks is discussed in Appendix C.1.1.
For  $N \geq 10,000$  images, we used a batch size of 1 and as the network size is fix, we also fix the initial learning rate to  $6.4 \times 10^{-4}$  and then applied learning rate decay as described in Appendix C.1.1. For the largest training set size  $N = 300k$  we trained once with initial learning rate  $6.4 \times 10^{-4}$  and once with  $1.6 \times 10^{-4}$  and picked the run that performed better.
As in Appendix C.1.1 we picked the best out of up to three training runs from different random initializations to account for the variance between runs for small and medium sized training sets.
C.1.3 Noisier2noise self-supervised denoising with a U-net
The performance curve for noisier2noise self-supervised denoising in Figure 2 is obtained with a U-net of fixed size with 128 channels in the first layer and 7.4M network parameters in order to match the setup for noise2noise in Appendix C.1.2.
In noisier2noise, we have only one noisy measurement per clean image, which is used as the training target. In the simplest version presented by Moran et al. [30] the training input is created by injecting additional noise of the same distribution and variance to the noisy image. That means we add Gaussian noise with noise level  $\sigma = 25$  to the images in the training set and train the network to predict the original noisy training image from its noisier version. As we have control over the noise that we inject during training, it is re-sampled in every epoch to improve performance.
Similar to training with the noise2noise loss with fixed network size described in Appendix C.1.2 we use a batch size of 1, the Adam optimizer with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  and an initial learning rate of  $6.4\times 10^{-4}$ . To be consistent with Appendix C.1.2 we also tried an initial learning of  $1.6\times 10^{-4}$  for  $N = 300\mathrm{k}$ , which performed slightly better.
We apply the same scheme for learning rate decay as described in Appendix C.1.2, where the self-supervised validation loss is computed between the noisy training target and the network output. We did not observe overfitting in the noisier2noise setup since the additional noise on the noisier training inputs is re-sampled in every epoch.
As in Appendix C.1.2 we picked the best out of up to three training runs from different random initializations to account for the variance between runs for small and medium sized training sets.
Figure 10: Gaussian image denoising with a SwinIR vs. U-net. While the SwinIR outperforms the U-net in terms of absolute performance, the performance of both models trained in a noise2noise manner approaches the performance when trained in a supervised manner as the number of training images increases.
C.1.4 Neighbor2neighbor self-supervised denoising with a U-net
To obtain the results for neighbor2neighbor in Figure 2 we trained the same U-net with 128 channels in its first layer, corresponding to 7.4M parameters as in the experiments for noisier2noise and noise2noise. As before, we set the batch size to 1 and use the Adam optimizer with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ .
In neighbor2neighbor training the given noisy example is subsampled into two images that are used as training input and target. We follow the approach described in Huang et al. [18] and we refer to their description for more details on the subsampling, the loss function and the regularization used during training.
As proposed by Huang et al. [18] we fix the number of training epochs to 100 for all training set sizes and halved the learning rate once in every 20 epochs. However, with the total number of epochs fixed the convergence of the network as the training set size changes might depend on the choice of the initial learning rate. Hence, instead of fixing the initial learning rate as for noise2noise and noisier2noise in Appendix C.1.3 and C.1.2, we ablate for a good choice of initial learning rate in the range of  $10^{-5}$  to  $10^{-2}$  for every training set size. Contrary to noise2noise and noisier2noise, for the neighbor2neighbor loss computing a self-supervised validation loss based only on a noisy validation set did not correlate well with the supervised validation loss. Hence, we assumed access to a version of our validation set that contains ground truth images to evaluate the results of the learning rate ablation study and to pick the best performing model over the training epochs. The curve in Figure 2 contains only the performance of the run with the best learning rate per training set size.
Same as in our noise2noise and noisier2noise experiments we picked the best out of up to three training runs from different random initializations to account for the variance between runs for small and medium sized training sets.
C.1.5 Noise2noise self-supervised denoising with a SwinIR
In Section 4 Figure 2 we empirically determined the sample complexity of noise2noise self-supervised and supervised Gaussian denoising with a U-net and showed that as the training set size gets large a model trained in a self-supervised manner approaches the performance of a model trained in a supervised manner. However, we expect that our results for comparing supervised and self-supervised training schemes translate qualitatively to other network architectures.
In this section we confirm this by re-running our experiments with the SwinIR [26] architecture, a recent state-of-the-art model for image denoising based on the Swin Transformer [27].
Figure 10 shows the performance of training a SwinIR on the training sets of size  $N \in \{0.1, 1, 10, 100\}$  thousand image patches with a supervised and a noise2noise self-supervised loss with noise level on the training targets  $\sigma_{e} = 25$ . As expected, the SwinIR outperforms the U-net (curves are taken from Figure 2) on all training set sizes. However the relative performance gap between the models trained in a supervised and self-supervised way is similar and at 100k training examples the gap is reduced to 0.098/0.097dB for the SwinIR/U-net.
We train the default SwinIR for color image denoising from Liang et al. [26] with 11.5M network parameters. The networks are trained with the MSE loss, the Adam optimizer and learning rates  $\{2e - 4, 1e - 4, 8e - 5, 8e - 5\}$  for training set sizes  $N \in \{0.1, 1, 10, 100\}$  thousand training examples respectively. Depending on the training set size, the networks trained with supervised/self-supervised loss are trained for  $\{100, 87, 78, 40\} / \{100, 87, 50, 26\}$  epochs, where we need less epochs for self-supervised training due to overfitting similar to what was observed for noise2noise training with the U-net in Figure 9. The initial learning rate is decayed by a factor of 2 at  $60\%$ ,  $75\%$  and  $90\%$  of the total number of training epochs.
C.2 Real-world camera noise denoising
In this section we provide additional details for the experiments on real-world camera noise denoising with the SID [1] dataset presented in Figure 3.
We train the same U-net as described in Appendix C.1, with 56 channels in the first layer resulting in 1.4M network parameters.
Each training set consists of non-overlapping patches of size  $128 \times 128$  drawn randomly from all available patches. From the 160 scene instances in the medium SIDD training set we use all scenes except scenes 0199 and 0200 as the remaining scenes already constitute to  $125\mathrm{k}$  patches and the largest training set we consider contains only  $100\mathrm{k}$  patches.
For self-supervised training input and target are cropped from the two noisy images given per scene in the medium SID dataset. For supervised training the noisy target is replaced with the ground truth image estimated from all noisy images of this scene as described in Abdelhamed et al. [1]. For the different runs with a fixed training set size displayed in Figure 3, training patches and network initializations are drawn independently between runs.
Training and testing is performed in the raw-RGB space, i.e., on single-channel images where every square of  $2 \times 2$  pixels contains one R,G,G and B value. The order of the 4 values varies over the dataset depending on the Bayer pattern of the various cameras used. Given the Bayer pattern of an image we rearrange the squares of size  $2 \times 2$  pixels to follow the order RGGB. Finally the raw images are packed into a 4-channel representation corresponding to their RGGB values before processed by the network.
All networks are trained with the Adam optimizer with an initial learning rate of 0.00032, which is halved at a fixed set of epochs depending on the training set size.
D Additional material for Section 5: Self-supervised compressive sensing
D.1 Compressive sensing for natural images
For the experiments in Section 5.1 we train the same U-net as described in Appendix C.1, but with 3 blocks in the encoder/decoder and with 24 channels in the first layer resulting in about 1M network parameters, and unlike in Appendix C.1 the network directly predicts the target measurement  $\mathbf{y}'$  instead of the residual  $\mathbf{y} - \mathbf{y}'$  during training. The input to the network consists of a complex-valued coarse reconstruction  $\mathbf{F}^{-1}\mathbf{y}$  in the image domain, which we normalize to zero-mean and standard deviation one before handed to the U-net. Further, the real and complex part are forwarded to the U-net as two separate input channels. The output of the network is denormalized before transformed to the frequency domain in which the (masked in case of self-supervised training)  $\ell_2$  training loss is computed. The test performance is evaluated between the absolute value of the network output and the ground truth image.
The networks are trained using the Adam optimizer [21] with an initial learning rate of  $10^{-3}$ . The networks are evaluated on the validation set in every second epoch and the learning rate is decayed by a factor of 10 if there is no improvement for certain number of consecutive validations. This patience parameter is gradually decreased from 20 for the smallest to 6 for the largest training set size. The training is terminated after training for a few epochs with the minimal learning rate  $10^{-5}$ . Only the best model according to the validation loss is evaluated on the test set, which we then report in Figure 5. The validation/test set consists of 80/300 patches. The batch size is set to 1.
All experiments were conducted on a NVIDIA A40 GPU. We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 500 GPU hours for the experiments in Figure 5.
D.2 Compressive sensing accelerated MRI
The data from the multi-coil fastMRI brain dataset consists of images of different contrasts. In our experiments we use the subset of about 55k AXT2-weighted images corresponding to a single contrast. We use 50k of those images to design training sets  $S_{N}$  of varying size  $N$  with  $S_{i} \subset S_{j}$  for  $i < j$ , 300 for validation and the remaining 4700 for testing. The images for validation and testing are taken from different subjects than for training.
The test performance for a given measurement is evaluated between the magnitude of the complex valued network reconstruction and the ground truth image. Note that the estimated sensitivity maps are strictly zero outside the region in which the object that we aim to reconstruct is located. Since we compute the training loss in the k-space after applying the sensitivity maps to the network output, the network output does not receive any supervision in regions, where the sensitivity maps are zero. Hence, in order to compute meaningful test scores on the magnitude of the network output we apply binary masking to remove artifacts in the background of the reconstructed images. The estimated sensitivity maps are normalized in a way such that a binary mask  $\mathbf{B} \in \mathbb{R}^{n \times n}$  can be obtained as  $\mathbf{B} = \sum_{j=1}^{C} \mathbf{S}_j^* \mathbf{S}_j$ .
D.2.1 Compressive sensing accelerated MRI with a U-net
For the results in Figure 6 we train the same U-net as described in Appendix D.1, but with 4 blocks in the encoder/decoder and with 64 channels in the first layer resulting in about 31M network parameters. We use the same input/output normalization as in Appendix D.1.
The networks are trained with the RMSprop optimizer as it is the default in the fastMRI repository [46] and with an initial learning rate of  $10^{-3}$ . The networks are evaluated on the validation set after every epoch and the learning rate is decayed by a factor of 10 if no improvement is observed for 10 consecutive epochs. The training is terminated after training 10 epochs with the minimal learning rate of  $10^{-6}$ . The best model according to the validation loss is evaluated on the test set.
All experiments were conducted on a NVIDIA A40 GPU. We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 600 GPU hours for the experiments presented in Figure 6.
D.2.2 Compressive sensing accelerated MRI with a VarNet
In this section we provide additional results on supervised versus self-supervised compressive sensing MRI, where we replace the U-net used in Figure 6 with the state-of-the-art end-to-end VarNet [38]. As our analysis pertains to comparing different training schemes, we expect our results to translate qualitatively to other types of network architectures.
As Figure 11 shows the VarNet outperforms the U-net significantly. However, similar to the U-net the relative performance gap between models trained in a noise2noise self-supervised way and trained in a supervised way is small already for small training set sizes.
For training the VarNets we follow the implementation from the official fastMRI repository [46] $^{1}$ , where the VarNet consists of a cascade of U-nets with intermediate data consistency steps and an additional U-net that estimates the sensitivity maps used during the data consistency steps and to compute the input to the first U-net.
We use a VarNet with 6 cascades, where each U-net consists of 4 blocks in the encoder and decoder and 14 channels in the top layer resulting in a total of about 9.4M network parameters. The VarNet outputs an estimate  $\hat{\mathbf{y}}_{j,\mathrm{full}}$  of the fully sampled k-space for  $j = 1,\dots ,C$  coils. We use the precomputed sensitivity maps  $\mathbf{S}_j$  estimated with ESPIRiT [39] instead of the sensitivity maps estimated
<sup>1</sup>https://github.com/facebookresearch/fastMRI
Figure 11: Compressive sensing MRI with a VarNet vs. U-net. While the VarNet outperforms the U-net in absolute numbers, for both types of network architecture the performance gap between supervised and self-supervised trained models vanishes already at small training set sizes.
by the network to compute the final reconstructed image as
$$
\hat {\mathbf {x}} = \left| \sum_ {j = 1} ^ {C} \mathbf {S} _ {j} ^ {*} \mathbf {F} ^ {- 1} \hat {\mathbf {y}} _ {j, \mathrm {f u l l}} \right|.
$$
This is analogous to how we compute the reference ground truth images; See Section 5.2.
Depending on the training set size  $N \in \{50, 100, 500, 1000, 5000, 10000, 50000\}$  the model is trained for  $N_{\mathrm{ep}} \in \{150, 130, 125, 120, 90, 75, 55\}$  epochs. All networks are trained with the RM-Sprop optimizer with an initial learning rate of  $10^{-3}$ , which is decayed by a factor of 10 once for the last 15 and again for the last 5 training epochs.
E Details for computing estimates of the variance of the stochastic gradient
In this section we provide additional details on the histograms of the normalized variance of the stochastic gradients, i.e., normalized estimates of the MSE  $\| \nabla \ell_{\mathrm{SS}}(f_{\pmb{\theta}}(\mathbf{y}_i),\mathbf{y}_i^{\prime}) - \nabla R(\pmb {\theta})\| _2^2$  after one epoch of training presented in Figures 2, 3, 5 and 6.
We consider the gradients with respect to the network weights  $\theta$  after training one epoch with the supervised loss. Note that the gradient distribution changes at each epoch.
We compute stochastic gradients  $\nabla \ell_{\mathrm{SS}}(f_{\pmb{\theta}}(\mathbf{y}_i),\mathbf{y}_i^{\prime})$  for  $i = 1,\dots ,N$  with  $N = 10000$  images. The gradient of the risk  $\nabla R(\pmb {\theta})$  is estimated with the empirical gradient of the supervised loss, i.e.,
$$
\nabla \hat {R} (\boldsymbol {\theta}) = \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla \ell \left(f _ {\boldsymbol {\theta}} \left(\mathbf {y} _ {i}\right), \mathbf {x} _ {i}\right).
$$
Finally, the normalized variance of one the  $i$ -th stochastic gradient is computed as
$$
\frac {\left\| \nabla \ell_ {\mathrm {S S}} (f _ {\pmb {\theta}} (\mathbf {y} _ {i}) , \mathbf {y} _ {i} ^ {\prime}) - \nabla \hat {R} (\pmb {\theta}) \right\| _ {2} ^ {2}}{\left\| \nabla \hat {R} (\pmb {\theta}) \right\| _ {2} ^ {2}}.
$$