SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At 100 training images the performance gap between supervised and self-supervised training with  $\sigma_{e} = 25$  ( $\sigma_{e} = 50$ ) is  $0.645\mathrm{dB}$  (1.497dB), and at 100k training images the gap is reduced to  $0.097\mathrm{dB}$  (0.277 dB). For this specific setup, we can read of the number of additional images required to meet a certain denoising performance relative to what can be achieved with supervised training. For example, to achieve the performance of supervised training on 3k images, self-supervised training with noise on the training targets  $\sigma_{e} = 25$  ( $\sigma_{e} = 50$ ) requires 10k (30k) images (gray dashed line in Figure 2).
The theory in Section 3 is based on a convergence analysis of the stochastic gradient method, and the rate at which performance improves as a function of the number of training examples,  $N$ , is determined by how well the stochastic gradients of the risk,  $\nabla \ell_{\mathrm{SS}}(f_{\theta}(\mathbf{y}_i),\mathbf{y}_i^{\prime})$  (where  $(\mathbf{y}_i,\mathbf{y}_i^{\prime})$  is a training example), approximate the gradient of the risk, i.e.,  $\nabla R(\pmb {\theta})$ . In Figure 2 (right panel) we show the histograms of the normalized variance of stochastic gradients, i.e., normalized estimates of the MSE  $\| \nabla \ell_{\mathrm{SS}}(f_{\theta}(\mathbf{y}_i),\mathbf{y}_i^{\prime}) - \nabla R(\pmb {\theta})\| _2^2$  after one epoch of training; see Appendix E for details. It can be seen that the larger the variance of the stochastic gradients, the slower the improvement as a function of the training examples,  $N$ .
Self-supervised training with noisier2noise and neighbor2neighbor in general does not yield networks as good as trained in a supervised fashion, even if abundant training data is available. Noisier2noise [30] and neighbor2neighbor [18] only require a single noisy measurement per clean image for training. However, noisier2noise adds additional noise to the training inputs, which introduces a mismatch between the problem solved during training and inference. Neighbor2neighbor relies on assumptions on the similarity between neighboring pixels in the clean image that only hold approximately in practice. In Figure 2 we see that noisier2noise and neighbor2neighbor training,
Figure 3: Real-world camera noise denoising on SIDD. Left: Similar to our results for Gaussian denoising a model trained with noisy targets in a noise2noise manner approaches the performance of a model trained with ground truth targets in a supervised manner as the number of training patches gets large. Right: Histogram of the variance of the stochastic gradients. The stochastic gradients of the self-supervised loss are slightly noisier than the ones of the supervised loss.
unlike noise2noise, yield methods that perform worse than methods trained in a supervised fashion even if abundant training data is available. For training details see Appendices C.1.3 and C.1.4.
For the results in Figure 2 we consider a U-net of fixed size. The performance of a network trained in a supervised and self-supervised manner depends on the network architecture and size. However, since we investigate training schemes, we expect our qualitative findings for relative performance differences to continue to hold for other architectures and network sizes. In Appendix C.1.1 we provide results for U-nets of varying sizes and in Appendix C.1.5 we provide results for SwinIR [26], a recent state-of-the-art network for image denoising. The results are as expected analogous to the ones for the U-net presented here.
4.2 Real-world camera noise
We now study self-supervised denoising for real-world camera noise from the Smartphone Image Denoising Dataset (SIDD) [1]. The noise in the SID dataset is structured and not Gaussian.
Setup. We train U-nets on increasing amounts of patches of size  $128 \times 128$  from the training set, validate on the first 10 scenes, and report the test performance on the remaining 30 scenes from the validation set. The dataset consists of 150 noisy images per scene of which 2 are used as input and target for self-supervised training. Ground truth images for supervised training are estimated from all 150 noisy images. Training and testing is done in the raw-RGB space; see Appendix C.2 for details.
Results and discussion. In Figure 3 we see that performance improves steeper as a function of the training set size compared to Gaussian denoising in Figure 2, which we attribute to the high variety of noise levels and lighting conditions present in the SID dataset. However, analogous to the Gaussian case the model trained in a self-supervised manner closely approaches the performance of the model trained in a supervised manner as the number of training examples grows large.
5 Empirical results for self-supervised compressive sensing
We now study the performance as a function of the training set size of a network trained for compressive sensing reconstruction by minimizing the self-supervised objective (2) with the noise2noise-like loss (4), relative to the performance achieved by the same network with supervised training.
We perform two sets of experiments: First we experiment on natural images obtained from ImageNet, since this allows us to explore the compressive sensing problem over a wider range of training set sizes than existing medical image datasets allow. Then, we investigate accelerated MRI reconstruction on real world multi-coil measurements from the fastMRI dataset [46]. We consider a U-net in the main body, but again our qualitative results are independent of the network, and we demonstrate this with experiments on the VarNet, a state-of-the-art architecture, in the appendix.
Figure 4: Self-supervised training scheme for compressive sensing MRI. The overlap between constructed measurements  $\mathbf{y},\mathbf{y}^{\prime}$  is indicated in green and stems from both masks sampling center frequencies with probability 1 and an expected overlap of non-center frequencies of size  $p^{\prime}q$ . The scheme depicts single-coil measurements for conciseness.
Sampling scheme. The setup from Section 2.2 requires a pair of source measurement  $\mathbf{y}$  (that can be arbitrarily obtained) and a target measurement that is sampled independently of the source measurement as  $\mathbf{y}' = \mathbf{M}'\mathbf{F}\mathbf{x}$ , where  $\mathbf{M}'$  is a random mask with a non-zero probability of selecting each frequency. While in clinical MRI protocols, we can acquire two independently undersampled measurements per image, we typically acquire only one. We therefore slightly adapt our sampling scheme as follows to generate two measurements  $\mathbf{y}$  and  $\mathbf{y}'$  from one undersampled measurement  $\bar{\mathbf{y}}$  that is undersampled with factor  $\mu$ . See Figure 4 for an illustration of the scheme.
Our goal is to train a network to perform reconstruction based on a measurement  $\mathbf{y}$  undersampled with factor  $p < \mu$ . We consider Cartesian undersampling, where the frequency domain (also called k-space) is sampled column-wise. Since the low-frequency components of an image contain the largest portion of the signal energy, we follow the common practice to sample a fraction  $\nu = 0.08$  of the center columns in the k-space with probability 1, and assign the center frequencies to both measurements  $\mathbf{y}$  and  $\mathbf{y}'$ . We then add a random fraction  $\frac{p - \nu}{\mu - \nu}$  of the non-center columns in  $\tilde{\mathbf{y}}$  to  $\mathbf{y}$ , which is now a measurement undersampled with factor  $p$  by design. The target  $\mathbf{y}'$  is constructed by adding the remaining frequencies that are in  $\tilde{\mathbf{y}}$  but not in  $\mathbf{y}$ , and an additional fraction  $p'q$  of the non-center columns in  $\mathbf{y}$ , where  $p' = \frac{p - \nu}{1 - \nu}$ . The rationale behind this approach is that if the masks  $\mathbf{M}$  and  $\mathbf{M}'$  sample fractions  $p'$  and  $q$  independently from all non-center frequencies, then the expected overlap of frequencies selected by both masks is  $p'q$  with  $q = \frac{\mu - p}{1 - p}$ .
We compute the diagonal weight mask from equation (4) as  $\mathbf{W} = \mathbb{E}\left[\mathbf{M}'\right]^{-1/2}$ , which yields 1's at entries corresponding to center frequencies and  $1/\sqrt{q}$  otherwise. In our experiments we set  $\nu = 0.08$ ,  $p = 0.25$  and  $\mu \in \{0.28, 0.33\}$  resulting in  $q \in \{0.05, 0.11\}$ .
5.1 Compressive sensing for natural images
We start with compressive sensing experiments on natural images, because the largest existing MRI dataset for image reconstruction research (fastMRI) only has 55k images, and we want to vary the training set size far beyond that. In the next section we conduct experiments on fastMRI.
Setup. We use cropped images of size  $100 \times 100$  from ImageNet [37] as ground truth images  $\mathbf{x}$  and obtain undersampled complex-valued measurements in the frequency domain as  $\mathbf{y} = \mathbf{M}\mathbf{F}\mathbf{x}$  and  $\mathbf{y}' = \mathbf{M}'\mathbf{F}\mathbf{x}$ . We draw random training sets with 50 to 1M patches from a fixed pool of 1M images.
For all training set sizes, we train a U-net with about 1M network parameters and pick the best run out of different runs with independently sampled training set and network initialization. The network takes as input a coarse reconstruction obtained as  $\mathbf{F}^{-1}\mathbf{y}$ , where  $\mathbf{y}$  is the zero-filled measurement (i.e.,  $f_{\theta}(\mathbf{y}) = \mathrm{U - net}_{\theta}(\mathbf{F}^{-1}\mathbf{y})$ ). The measurements are complex-valued, and we take one channel of the U-net as the imaginary part and one as the real part. The reconstructed image is obtained from the complex-valued network output by taking the absolute value. See Appendix D.1 for training details.
Figure 5: Compressive sensing for natural images. Left: The larger the fraction of frequencies in the target,  $\mu$ , the faster the network's performance improves as a function of the number of training examples,  $N$ . As the training set size becomes large, self-supervised performance approaches supervised performance. Right: Histogram of the variance of the stochastic gradients. The noisier the gradients (i.e., the smaller  $\mu$ ), the slower the convergence as a function of the training examples.
Results and discussion. Figure 5 shows the performance in SSIM as a function of the training set size for supervised and self-supervised training with fractions  $\mu = 0.33$  and  $\mu = 0.28$  of all frequencies available for self-supervised training. Note that the fraction of frequencies in the target plays the same role as the noise variance for noise2noise denoising: The larger the fraction, the better in general. Since we keep the undersampling factor  $p = 0.25$  of the training inputs  $\mathbf{y}$  fixed, a smaller  $\mu$  implies a smaller undersampling factor  $q$  (less frequencies) of the training targets  $\mathbf{y}'$ .
Networks trained in a self-supervised manner with sufficiently many training examples perform as well as the same network trained in a supervised manner (but with fewer examples). The fewer measurements are available, i.e., the smaller  $\mu$ , the more training examples are required to achieve the same performance as the same network trained in a supervised manner. This parallels our theoretical and empirical findings for denoising where noise2noise training approaches the performance of supervised training as a function of the training set size and at a rate dependent of the training targets.
Figure 5, right panel, depicts histograms of the variance of the stochastic gradients. The noisier the gradients, the slower the performance increase as a function of the training examples.
The performance of the zero-filled baseline in Figure 5 is computed between the network inputs  $\mathbf{F}^{-1}\mathbf{y}$  and ground truth images  $\mathbf{x}$  in the test set. As we can see self-supervised training requires at least 250 training images to outperform this trivial baseline.
5.2 Compressive sensing MRI
Next, we discuss the performance of noise2noise self-supervised and supervised training for multi-coil MRI reconstruction. MRI scanners use parallel imaging, where a single scan simultaneously collects measurements  $\mathbf{y}_1,\ldots ,\mathbf{y}_C$  of the same object with  $C$  coils, where  $\mathbf{y}_j = \mathbf{MFS}_j\mathbf{x}$  with complex valued images  $\mathbf{x}\in \mathbb{C}^n$  and diagonal coil specific sensitivity maps  $\mathbf{S}_j\in \mathbb{C}^{n\times n}$ , which we estimate from the center of the k-space using ESPIRiT [39].
Setup. We conduct our experiments on a subset of the fastMRI brain dataset [46]. We design fixed training sets with 50 to 50k images. We train U-nets with 31M network parameters. To demonstrate that our results for comparing supervised and self-supervised training schemes translate qualitatively to other network architectures, we further train end-to-end variational networks (E2E-VarNet) [38], a state-of-the-art architecture, of size 9M parameters. We pick the best run out of different runs with independently sampled network initialization. See Appendix D.2 for training details.
The network inputs are computed through SENSE-1 coil combination [34] of the coil measurements as  $\sum_{j=1}^{C} \mathbf{S}_{j}^{*}\mathbf{F}^{-1}\mathbf{y}_{j}$ , where  $\mathbf{S}_{j}^{*}$  denotes the complex conjugate of the sensitivity map. Ground truth images for evaluation are computed analogously as  $|\sum_{j=1}^{C} \mathbf{S}_{j}^{*}\mathbf{F}^{-1}\mathbf{y}_{j,\mathrm{full}}|$ , where  $\mathbf{y}_{j,\mathrm{full}}$  is the fully sampled coil k-space provided by the fastMRI dataset. To compute the training loss in the k-space between given and predicted coil measurements, the set of predicted coil measurements is obtained from the network output as  $\mathbf{FS}_{1}f_{\theta}(\mathbf{y}), \ldots, \mathbf{FS}_{C}f_{\theta}(\mathbf{y})$ .
Figure 6: Compressive sensing MRI. Left: Already at small training set sizes the performance gap between supervised and self-supervised training vanishes. Right: The small performance gap on the left is explained by a small difference in the variances of the supervised and self-supervised gradients.
We follow Yaman et al. [45] and make best use of the given data by re-sampling the input mask  $\mathbf{M}$  in every epoch for supervised training. For self-supervised training the given frequencies  $\tilde{\mathbf{y}}$  are fixed, but we can re-sample the split into measurements  $\mathbf{y},\mathbf{y}^{\prime}$ . As the network sees more combinations of input/target pairs, the performance of supervised and self-supervised training improves.
Results and discussion. Figure 6 shows the performance of U-nets trained in a supervised and self-supervised way for multi-coil MRI as a function of the training set size. We report the performance in SSIM, since a high score in SSIM was shown to correlate well with a high rating by radiologists in the fastMRI challenge [31]. We find that already for small training set sizes the performance gap between supervised and self-supervised training vanishes to approximately 0.002 in SSIM.
Figure 6, right panel, depicts the histogram of the variance of the stochastic gradients. The difference in variance of the supervised and self-supervised gradients is very small, which explains the small performance gap between supervised and self-supervised training for this setup. The performance of the zero-filled baseline in Figure 6 is computed between the magnitude of the network inputs  $\left| \sum_{j=1}^{C} \mathbf{S}_j^* \mathbf{F}^{-1} \mathbf{y}_j \right|$  and ground truth images in the test set.
In Appendix D.2.2 we provide additional results for the E2E-VarNet. As expected, the VarNet outperforms the U-net by a significant margin for every training set size, especially for small training set sizes. However, the relative performance gap between VarNets trained in a supervised and self-supervised way is very small, analogous as for the U-net.
We characterized the sample complexity of deep learning based image reconstruction models trained with a class of self-supervised losses including noise2noise based on computing unbiased estimates of the gradients of the supervised loss. Our work shows that if sufficiently many training examples are available, it is as good to work with pairs of independent measurements as with training data with ground-truth images. In particular for accelerated MRI, training networks only based on undersampled data is sufficient for peak performance.
One advantage of supervised training over self-supervised training that is not reflected in our results is that supervised training has more flexibility in choosing the loss function, and choosing a loss function such as SSIM might give better visual image quality than training with the MSE. Our results of self-supervised training approaching supervised training pertain to training with the MSE.
We hasten to note that the class of methods that enable the computation of unbiased estimates of gradients of the supervised loss studied in this work, are in practice limited to problem setups in which obtaining two independent measurements of the same underlying signal is feasible.
Reproducibility The repository at https://github.com/MLI-lab/sample_complexity_ss_recon contains the code to reproduce all results in the main body of this paper.
The authors are supported by the Institute of Advanced Studies at the Technical University of Munich, the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 456465471, 464123524, the DAAD, the German Federal Ministry of Education and Research, and the Bavarian State Ministry for Science and the Arts. The authors also acknowledge the financial support by the Federal Ministry of Education and Research of Germany in the programme of "Souveraen. Digital. Vernetzt". Joint project 6G-life, project identification number: 16KISK002.
[40] D. Ulyanov, A. Vedaldi, and V. Lempitsky. "Deep Image Prior". In: IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[41] D. Ulyanov, A. Vedaldi, and V. Lempitsky. "Instance Normalization: The Missing Ingredient for Fast Stylization". In: arXiv:1607.08022 [cs] (2017).
[42] A. Q. Wang, A. V. Dalca, and M. R. Sabuncu. “Neural Network-Based Reconstruction in Compressed Sensing MRI Without Fully-Sampled Training Data”. In: Machine Learning for Medical Image Reconstruction. 2020.
[43] S. J. Wright and B. Recht. Optimization for Data Analysis. Cambridge, 2022.
[44] B. Yaman, S. A. H. Hosseini, and M. Akcakaya. “Zero-Shot Self-Supervised Learning for MRI Reconstruction”. In: International Conference on Learning Representations. 2022.
[45] B. Yaman, S. A. H. Hosseini, S. Moeller, J. Ellermann, K. Uğurbil, and M. Akçakaya. "Self-Supervised Learning of Physics-Guided Reconstruction Neural Networks without Fully Sampled Reference Data". In: Magnetic Resonance in Medicine (2020).
[46] J. Zbontar et al. "fastMRI: An Open Dataset and Benchmarks for Accelerated MRI". In: arXiv:1811.08839 [physics, stat] (2018).
[47] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. "Scaling Vision Transformers". In: IEEE Conference on Computer Vision and Pattern Recognition. 2022.
[48] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising". In: IEEE Transactions on Image Processing (2017).
[49] K. Zhang, Y. Li, J. Liang, J. Cao, Y. Zhang, H. Tang, R. Timofte, and L. Van Gool. "Practical Blind Denoising via Swin-Conv-Unet and Data Synthesis". In: arXiv:2203.13278 [cs, eess] (2022).
[50] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. "Plug-and-Play Image Restoration with Deep Denoiser Prior". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[51] Q. Zhang, H. Dong, and J. Zhao. "Channel Estimation for High-Speed Railway Wireless Communications: A Generative Adversarial Network Approach". In: Electronics (2023).
[52] B. Zhou, N. Dey, J. Schlemper, S. S. M. Salehi, C. Liu, J. S. Duncan, and M. Sofka. "DS-Former: A Dual-Domain Self-Supervised Transformer for Accelerated Multi-Contrast MRI Reconstruction". In: IEEE Winter Conference on Applications of Computer Vision. 2023.
A Proofs for propositions in Section 2
Proof of Proposition 1. We have that
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x}, \mathbf {y}, \mathbf {y} ^ {\prime}} \left[ \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {y} ^ {\prime} \| _ {2} ^ {2} \right] = \mathbb {E} _ {\mathbf {x}, \mathbf {y}, \mathbf {e}} \left[ \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} - \mathbf {e} \| _ {2} ^ {2} \right] \\ = \mathbb {E} _ {\mathbf {x}, \mathbf {y}} \left[ \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} \right] - \mathbb {E} _ {\mathbf {x}, \mathbf {y}, \mathbf {e}} \left[ (f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x}) ^ {T} \mathbf {e} \right] + \mathbb {E} \left[ \| \mathbf {e} \| _ {2} ^ {2} \right] \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {(\mathbf {x}, \mathbf {y})} \left[ \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2} \right] + \mathbb {E} \left[ \| \mathbf {e} \| _ {2} ^ {2} \right], \\ \end{array}
$$
where equation (i) follows from condition in Proposition 1. Noting that the expectation over the norm of the random variable  $e$  above is constant as a function of  $\theta$  proves the claim.
Proof of proposition 2. We show that
$$
\mathbb {E} _ {\mathbf {M} ^ {\prime}} \left[ \| \mathbf {W} (\mathbf {M} ^ {\prime} \mathbf {F} f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {y} ^ {\prime}) \| _ {2} ^ {2} \right] = \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2}.
$$
To see this, note that
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {M} ^ {\prime}} \left[ \| \mathbf {W} (\mathbf {M} ^ {\prime} \mathbf {F} f _ {\theta} (\mathbf {y}) - \mathbf {y} ^ {\prime}) \| _ {2} ^ {2} \right] = \mathbb {E} _ {\mathbf {M} ^ {\prime}} \left[ \| \mathbf {W} (\mathbf {M} ^ {\prime} \mathbf {F} f _ {\theta} (\mathbf {y}) - \mathbf {M} ^ {\prime} \mathbf {F} \mathbf {x}) \| _ {2} ^ {2} \right] \\ = \mathbb {E} _ {\mathbf {M} ^ {\prime}} \left[ \| \mathbf {W M} ^ {\prime} \mathbf {F} (f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x}) \| _ {2} ^ {2} \right] \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {\mathbf {M} ^ {\prime}} \left[ \sum_ {j} w _ {j} ^ {2} m _ {j} ^ {2} [ \mathbf {F} (f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x}) ] _ {j} ^ {2} \right] \\ \stackrel {\text {(i i)}} {=} \sum_ {j} [ \mathbf {F} (f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x}) ] _ {j} ^ {2} \\ = \| f _ {\boldsymbol {\theta}} (\mathbf {y}) - \mathbf {x} \| _ {2} ^ {2}, \\ \end{array}
$$
where equation (i) follows from  $\mathbf{M}'$  and  $\mathbf{W}$  being diagonal matrices and equation (ii) from  $\mathbf{W} = \mathbb{E}\left[\mathbf{M}'\right]^{-1/2}$  the condition in Proposition 2.
B Proof of Theorem 1 in Section 3
Our result is based on a relatively standard analysis of the convergence of the stochastic gradient method. In order to minimize the loss, we perform one pass of the stochastic gradient method, i.e., starting at  $\theta_0 = 0$ , we compute the iterations
$$
\boldsymbol {\theta} _ {k + 1} = \boldsymbol {\theta} _ {k} - \alpha_ {k} G _ {k} (\boldsymbol {\theta} _ {k}), \quad k = 1, \dots , N,
$$
where, for notational convenience, we defined the stochastic gradient
$$
G _ {k} (\boldsymbol {\theta}) = \nabla_ {\boldsymbol {\theta}} \| f _ {\boldsymbol {\theta}} \left(\mathbf {y} _ {k}\right) - \mathbf {y} _ {k} ^ {\prime} \| _ {2} ^ {2}, \tag {6}
$$
where the pair  $(\mathbf{y}_k, \mathbf{y}_k')$  is chosen from the data distribution, independently over  $k$ .
The stochastic gradient is