Analyzing the Sample Complexity of Self-Supervised Image Reconstruction Methods
Authors: Tobit Klug
Dogukan Atik
Reinhard Heckel
Venue: NeurIPS 2023 poster (2023)
Keywords: image reconstruction
denoising
accelerated MRI
self-supervised
sample complexity
TL;DR: We analyze the sample complexity of models for image reconstruction problems trained in a self-supervised manner.

Abstract: Supervised training of deep neural networks on pairs of clean image and noisy measurement achieves state-of-the-art performance for many image reconstruction tasks, but such training pairs are difficult to collect. Self-supervised methods enable training based on noisy measurements only, without clean images. In this work, we investigate the cost of self-supervised training in terms of sample complexity for a class of self-supervised methods that enable the computation of unbiased estimates of gradients of the supervised loss, including noise2noise methods. We analytically show that a model trained with such self-supervised training is as good as the same model trained in a supervised fashion, but self-supervised training requires more examples than supervised training. We then study self-supervised denoising and accelerated MRI empirically and characterize the cost of self-supervised training in terms of the number of additional samples required, and find that the performance gap between self-supervised and supervised training vanishes as a function of the training examples, at a problem-dependent rate, as predicted by our theory.

Analyzing the Sample Complexity of Self-Supervised Image Reconstruction Methods
Tobit Klug, Dogukan Atik, Reinhard Heckel
School of Computation, Information and Technology Technical University of Munich tobit. tobit.klug, dogukan.atik, reinhard. heckel {@tum.de)
Abstract
Supervised training of deep neural networks on pairs of clean image and noisy measurement achieves cate-of-the-art performance for many image reconstruction tasks, but such training pairs are difficult to collect. Self-supervised methods enable training based noisy measurements only, without clean images. In this work, we investigate the cost of self-supervised training in terms of sample complexity for a class of self-supervised methods that enable the computation of unbiased estimates of gradients of the supervised loss, including noise2noise methods. We analytically show that model trained with such self supe-vupe training is as good the same model trained in a supervised fashion, but self-supervised training requires more examples than supervised training. We then study self-supervised denoising and accelerated MRI empirically and characterize the cost of self- supervised training in terms of the number of additional samples required, and find that the performance gap between self-supervised and supervised training vanishes as function of the training examples, depeeded rate, predicted by our theory.
Introduction
Deep neural networks trained a supervised fashion to map a noisy measurement to clean image achieve state-of-the-ard performance for image reconstruction tasks including image denoising [48]
12]. image super-resolution g 26 and accelerated magnetic resonance imaging (MRI) [3
However, collecting clean training images is sometimes not possible, and is often expensive and time-consuming. For example, to collect clean target images for denoising is difficult since sensor in camera only collects noisy images (33). To collect target images for MRI is difficult as it requires acquiring fully sampled data with long sean times which, introduces difficulties like increased motion artifacts.
This has motivated research on self-supervised methods that enable the training of neural networks from noisy measurements of images only. Self-supervised approaches are generally based on constructing a self-supervised loss. For example, noise2noise (24) constructs self-supervised loss based on a noisy image and second noisy observation of the same image. Noisier2noise [30), noise2self [2), and beighbor2neighbor [18] construct losses based on a single noisy image for denoising. One might expect that a model trained with a self-superviser loss computed from noisy images performs worse than the same model trained in a supervised manner. And indeed, for image reconstruction problems model trained with many of the self-supervised losses (in particular noisier2noise, noise2self, and neighbor2neighbor performs worse than model trained with supervised loss, even when abundant training data is available.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, with infinite training data, noise2noise (24) self-supervised training promises to achieve the performance of supervised training, since the noise2noise loss enables the computation of unbiased estimates of the gradients of the supervised loss.
In this paper, we study a class of self-supervised methods including noise2noise based constructing unbiased estimates of the gradients of the supervised loss. We characterize the cost of self-supervised training in terms of the number of training examples required to achieve the same performance as supervised training. Our contributions are:
Finite sample theory. We start by viewing the noise2noise loss enabling computation of unbiased estimates of the gradients of the supervised loss. Based on this view, we characterize the sample complexity of noise2noise like self- training. We find that the risk of method trained in a noise2noise- fashion as a function of training examples approaches the optimal risk at the same rate supervised training (i.c., at rate 1/N, where is the number of training examples). However, to reach the same performance as with supervised training more training examples required. The amount of extra training examples required is function of the sup-suueee loss and of the reconstruction problem.
Empirical sample complexity of self-supervised denoising. We empirically characterize the performance of noise2noise supervvue training for Gaussian and camera-nov noise image denoising a function of the training examples, and find that, as predicted by theory, once the training set size becomes sufficiently large noise2noise training yields a denoiser essentially on par with supervised training. For Gaussian denoising, we also characterize the performance of noisier2noise and neighhor2neighhor like training as a function of the number of training examples, and find, again as expected, models trained with a noisier2noise and neighbor2neighhor self supersueee loss perform worse than models trained in a supervised manner, even when abundant training data is available.
Empirical sample complexity of self-supervised compressive sensing. Finally, we charac- terize the performance of noise2noise- from Yaman et al. (45) and Millard and Chiew (29] for compressive sensing reconstruction as a function of the training examples. Again the performance gap between models trained with a self-supervised and supervised loss goes to zero function of the training examples at a problem dependent rate. Perhaps surprisingly, the performance gap is vanishing already for small training set sizes because the gradients constructed from the self-supervised loss have similar variance than the gradients constructed from the supervised loss.
superviedd training similar to the approaches
Together, our results show that networks trained with a self-supervised loss based on computing unbiased estimates of the gradients of the supervised loss perform as well as model trained a supervised fashion at the cost of additional training examples required.
Related work. We consider self-supervisee learning methods since they enable training neural networks for problems where clean examples are not available or are searce. We discuss the most related works throughout the paper. Another training technique that is useful when clean data is searce are data augmentation techniques 7 [0]. Another class of methods that interesting for the regime, where no clean data is available are zero-shot methods that are not based on any data, such as dcep-image-prion based methods [40] [5 [4 6). There is also a variety of zero shot methods that rely partly self-supervised loss functions, such as Yaman et al. [44]'s 's method for compressive sensing,
noise2last (25), and zero-shot noise2noise 28
2 Background on self-supervised learning based on estimates of gradients of the supervised loss
Let fo: 1 + _ be a neural network with parameters 0 for estimating an image x E I based on a measurement E R Our goal is to find a neural network fo that has small risk
R () (x)) [((fo(y), x)]
Here, expectation over the signal- measul-mmasurementt pairs (x, y) (for example clean image x and noisy measurement y x + 2), and is supervised loss, which we take the mean-squared error.

---

Since the data distribution is unknown, we can't compute and minimize the risk. In supervised training, we approximate the risk with an empirical risk computed based on pairs of training examples. This requires pairs of ground-truth signal x and associated measurements, y.
We consider self-supervised training with self-supervised loss that, unlike the supervised loss, does not require ground-truth images. Instead, it depends on another, randomized measurement of the ground-truth image, denoted by y'. We are given N pairs of randomized measurement and original measurement v1)........ (Yx. NN), and train a network to reconstruct a clean image from the original measurement by minimizing
N
1 N (o(())))))
Las (
=
Σ
(2)
We consider loss functions lss, like the noise2noise loss discussed below, with the property that in expectation over the training data, gradient of the self-supervised loss is also gradient of the risk R(O), and thus with sufficiently many training examples, we expect network trained with such a self-supervised loss to achieve the same performance as when trained in supervised manner.
However, the 'noise' induced by relying on randomized measurements of the original image instead of relying the original image increases the variance of the gradients computed from the self-supervised loss. Therefore, self-supervised training requires more training examples to yield a network on par with supervised training. Next, discuss two self-supervised losses, one for denoising and one for compressive sensing in the context of MRI.
2.1 Noise2noise loss for denoising
For denoising, our goal is train neural network fe estimate an image from a noisy observation = x + z, where z is additive Gaussian, non-Gaussian, or even structured noise. Noise2noise assumes access to pairs of noisy observations (y1.y'),.... (yN.y'), y'v), where y; = x; + z; and y x; te. Here, e; is zero-mean noise, independent of but not necessarily of the same distribution as the noise z, Lehtinen et al. (24) introduced the (noise2noise) self sulf-supe loss
(ss(fely). (ly - y'
(3)
which aims at finding network that predicts one noisy observation of an image based on another one, therefore the name noise2noise.
In expectation, a minimizer of the self self-supervised loss is also minimizer of the associated risk (I). as formalized by the proposition below. Thus, training with the celf-super loss (3) with infinitely many training examples is as good as supervised training with infinitely many training examples. The proof in Appendix [
Proposition 1. Suppose that a signal x and a corresponding measurement y are drawn from a joint distribution, and let y = + be another randomized measurement of the signal. Assume that the noise e uncorrelated with the residual, i.e., Exxxy.)) (Joly) -x)" = 0, for all 0. Then, the
minimizer f of the self-supervised risk Eiy [[ - y116]] is also a minimizer of the supervised risk (xxx)) [IIool)) -
Noise2noise self-supervised training has been applied to many domains including biomedical imag- ing 5 19 channel estimation [5]] or acoustic sensing (3
2.2 Compressive sensing
For accelerated MRI, our goal is to train a network fe to reconstruct an image x from an undersampled measurement y MFx in the frequency domain. Here, M E I "x is undersampling mask (i.e., a diagonal matrix with zeros and ones on its diagonal), and F Caxn is the Fourier transform.
We do not have access to ground-truth image Instead, we are given an undersampled measurement y MFx as well as a second randomized measurement of the same image, obtained as y' = M'Fx, where M' E gaxx is randomized mask which is zero or its diagonal, and is one with
3

---

non-zero probability, so that w = E [M'] 1/2 exists. From this measurement, we can compute self-supervised loss defined as
(ss(foly). [W(
(4)
where W = EE [M' -1/2 is diagonal weighting mask. The loss is evaluated only the frequencies that are given through the randomized mask M' and are weighted by how often a frequency occurs in the randomized mask M' (thus, the multiplication with the weighting mask W).
Analogous as for the noise2noise loss for denoising, expectation, a minimizer of the self-supervised loss is also a minimizer of the associated risk, as formalized by the following proposition. The proof Appendix A
Proposition 2. Suppose that signal x and a corresponding measurement y are drawn from some joint distribution (e.g., as y MFx, where M is mask), and let y = M'Fx be an independent measurement taken with a randomized mask M' with O's or I's on it's diagonal and non-zero probability of a one on the diagonal. Then minimizer O of the self-supervised risk [W foly) - -Y)1 with W = E [M]
E(yar")
1/2 1/2 is also a minimizer of the supervised risk
(xxy))
[ - x12
variety of works consider training neural network with self-supervised loss similar to (4)
45, 42. 17
T6. 52 29]. However, none of those works studied the sample complexity. Further,
Lehtinen et al. 24 train with noise2noise loss, but sample new masks M and M' in every training
epoch, which requires access to fully sampled measurements. Similar to setup, Yaman et al.
[45] and Millard and Chiew (29) fix the set of undersampled measurements per image available for creating the masks M and M'. However, Yaman et al. [55 and Millard and Chiew (29 construct noiser2noise- self-supervised loss, i.e., the input mask M implements a higher undersampling
factor on the training inputs than the undersampling factor used at inference. We assume the mask M
to have the same distribution at training and inference, which is important for Proposition 2 to hold.
3 Finite sample theory for self-supervised denoising
We start by studying the performance of denoiser trained with a self-supervised noise2noise loss analytically in the finite sample regime. We measure performance in terms of the risk R(0)
E f - x 2 and learn an estimator fe by minimizing the self ceff-supervised objective function where we take the noise2noise loss as the self-supervised loss.
Finite-sample risk bound for linear denoising. We consider simple linear denoising problem, where the joint distribution of the image and corresponding measurement is as follows. The signal is drawn from d-dimensional linear subspace according to x Uc. where U E xdd is an orthonormal basis for the subspace, and where с ~ (O, 1/dI). Thus, for large d, the vector is drawn approximately uniformly from the intersection of the subspace with the unit sphere. We draw a measurement as = x + z, where z ~ (0, o² /nI) is Gaussian noise, and then draw a second e, where expected SNR of the denoising problem is 1/02-
measurement y'
NN( o (n/nI) is the target noise. With this scaling, the
We consider linear estimator of the form fw (y) Wy. Note that the optimal linear estimator (i.e., the estimator that minimizes the risk) is W I+o?2 UUT. The optimal estimator projects the measurement onto the subspace and shrinks the projected measurement by noise-dependent factor.
We provide bound on the expected risk of the estimator that is learned by minimizing the self- supervised loss function 2 by running the stochastic gradient method for one epoch. Given a dataset D = ((y,.y)).... (yN, } consisting of pairs of two noisy measurements y; = x; + 2., = x; + drawn i.i.d. from the distribution specified above, we start the stochastic gradient method at
W, 0 = and update
W A++
W as V Wyk - -yhl
for k = 1,... N. Here, ag is the stepsize.

---

10a
0.=0.2
o. 0.2
0==0.
R(W) - -R(WW)))))
10-
0.
0.=0
R(W)
10-
R(W*)
1/N
10
10
10-3
10
10"
10
10a
10
10
10
10a
10a
Training set size N
Training set size N
Figure Subspace denoising. Simulated risk of linear estimator learned with early stopped gradient descent for different levels of target noise de signal dimension d' = 10, ambient dimension 100, and input noise level o; 0.1. Left: The larger the target noise level, the more training examples are required for self-supervised training to approach the optimal risk R R(W*). Right: Plotting the risk minus the optimal risk reveals the convergence rate be 1/N. Error bars show the standard deviation over 5 independent runs.
Theorem 1. Consider the estimate W, obtained by running the SGM for N iterations the training set 'D with decaying stepsize ag ctk where c a constant. Then, the expected generalization error, where expectation over the random training set D. obeys
1 E [[(W R(W") + (o2/n)² -
12o + +
2
(5)
The proof (detailed in Appendix B). is based on standard convergence analysis of the stochastic gradient method [32] 43 35). In a nutshell, from the noise2noise self-suuervised loss we compute stochastic gradients of the risk as (fss(fo (y:), y(), where (y;, yy), is training example. This stochastic gradient is an unbiased estimate of the gradient of the risk, VR(O). The variance of the stochastic gradient determines the rate on the RHS of equation
Theorem Daarremmeessllliin esstablishes that the risk is upper bounded by problem-dependent noise floor, which is the risk of the optimal estimator, R(W*) plus a term associated with minimizing a self-supervised loss constructed from finitely many training examples. This term decays as c/N, which is exactly the rate we see in the simulations below for this setup. Moreover, the term associated with minimizing the self-supervised loss becomes larger in the noise variance o2 and thus reflects that the self-supervised 5. loss is a worse approximation of the supervised loss, as oe increases.
Numerical results for linear denoising. Figure 1 shows the risk of the linear denoiser W trained on N examples from the linear subspace model. The linear estimator is learned by applying gradient descent to the self-supervised loss function 2 regularized with early stopping. As predicted by Theorem I the performance of the linear estimator W converges to the performance of the optimal estimator W at the rate 1/N (right plot in Figure I). Also as predicted by the Theorem, larger levels of target noise a. require more training examples, reflected by the term that multiplies with 1/N increasing in ?
Empirical results for self-supervised denoising
In this section, we study the performance of network trained for denoising with the self-supervised objective function (2) and noise2noise loss function of the number of training examples, and compare to the performance of the network trained in a supervised fashion. We consider Gaussian denoising and denosing real-world-came
noise.
Throughout this section, we focus on training a U-net 36) with 7.4M network parameters. U-net good choice, since it is widely used and gives near SOTA models for image denoising 4 12 50
49 However, our qualitative results are independent of the network architecture. In the appendix, present results for SwinIR (26). attention-base based SOTA architecture, confirming this.
5

---

32
31.52 dB
mean=0.77
mean=
PSNR (dB)
31
Frequency
mean=1137
30
Supervised a,=0
Noise2noise 225
Noisc2noise di =50
29
Neighbor2ncighhbr
Noister2noise
102
10a
10a
109
2
3
Training set size N
NMSE
Figure 2: Gaussian image denoising, Left: Noise2noise training approaches the performance of supervised training as the number of training images increases rate dependent on the target noise level oee Networks trained with noisier2noise and neighbor2neighhor also improve as a function of the training examples, but are far from approaching the performance of networks trained in supervised fashion. Right: Histogram of the variance of the stochastic gradients. The noisier the gradients, the slower the convergence function of the training examples, N.
4.1 Gaussian denoising
Setup. We consider denoising images x from noisy measurements y x + with noise level o, 25 (for pixel values in [0, 255)). For the noisy targets y' x + c, we study noise levels o, E {0, 25, 50}. where o, = 0 corresponds to supervised training. The target noise level determines how fast as function the training set size self-supervised approaches the performance of supervised training. In all our experiments, the measurement noise 2, and target noise are only sampled once per clean image x; during training, which corresponds to the practical setup in which we are given two noisy measurements per image. We use cropped patches of size 128 x 128 from ImageNet (37) to create training sets with 100 to 300k images. We pick the best run out of different runs with independently sampled network initialization. See Appendix C. for training details.
Results and discussion. Figure 2 shows the performance in PSNR of U-nets trained in supervised and self-supervised manner for different training set sizes N. We find, as expected from the theory in Section 3] that the gap between noise2noise like self-supervised training and supervised training vanishes function of the training examples, a rate that increases de decreases.
At 100 training images the performance gap between supervised and self-supervised training with o, = 25 (a, 50) is 0.645dB (1.497dB), and at 100k training images the gap is reduced to 0.097dB (0.277 dB). For this specific setup, we can read of the number additional images required to meet a certain denoising performance relative to what can be achieved with supervised training. For example, to achieve the performance supervised training on 3k images, self-supervised training with noise on the training targets oe 25 (a 50) requires 10k (30k) images (gray dashed line in Figure (
The theory in Section B is based on convergence analysis of the stochastic gradient method, and the rate at which performance improves as a function of the number of training examples, N, is determined by how well the stochastic gradients of the risk, Vlss fo (y:), (.)y)) yy) (where (y:, is training example), approximate the gradient of the risk, i.e., V R(0). In Figure 2 (right panel) we show the histograms of the normalized variance of stochastic gradients, i.e., normalized estimates of the MSE VVsss (ye())
- v ( l/2 after one epoch training; see Appendix E for details. can be seen that the larger the variance of the stochastic gradients, the slower the improvement function of the training examples, N.
Self- superviee training with noisier2noise and neighbor2neighbor in general does not yield net- works as good as trained in supervised fashion, even if abundant training data is available. Nois- ier2noise [30] and neighbor2neighbon (18) only require a single noisy measurement per clean image for training. However, noisier2noise adds additional noise to training inputs, which introduces mismatch between the problem solved during training and inference. Neighbor2neighbor relies on assumptions on the similarity between neighboring pixels in the clean image that only hold approximately in practice. In Figure 2 we see that noisier2noise and neighbor2neighbor training,

---

50
M mean=1.44
mean=
PSNR (dB)
48
Frequency
46
44
Supervised 2oise
42
10a
10
10
10
11
0
10-1
10
10
102
Training set size
NMSE
Figure 3: Real-world camera noise denoising on SIDD. Left:: Similar to our results for Gaussian denoising model trained with noisy targets in noise2noise manner approaches the performance of a model trained with ground truth targets supervised manner as the number of training patches gets large. Right: Histogram of the variance of the stochastic gradients. The stochastic gradients of the superviud loss are slightly noisier than the ones of the supervised loss.
unlike noise2noise, yield methods that perform worse than methods trained in supervised fashion even if abundant training data available. For training details see Appendices C.1.3 and C.T.4
For the results in Figure 2 we consider a U-net of fixed size. The performance of a network trained in supervised and self-supervised manner depends on the network architecture and size. However, since we investigate training schemes, we expect our qualitative findings for relative performance differences to continue to hold for other architectures and network sizes. In Appendix C.I.I we provide results for U-nets of varying sizes and in Appendix C.I.5 we provide results for SwinIR (6)]. recent tfate-of-the-an network for image denoising. The results are as expected analogous to the ones for the U-net presented here.
4.2 Real-world camera noise
We now study self-supervised denoising for real-world camera noise from the Smartphone Image Denoising Dataset (SIDD) T]. The noise in the SID dataset is structured and not Gaussian.
Setup. We train U-nets on increasing amounts of patches of size 128 x 128 from the training set, validate on the first 10 scenes, and report the test performance on the remaining 30 scenes from the validation set. The dataset consists of 1,50 noisy images per scene of which 2 are used as input and target for self-supervised training. Ground truth images for supervised training are estimated from all 150 noisy images. Training and testing is done in the raw-RGB space; Appendix C.2 for details.
that performance improves steeper function of the training set size compared to Gaussian denoising in Figure which we attribute to the high variety of noise levels and lighting conditions present in the SID dataset. However, analogous to the Gaussian case the model trained in a self-supervised manner closely approaches the performance of the model trained in supervised manner as the number of training examples grows large.
Results and discussion. In Figure 3
5 Empirical results for self-supervised compressive sensing
We now study the performance function of the training set size of network trained for compres- sive sensing reconstruction by minimizing the self-supervised objective (2) with the noise2noise-like loss relative to the performance achieved by the same network with supervised training.
We perform two sets of experiments: First we experiment on natural images obtained from ImageNet, since this allows us to explore the compressive sensing problem over wider range of training set sizes than existing medical image datasets allow. Then, we investigate accelerated MRI reconstruction on real world multi-coil measurements from the fastMRI dataset [46). We consider a U-net in the main body, but again our qualitative results are independent of the network, and we demonstrate this with experiments on the VarNet, ofte-of-the-an architecture, in the appendix.
7

---

Self-supervised loss
M
Y
[ - -
F
M'F
M
Figure 4: Self-supervised training scheme for compressive sensing MRI. The overlap between constructed measurements y, is indicated in green and stems from both masks sampling center frequencies with probability and an expected overlap of non-center frequencies of size p'q. The scheme depicts single-coil measurements for conciseness.
Sampling scheme. The setup from Section 2.2 requires pair of source measurement y (that can be arbitrarily obtained) and target measurement that is sampled independently of the source measurement as y' = M'Fx, where M' is random mask with non-zero probability of selecting each frequency. While in clinical MRI protocols, we can acquire two independently undersampled measurements per image, typically acquire only one. We therefore slightly adapt our sampling scheme as follows to generate two measurements y and y' from one undersampled measurement y that is undersampled with factor μ.. See Figure for illustration of the scheme.
Our goal is to train a network to perform reconstruction based on measurement y undersampled with factor p < We consider Cartesian undersampling, where the frequency domain (also called k-space) sampled column-wise, Since the frequencyy components of an image contain the largest portion of the signal energy, we follow the common practice to sample a fraction of the center columns in the k-space with probability and assign the center frequencies to both measurements y and y'. We then add a random fraction P=v of the non-center columns in ý y. which is now a measurement undersampled with factor p by design. The target y' is constructed by adding the remaining frequencies that in ý but not in y, and an additional fraction p'q of the non-center columns in y, where p' p-2 The rationale behind this approach is that if the masks M and M' sample fractions p' and g independently from all non-center frequencies, then the expected overlap of frequencies selected by both masks is p'a q with K-P
0.08
μーv
I-
1-p"
We compute the diagonal weight mask from equation (4) as W = E I/ which yields I's at entries corresponding center frequencies and 1/ / otherwise. In our experiments we set y 0.08,
p 0.255 and {0.28, 0.33) resulting in E {0.05, 0.11).
5.1 Compressive sensing for natural images
We start with compressive sensing experiments natural images, because the largest existing MRI dataset for image reconstruction research (fastMRI) only has 55k images, and we want to vary the training set size far beyond that. In the next section we conduct experiments fastMRI.
Setup. We use cropped images of size 100 x 100 from ImageNet 37 as ground truth images and obtain undersampled complex-vall valued measurements in the frequency domain as y = MFx and y' M'Fx. We draw random training sets with 50 to 1M patches from fixed pool of IM images.
For all training set sizes, we train a U-net with about 1M network parameters and pick the best run out of different runs with independently sampled training set and network initialization. The network takes as input a coarse reconstruction obtained as F-ly, where y is the zero-filled measurement (i.e., so(y) U--eto(F-1y (F-ly). The measurements are complex-valus valued, and we take one channel of the U-net as the imaginary part and one as the real part. The reconstructed image obtained from the complex- cimplex-valued network output by taking the absolute value. See Appendix D.I for training details.
8

---

0.80
meam=114 11.4
meem=23.3
0.75
Frequency
meanm43.5
SSIM
0.70
Supervised a=l
self-sup, ,=00.33
Self p=0.28
0.65
Zero-filled baseline
10a
103
104
10
10c
10
10a
Training set size N
NMSE
The larger the fraction of frequencies in the target, μ., the faster the network's performance improves as a function of the number of training examples, N. As the training set size becomes large, self-supervised performance approaches supervised performance. Right: Histogram of the variance of the stochastic gradients. The noisier the gradients (i.e., the smaller μ)), the slower the convergence function of the training examples.
Figure 5: Compressive sensing for natural images. Left:
Results and discussion. Figure 5 shows the performance in SSIM as a function of the training set size for supervised and self self-supervised training with fractions μ = 0.33 and = 0.28 of all frequencies available for self superrise training. Note that the fraction of frequencies in the target plays the same role as the noise variance for noise2noise denoising: The larger the fraction, the better in general. Since keep the undersampling factor p 0.25 of the training inputs y fixed, smaller implies a smaller undersampling factor q (less frequencies) of the training targets y'.
Networks trained in self-supervised manner with sufficiently many training examples perform as well as the same network trained a supervised manner (but with fewer examples). The fewer measurements are available, i.e., the smaller jr, the more training examples are required to achieve the same performance as the same network trained in a supervised manner. This parallels our theoretical and empirical findings for denoising where noise2noise training approaches the performance of supervised training as function of the training set size and at a rate dependent of the training targets.
Figure 5 right panel, depicts histograms of the variance of the stochastic gradients. The noisier the gradients, the slower the performance increase as function of the training examples.
The performance of the zero-filled baseline in Figure S is computed between the network inputs F-ly and ground truth images in the test set. As we can see self-supervised training requires least 250 training images to outperform this trivial baseline.
5.2 Compressive sensing MRI
Next, we discuss the performance of noise2noise self-supervised and supervised training for multi-coil MRI reconstruction. MRI scanners use parallel imaging, where single scan simultaneously collects measurements Y1.....ye Yc the same object with C' coils, where y, MFS, with complex valued images x E C and diagonal coil specific sensitivity maps S, E Cuxa, which we estimate from the center of the k-space using ESPIRIT
Setup. We conduct our experiments on subset of the fastMRI brain dataset 46 We design fixed training sets with 50 to 50k images. We train U-nets with 31M network parameters. To demonstrate that results for comparing supervised and self- self-supervised training schemes translate qualitatively to other network architectures, we further train end-to-end variational networks (E2E- VarNet) (38), state-of-the-art architecture, of size 9M parameters. We pick the best run out of different runs with independently sampled network initialization.. See Appendix [.2 for training details.
The network inputs are computed through SENSE-1 coil combination (34 of the coil measurements Σ S where S denotes the complex conjugate of the sensitivity map. Ground truth images for evaluation are computed analogously E S,Fuly,,unl, where yj, the fully sampled coil k-space provided by the fastMRI dataset. To compute the training loss in the k-space between given and predicted coil measurements, the set of predicted coil measurements obtained from the network output as FS, fo()........ FScJely).

---

0.93
2.can=2.23
0.90
mean==228
Frequency
SSIM
0.87
0.84
Supervised jml
Self-sup. p=0.33
Zem filled haseline
0.81
10a
103
104
0.5
2
4
8
16
Training size
NMSE
Figure 6: Compressive sensing MRI. Left: Already at small training set sizes the performance gap between supervised and self-supervised training vanishes. Right: The small performance gap on the left is explained by small difference in the variances of the supervised and self-supervised gradients.
We follow Yaman et al. 445 and make best use of the given data by re-sampling the input mask M in every epoch for supervised training. For self-supervised training the given frequencies y are fixed, but we can re- re-sa the split into measurements y,y'. As the network sees more combinations of input/target pairs, the performance of supervised and supervvee training improves.
Results and discussion. Figure 6 shows the performance of U-nets trained in supervised and self- supervised way for multi-coil MRI as function of the training set size. We report the performance in SSIM, since high score in SSIM was shown to correlate well with high rating by radiologists in the fastMRI challenge B1.. We find that already for small training set sizes the performance gap between supervised and self- suee--uueeeieed training vanishes to approximately 0.002 in SSIM.
Figure 6 right panel, depicts the histogram of the variance of the stochastic gradients. The difference in variance of the supervised and self-supervised gradients is very small, which explains the small performance gap between supervised and self-supervised training for this setup. The performance of the zero-filled baseline in Figure 6 is computed between the magnitude of the network inputs EE S;F Fyyl and ground truth images in the test
In Appendix D.2.2 we provide additional results for the E2E- VarNet. As expected, the VarNet outperforms the U-net by a significant margin for every training set size, especially for small training set sizes. However, the relative performance gap between VarNets trained in supervised and self-supervised way is very small, analogous as for the U-net.
Conclusion
We characterized the sample complexity of deep learning based image reconstruction models trained with class self-supervised losses including noise2noise based on computing unbiased estimates of the gradients of the supervised loss. Our work shows that if sufficiently many training examples are available, as good to work with pairs of independent measurements as with training data with ground-truth images. In particular for accelerated MRI, training networks only based undersampled data sufficient for peak performance.
One advantage of supervised training over self-supervised training that is not reflected in our results that supervised training has more flexibility choosing the loss function, and choosing loss function such as SSIM might give better visual image quality than training with the MSE. Our results of self-supervised training approaching supervised training pertain to training with MSE
We hasten to note that the class of methods that enable the computation of unbiased estimates of gradients of the supervised loss studied in this work, in practice limited problem setups in which obtaining two independent measurements of the same underlying signal is feasible.
Reproducibility
The repository at hhttp:::/// ptttps://githan //g.ttub.b.....t.. ht/g/tthb.ccom//M.I.. cww/com///..... http///Lllabb/tmmpee_oommmpppppp
recon contains the code to reproduce all results in the main body of this paper.
10

---

Acknowledgments
The authors are supported by the Institute of Advanced Studies at the Technical University of Munich, the Deutsche Forschungsemmeischatfi (DFG, German Research Foundation) 456465471, 464123524, the DAAD, the German Federal Ministry of Education and Research, and the Bavarian State Ministry for Science and the Arts. The authors also acknowledge the financial support by the Federal Ministry of Education and Research of Germany in the programme of "Souveraen. Digital. Vernetzt.". Joint project 6G-life, project identification number: 16KISK002.
References
[1] A. Abdelhamed, S. Lin, and M. S. Brown. "A Bugh-Quality Denoising Dataset for Smartphone Cameras". In: JEEE Conference on Computer Vision and Pattern Recognition. 2018.
[2] J. Batson and L. Royer. "Noise2Self: Blind Denoising by Self-Superrision.. Conference on Machine Learning. 2019.
In: International
[3] T. Bepler, K. Kelley, A. J. Noble, and B. Berger. "Topaz-Denniise:: General Deep Denoising Models for cryoEM and cryoFT". In: Nature Communications (2020).
[4] T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, and J. T. Barron. "Unprocessing Images for Learned Raw Denoising". In: IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[5] T.-O. Buchholz, M. Jordan, G. Pigino, and F. Jug. "Cryo-CARE: Content-Aware Image Restora- tion for Cryo-Transmission Electron Microseopy Data". In: IEEE International Symposium on Biomedical Imaging- 2019.
[6] M. Z. Darestani and R. Heckel. "Accelerated MRI With Un- Un-Trained Neural Networks". In: IEEE Transactions on Computational Imaging (2021).
[7] A. D. Desai, B. Gunel, B. M. Ozturkler, H. Beg, S. Vasanawala, B. A. Hargreaves, C. Ré,
J. M. Pauly, and A. S. Chaudhari. "VORTEX: Physics-Driven Data Augmentations Using Consistency Training for Robust Accelerated MRI Reconstruction" In: Medical Imaging with
Deep Learning. 2022.
[8] C. Dong, C. C. Loy. K. He, and X. Tang. "Image Ceper-Resoouttinn Using Deep Convolutional Networks". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2016).
[9] Dubois, T. Hashimoto, and P. Liang. "Evaluating Self-Supervised Learning via Risk De- composition". In: arXiv:2302..m 03068 [cs, stat] (2023).
[10] Z. Fabian, R. Heckel, and M. Soltanolkotabi. "Data Augmentation for Deep Learning Based Accelerated MRI Reconstruction with Limited Data". In: International Conference on Machine Learning. 2021.
[11] Z. Fabian, B. Tinaz, and M. Soltanolkotabi. "HUMUS-Net, Hybrid Unrolled Multi-scal scale Net- work Architecture for Accelerated MRI Reconstruction". In: Advances in Neural Information Processing Systems (2022).
[12] J. Murrola-Ramos O. Dalmau, and T. E. Alarcón. "A Residual Dense U-Net Neural Network for Image Denoising". In: JEEE Access (2021).
[13] K. Hammernik, T. Klatzer, Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and Knoll. "Learning a Variational Network for Reconstruction of Accelerated MRI Data". " In: Magnetic Resonance in Medicine (2018).
[14] R. Heckel and Hand. "Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks". In: International Conference Learning Representations (2019).
[15] R. Heckel and M. Soltanolkotabi. "Compressive Sensing with Un-Trained Neural Networks: Gradient Descent Finds the Smoothest Approximation". In: International Conference on Machine Learning (2020).
[16] Hu, C. Li, H. Wang, Q. Liu, H. Zheng, and S. Wang. "Self-Supervise" Learning for MRI Reconstruction with Parallel Network Training Framework". In: Medical Image Computing and Computer Assisted Intervention. 2021.
[17] P. Huang, C. Zhang, H. Li, S. K. Gaire, R. Liu, X. Zhang, X. Li, L. Dong, and L. Ying. "Unsupervised Deep Learning Reconstruction Using the MR Imaging Model". In: Annual Meeting of the International Society for Magnetic Resonance in Medicine 28 (2020).
11

---

[18] T. Huang, S. Li, X. Jia, H. Lu, and J. Liu. "Neighbor2Neighbor:
Denoising From Single Noisy Images". In: IEEE Conference on Computer Vision and Pattern Recognition. 2021.
[19] Y. Huang, N. Zhang, and Q. Hao. "Real-Time Noise Reduction Based on Ground Truth Free
Deep Learning for Optical Coherence Tomography". In: Biomedical Optics Express (2021).
[20] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford,
J. Wu, and D. Amodei. "Scaling Laws for Neural Language Models", In: arXiv:2001.0832
[cs, stat] (2020).
[21] D. P. Kingma and J. Ba. "Adam: A Method for Stochastic Optimization". In: International
Conference on Learning Representations (2014).
[22] T. Klug and R. Heckel. "Scaling Laws For Deep Learning Based Image Reconstruction". In:
International Conference on Learning Representations. 2023.
[23] S. Lapins. A. Butcher, J.-M. Kendall, T. S. Hudson, A. L. Stork, M. J. Werner, Gunning, and
A. M. Brishourne. "DAS-N2N: Machine Learning Distributed Acoustic Sensing (DAS) Signal Denoising without Clean Data". In: arXiv:2.304.0 /8/20 [physics] (2023).
[24] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila. "Noise2Noise: Learning Image Restoration without Clean Data". In: International Conference on Machine Learning. 2018.
[25] J. Lequyer, R. Philip. A. Sharma, W.-H. Hsu, and L. Pelletier. "A Fast Blind Zero-Shot Denoiser". In: Nature Machine Intelligence (2022).
[26] J. Liang. J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. "SwinIR: Image Restoration Using Swin Transformer". In: IEEE International Conference on Computer Vision Workshops. 2021.
[27] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows". In: IEEE International Conference on Computer Vision. 2021.
[28] Y. Mansour and R. Heckel. "Zero-Shot Noise2Noise: Efficient Image Denoising Without Any Data". In: IEEE Conference on Computer Vision and Pattern Recognition. 2023.
[29] C. Millard and M. Chiew. "A Theoretical Framework for Self-Supervised MR Image Recon- struction Using Sub-Sampling via Variable Density Noisier2Noise" In: arXiv:2205.000 [cs, eess] (2023).
[30] N. Moran, D. Schmidt, Y. Zhong, and P. Coady. "Noisier2Noise: Learning to Denoise From Unpaired Noisy Data". In: IEEE Conference on Computer Vision and Pattern Recognition. 2020.
[31] M. J. Muckley et al. "Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction". In: IEEE Transactions on Medical Imaging (2021).
[32] A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. "Robust Stochastic Approximation Approach to Stochastic Programming". In: SIAM Journal on Optimization (2009).
[33] T. Plotz and S. Roth. "Benchmarking Denoising Algorithms With Real Photographs". In: IEEE
Conference on Computer Vision and Puttern Recognition. 2017.
[34] K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger. "SENSE: Sensitivity Encoding for Fast MRI" In: Magnetic Resonance in Medicine (1999).
[35] A. Rakhlin, O. Shamir. and K. Sridharan. "Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization". In: International Conference on Machine Learning. 2012. [36] O. Ronneberger, P. Fischer, and T. Brox. "U-Net: Convolutional Networks for Biomedical Image Segmentation". In: Medical Image Computing and Computer-Assisted Intervention (2015).
[37] O. Russakovsky et al. "ImageNet Large Scale Visual Recognition Challenge". In: International Journal of Computer Vision (2015).
[38] A. Sriram, J. Zbontar, T. Murrell, A. Defazio, C. L. Zitnick, N. Yakubova, Knoll, and P. Johnson. FEnd-to-EEd Variational Networks for Accelerated MRI Reconstruction". In: Medical Image Computing and Computer Assisted Intervention. 2020.
[39] M. Uecker, P. Lai, M. J. Murphy, P. Virtue, M. Elad, J. M. Pauly, S. S. Vasanawala, and M. Lustig. "ESPIRiT an Eigenvalue Approach to Autocalibrating Parallel MRI: Where SENSE Meets GRAPPA". In: Magnetic Resonunce in Medicine (2014).
12

---

[40] D. Ulyanov, A. Vedaldi, and V. Lempitsky. "Deep Image Prior". In: JEEE Conference on Computer Vision and Pattern Recognition. 2018.
[41] D. Ulyanov, A. Vedaldi, and V. Lempitsky. "Instance Normalization: The Missing Ingredient for Fast Stylization". In: arXiv: /00.00002 [cs] (2017).
[42] A. Q- Wang, A. V. Dalca, and M. R. Sabuncu. "Neural Network-Based Reconstruction in Compressed Sensing MRI Without Fully-Sane Fully-Sampled Training Data". In: Machine Learning for Medical Image Reconstruction. 2020.
[43] S. Wright and B. Recht. Optimization for Data Analysis. Cambridge, 2022.
[44] B. Yaman, S. A. H. Hosseini, and M. Akcakaya. "Zero-Shot Self-Supervised Learning for MRI Reconstruction" In: International Conference Learning Representations. 2022.
[45] Yaman, A. H. Hosseini, S. Moeller, J. Ellermann, Uğurbil, and M. Akçakaya. "Self- Supervised Learning of Physics-Guides Reconstruction Neural Networks without Fully Sam- pled Reference Data". In: Magnetic Resonance Medicine (2020).
[46] J. Zbontar et al. "fastMRI: An Open Dataset and Benchmarks for Accelerated MRI". In: ArXiv:1811.00 1.08839 [physics, stat) (2018).
[47] X. Zhai, A. Kolesnikov, N. Houlsby, and L.. Beyer. "Scaling Vision Transformers", In: IEEE Conference on Computer Vision and Pattern Recognition. 2022.
[48] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising". In: IEEE Trunsactions on Image Processing (2017).
[49] K. Zhang, Y. Li, J. Liang, J. Cao, Y. Zhang, H. Tang, R. Timofte, and L. Van Gool. "Practical Blind Denoising via Swin-Conv-UNee and Data Synthesis". In: arXiv:2203 [cs, eess] (2022).
[50] Zhang, Y. Li, W. Zuo, L. Zhang, Van Gool, and R. Timofte. "Plug-and-Play Image Restoration with Deep Denoiser Prior". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[51] Q. Zhang, H. Dong, and J. Zhao. "Channel Estimation for Sregh-Speed Railway Wireless Communications: A Generative Adversarial Network Approach" In: Electronics (2023). [52] Zhou, N. Dey, J. Schlemper, S. M. Salehi, C. Liu, S. Duncan, and M. Sofka. "DS- Former: A Dual-Domain Self-Supervised Transformer for Accelerated Multi-Contrast MRI Reconstruction" In: JEEE Winter Conference on Applications of Computer Vision. 2023.
13

---

A Proofs for propositions in Section
2
Proof of Proposition 7
We have that
Ex,y,' [Ilfo(y) - -''le] = Ex
[|s(l)) - x c11
Ex,y
[IJo(y) -x12]
yx (Jo(y) -
x)²
e)
+ E
|
E xx,)
(lfo(y)
-x11
+ E
||e|||
where equation (i) follows from condition in Proposition Noting that the expectation over the norm of the random variable above constant as a function proves the claim.
Proof of proposition 2
We show that
EM [IW(M'F/6(>)>) - - = -IJo(y) - x1
To see this, note that
EM' [ - -3'11a]
EM'
[W(MFff(y) - M'Fx)12
EM'
WWMF( -x11
Σ wfm F(fo(y)) - xx13
EM
F(foly) -x11
llfoly) - x11
where equation (i) follows from M' and W being diagonal matrices and equation (ii) from W E [M]] 1/2 the condition in Proposition
B Proof of Theorem 1 in Section 3
Our result is based relatively standard analysis of the convergence of the stochastic gradient method. In order to minimize the loss, we perform one pass the stochastic gradient method, i.e., starting at On O, compute the iterations
i 1+1 0₁ Ga (6x),
== N,
where, for notational convenience, defined the stochastic gradient
( Vellfo(ya) - -illi
(6)
where the pair (yk.y() is chosen from the data distribution, independently over k.
The stochastic gradient is
G(0))
y'1
Vollse(y) (x+
Vo (Ife(y) - -x122 2 (fe x, e) + ||e|
Vollof(() - -x112 - -J-yc
where Jo.y E RRapp is the Jacobian of the function So with respect to the parameters Note that the function fo has parameters 0 e I2 and maps vector in gn to another vector in R"; consider the
14

---

Jacobian with respect to the parameters, not with respect to the input vector. The stochastic gradient an unbiased estimate the gradient of the risk, i.e.
Vo [I[o(y) -x10) VR(O). I [G(W)]
(7)
The variance of the stochastic gradient is
E
[ E VollJoly)
x112 -x12 J E Vollfo(y) - + de n
2
Jo 11
(8)
We are now ready to prove Theorem Recall that for the theorem, we consider a simple de- Uc, c (0, 1 1/dI) and the measurement y x z, where
noising problem where NO, of. /nI). Also, we consider simple linear estimator f(y) Wy parameterized by the matrix _2xx, For this setup, the risk becomes
R(W) E Wyy x12
1 d (W- DUU + of W
2
n
Thus, the risk is m-strongly convex with o since the risk is bowndebunded by the quadratic WI F. Moreover, the gradient of the risk is
VR(W)= J w no o W,
(9)
and setting this to zero yields the risk-minimizing parameter
W
UUT.
1 2202 2d
Below, we show that the second moment of the risk is (M, bbuunded in that for all W,
E [[G(W)]]
s < w W + B
(10)
where W is minimizer of the risk R(W), and where M 10/d and B 120 d + o + o?).
We proceed by applying the following standard result for the convergence of the SGM. For the sake of completeness, the result proven below.
Lemma 1. Let f be m-strongly convex (i.e., ( f(0) and let G be stochastic gradient that is (M, -bbunded Ássume start the SGM iterates at 0.- = 0, with a stepsize O#k-
m |
? is convex) with minimizer 0
2
Then
2
M2
+k
m2
1 m²
E
1
1
(2M + B²)
Below we show that for all W
R(W) R(W") < (1/d+ ( o2/n)] w
(11)
Thus, an application of Lemma 1 to the RHS yields
1
M²
B²
R(WN) x) R(W") < (1/d + / m)
2
W
m²
m²
1/d 02/n
1
(2 B²),
(02/n)²
2
which concludes the proof of Theorem
I
15

---

B.1 Bound on variance of stochastic gradient, proof of equation (
From equation (
we have
V Wy x|]][]] + oe E
E
G (G(
= E
|[[ + 2, ................................ x 2]1(²²
= E ||((w - -)x+ + Wz)(x + 2) 2>1
| 2) 2)77|11
R + c.(1 + o..
The term R can be bounded
(WW W + W 1Jx + (W- - W + w (x 2)²
R = E
F)
E
[I(W- (xX + xxx + zx + zzz))
(W* I)(xx² + +xx²) + W (2x? + zzz" )|a)
[[W WXX(Xx" xx + 2x" nxx [
S 21E
+ 2E
|I(W* (xx + xxx) + W' (zx" zzz" [
= 2R, 22
We next bound RS and R2.
Bound on Ry: To bound RG, first note that for x ~ NO, 1/dI),
E |xxx "
[r] traccl [Axx" xx" A < 2traceE [Axx" A = 2E ||A|| ?
2
A
d'
Similarly, for x ~ (O0 1/dI) and z ~ (O, /1/
02
AAx 1 [|]] tracelE [Axx" 2x" A A o traceE [Axx" A' =
||Ax| 1
AU |
and
T [ ||x Also similarly,
E
tracelE [Axx xz' A
tracelE' Azz T A = E
Az
02 A 2
F1
E Azz
T
2
<
204
|A F-
follows that
2
20
20
R
<
W
W 1
Bound on Ro: Note that
R₂ <
(I
2
F
2 2+
d
+ |||
12 02
20
2 -
0 20
n
n
02
+ 20
33 3
+ d
d
< 6
6--- o2.
where we used that IW - 1 < (02d)² 2
d and W 13 s d, and for the last inequality we
used that 03 < 1, by assumption.
Application of the bounds on Ri and R₂ above concludes the proof of Equation 8
16

---

B.2
Proof of equation
Note that the risk can be written as
R(W)- = E (wy xx12 where w [w wHl w
E [|\Y -
x12]
(12)
? p²n and Y E gux=" is a block diagonal matrix containing the row-vector I on the diagonal. With this notation.
R(W)- = Yw x|1]
E
Yw Yw"
Yw" - -x12
E
Y((w w
+ 2E (X((w - (YW" -x] + E \ - xx11
E
Y(w - w')16
+ 2(w W' [Y" (Yw" - -x] R(W*)
|Y(w w
+ R(W")
S (1/d+02/m) W-WW| + R(W*)
where equality follows from 0 VR(w*)) = E [2Y" (Yw" xx] for minimizer w and the
last inequality follows from y Uc where (O 1/II) and x (N0
B.3 Proof of Lemma
We start by upper bounding the expectation e₁ = E [10
0- 12 We write Gx((,) G(O,)
for the stochastic gradient at iteration & to emphasize that at each iteration, the gradient is drawn independently from the gradient at other iterations. First note that
00+1 00
1
00x ax - 0
10x 0-112 -2ox (Gx (0x), O2 0-) GZ
The expectation of the middle term above is E [[())(0)),
0²) = E (E (0x))))) 02 0') = E (V 0: 0²)), (13) where we used that the random variable Gg is independent of the random variable G,, < since each stochastic gradient is drawn independently, and is therefore independent of the iterate 0g. Thus, iterating the expectation allows us to replace the stochastic gradient by the gradient.
Using the assumption that the stochastic gradient G is (M, B) bbounded yields
Pk+1 & (1 + a M 20 ((VP(O)). 0s 0*)) o B²
(14)
With Vh(e-) = O. we have that
(V)(0,), 0₂ - 0*) = (vh(O,) - Vh(0"), 02 - 0-) 2 m 0g 0 1|
where the inequality follows from being m-strongly convex. Application of this inequality to yields
(44
ck+1 S (1 + of 20 + of B².
(15)
M²² Since we choose the stepsize as a₁ = 2 2344 with B m yields
we have an S m
Then Equation
15
M
ck+1 S (1 axmmes + B²
Unrolling the iterations gives
ek S (1 ak 0k-im)ee-- + 2 12
< (1 a1-1m)(1 a ( (1 ag-1maa + a B²
k-1
A-1
k-1
(1 - asm)eo + B
(1 - a)m)
(=0
1=0
j=++
k-1
2 k-1
II
1
2
eg + B
2
II
2
2, +1
m. 2, +1
2 +
1=0
j=it1
17

---

32
PSNR (dB)
31
30
Fixed network size, E {0, 25, Adapted network size a. {0, 25, 50)
29
10a
103
10a
10a
Training set size
Figure 7: Effect of adapting the network size for denoising, For each training set size the dark blue curves show the best performing network over range of network sizes (see Figure x 8 all results). Light blue curves show the performance for fixed network size. Since the performance curves of supervised training (a, = O, top curves) and of self-superrised training (øe {25, 50), middle and bottom curves) are affected very similarly from fixing the network size, the relative comparison between supervised and self-supervised training remain valid.
Using that 11 (1-3)²
(1-2)(1-))
(k-1Ik
we get
(23 - (23 - 1)
B
k-1
(28 + 1)(28 +) (k 2) - 1)
ek v
C0 +
(k- (k - 1)
m
2, +
1=0
4
B
S (k: 2)²
+
m² k
1
B
< k 2,80 +
m
which concludes the proof.
Additional material for Section 4 Self-supervised image denoising
C.1
Gaussian image denoising
The Gaussian image denoising experiments are conducted the following datasets. From the ImageNet dataset (37) we reserve 80/230 images for validation/testing, and create training sets Sy of size N ranging from 100 to 300k images and S, S, for i j. For the training set we center crop images to size 128 x 128.
Further, all following experiments that rely on the U-net as the network architecture, use U-net with two blocks the encoder and decoder part respectively and skip connections between blocks. Each block consists of two convolutional layers with LeakyReLU activation and instance normalization [4]] after every layer, where number of channels doubled (halved) after every block (decoder). The downsampling in the encoder is implemented as average pooling and the upsampling the decoder as transposed convolutions. As proposed by Zhang et al. [48] we train a residual denoiser that learns to predict y y' instead of directly predicting y'. which improves performance.
encoder
All experiments were conducted on NVIDIA A40, NVIDIA RTX A6000 and NVIDIA Quadro RTX 6000 GPUs. We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 700 GPU hours for the experiments on self-supervised Gaussian denoising presented in Figure
C.1.1 Noise2noise self-supervised denoising with U-net of varying size
The results in Figure 2 are for a U-net of fixed network size, which is the natural setup since we'n interested in studying differences of training schemes, and therefore we have to keep the network architecture constant.
18

---

However, for different training set sizes, networks of different sizes perform best; if the training set size is larger, typically a larger network performs best. In the following, we therefore perform a comparison of self- supervisde training and supervised training, where we adapt the network size along with the training set size, N.
We find that adapting the network size to the training set size improves the performance of models trained in supervised and a sell-supervised fashion similarly, thus our findings on self-supervised versus supervised training continue to hold even if we adopt the network size as we change the training set size.
To show this, we follow an approach from the scaling law literature, where the goal is to observe the isolated effect of one resource on the model performance by providing unlimited access to all other resources. In the context of image reconstruction Klug and Heckel (22). investigate the performance as function of the training set size without limiting network size and invested compute, i.e., by only considering the best performing model per training set size over a wide range of network sizes and by training each model until the performance converges on the validation set. Similar work has been done in other fields including computer vision [47] and natural language processing (20).
Figure X shows our additional results for adapting the network size for noise2noise self-supervised training with target noise levels de E {0, 25, 50). The panels on the right show the performance as a function of the network size for a fixed training size. The left panels show the performance as function of the training set size, where only the best model per training set size is considered.
Figure 7 compares the three curves with adapted network size from the left panels in Figure 8 to the performance curves for a fixed network size from Figure P
We chose the fixed network to have a moderate size (7.4M) compared to the smallest (0.IM) and largest networks (46.5M) considered for adapting the network size. As consequence in Figure 7 the smallest/largest training set sizes gain most performance from adapting to smaller/larger network size. However, the gain in performance small, which is also indicated by the relatively flat curves in the right panels of Figure 8 which show the scaling of the reconstruction performance as a function of the network size. Further, the three training methods, supervised and self-supervised training with o. E (25, 50}, are affected similarly from fixing the network size. Hence, our findings regarding a relative comparison of supervised and supervised methods are not affected.
Training details for varying the network size. The left column in Figure g is a sum- mary of all models that we trained for Gaussian denoising with a U-net under different levels of noise on the training targets de E {0, 25, 50). For different training set sizes N E {0.1, 0.3, 0.6, 1,3, 6, 10,30 60, 100, 300) thousand images we varied the number of chan- nels in the first layer of the U-net in {16, 32, 64, 128, 192, 256, 320} corresponding to P (0.1, 0.5, 1.9, 7.4, 16.7, 30.0, 36.5) million network parameters.
For small training set sizes the curves in Figure g use the best performance out of up to 5 training runs with different random network initialization, indicated by the additional points underneath the different curves. In Figure 7 only the best runs are reported for brevity. As the variation among runs decreases as the training set size increases, a single run is performed for large training set sizes, and further runs are only added in case of obvious outliers.
We tested two different batch sizes and 10 for training set sizes up 60k images and found that while models trained with small training set sizes significantly benefit from batch size 1, models trained with larger training set sizes are indifferent to the choice of the batch size. Hence, set the batch size to 1 for training set sizes N < 6k and to 10 otherwise.
All models are trained with the Adam optimizer [2]] with 31 0.9 and B2 0.999. As we train on different combinations of network size, batch size and training set size we have to adjust the initial learning rate accordingly.
We found that the following automated scheme for learning rate annealing reliably finds an initial learning rate that lies within the range of suitable learning rates. Starting from a small learning rate 1.25 x 10-6 we double the learning rate after every epoch as long as the validation error improves. training with learning rate for 3 epochs does not yield an improvement, the annealing is terminated. We then continue the training with the model checkpoint corresponding to a learning rate 4-times smaller than the current learning rate.
19

---

(a) Supervised denoising (o, 0)
32
32
PSNR (dB)
31
31
30
30
10a
10
10a
10a
10
10
10
(b) Self-supervised denoising (a.
25)
32
32
PSNR (dB)
31
31
30
30
10a
10a
10a
10a
10a
10
107
(c) Self-supervised denoising (0. 50)
32
32
PSNR (dB)
31
31
30
10a
10a
10a
10
10
10
107
Training set size N
Network parameters P
Figure 8: Complete results for adapting the network size for demoising. Each curve the right is obtained by fixing the training set size and sweeping across different network sizes. Then, the curve the left created by taking the best performing network for each training set size. Colors in the plots on the left and right correspond to the same training set size.
Once the learning rate is set, we apply an automated learning rate decay. We halve the learning rate once the validation PSNR did not improve for eight consecutive epochs. If the learning rate was decayed twice in row without an improvement, the best validation PSNR is picked for evaluation on the test set. Picking the model based on the validation error was necessary since we faced overfitting during training for E {25, 50], as shown in the example in Figure The overfitting to be expected as we fix the noise per training example over all epochs on both noisy inputs and noisy targets y'.
terminate the training. Then, the model with
To perform early stopping we compute self- cufe-supereieed validation PSNR between measurements y and y' of the images x in the validation set. It is well known that the self-supervised validation PSNR proportional to the actual PSNR, see (9) for a discussion. As the eeffssuerriiee training error approximates the supervised one, so does the self-supervised validation error. Figure shows an example for the correlation between self-supervised and supervised validation error.
C.1.2 Noise2noise self-supervised denoising with a U-net of fixed size
The results in Figure 2 for noise2noise self-supervised denoising were obtained with fixed network size with 128 channels the first layer resulting in 7.4M parameters. The training procedure for fixed size networks is similar to training with networks of varying size described in Appendix C.I.I
20

---

32
20.4
Self-sup validation PSNR
Sup. validation PSNR
31
20.3
30
20.2
Supervised Self supervieed
0
20 40
60
80
100
Epochs
Figure 9: Overfitting and supervised vs. self-supervised validation PSNR for denoising. In this example, supervised and self celf-super validation PSNR of a network trained in noise2noise self-supervised manner (with a, = 25) improves until epoch 20. At epoch 20, performance declines due to overfitting to the fixed noise realizations on the training inputs and targets. As supervised and self-supervised PSNR behave qualitatively the same we can use the self-supervised PSNR to pick best model over all epochs. This experiment uses training set of size 100k and a U-net with 320 channels in the first layer.
For 6000 images, we reused the networks trained for the adaptive network scenario from Figure since already have 128-channeled networks for them. The training procedure for those networks is discussed in Appendix C.I.I
For N 2 10.000 images, we used batch size of and as the network size is fix, we also fix the initial learning rate 6.4 10-4 and then applied learning rate decay as described in Appendix C. For the largest training set size N = 300k we trained once with initial learning rate 6.4 x 10 and once with 1.6 x 10-4 and picked the run that performed better.
As in Appendix C.I.I we picked the best out of up to three training runs from different random initializations to account for the variance between runs for small and medium sized training sets.
C.1.3 Noisier2noise self-supervised denoising with a U-net
The performance curve for noisier2noise self-supervised denoising in Figure 2 is obtained with U-net of fixed size with 128 channels in the first layer and 7.4M network parameters in order to match the setup for noise2noise Appendix C.I.2.
In noisier2noise, we have only one noisy measurement per clean image, which is used as the training target. the simplest version presented by Moran et (30 the training input created by injecting additional noise of the same distribution and variance to the noisy image. That means we add Gaussian noise with noise level a 25 to the images in the training set and train the network to predict the original noisy training image from its noisier version. As we have control over the noise that we inject during training, is re-sampled every epoch to improve performance.
Similar to training with the noise2noise loss with fixed network size described in Appendix C.1.2 we use batch size of 1, the Adam optimizer with 0.9 and 3g 0.999 and initial learning rate of 6.4 x 10-4, To be consistent with Appendix C.2.2 we also tried an initial learning of 1.6 x 10 for NV 300k, which performed slightly better.
We apply the same scheme for learning rate decay as described in Appendix C.1.2 supervised validation loss is computed between the noisy training target and the network output. We did not observe overfitting in the noisier2noise setup since the additional noise on the noisier training inputs is re-sampled in every epoch.
where the self-
As in Appendix C.1.2 we picked the best out of up to three training runs from different random initializations to account for the variance between runs for small and medium sized training sets.
21

---

32
PSNR (dB)
31
SwinIR supervised -=
SwinIR noisclnuise o2=
30
U-net supervised =0=
U-net noise2noise 25
102
103
10
109
Training set size
Figure 10: Gaussian image denoising with a SwinIR vs. U-net. While the SwinIR outperforms the U-net in terms of absolute performance, the performance of both models trained a noise2noise manner approaches the performance when trained in supervised manner the number of training images increases.
C.1.4 Neighbor2neighhor self-supervised denoising with a U-net
To obtain the results for neighbor2neighhor Figure 2 we trained the same U-net with 128 channels in its first layer, corresponding to 7.4M parameters as in the experiments for noisier2noise and noise2noise. As before, we set the batch size and use the Adam optimizer with 31 0.9 and B2 = 0.999.
In neighbor2neighbor training the given noisy example is subsampled into two images that are used as training input and target. We follow the approach described in Huang et al. (18) and we refer to their description for details the subsampling, the loss function and the regularization used during training.
As proposed by Huang et al. [18] fix the number of training epochs 100 for all training set sizes and halved the learning rate once in every 20 epochs. However, with the total number epochs fixed the convergence of the network the training size changes might depend the choice of the initial learning rate. Hence, instead fixing the initial learning rate as for noise2noise and noisier2noise in Appendix [C.133] C.I.2] ablate for good choice of initial learning rate in the range of 10-5 to 10-2 for every training set size. Contrary to noise2noise and noisier2noise, for the neighbor2neighbor loss computing self-supervised validation loss based only set did not correlate well with the supervised validation loss. Hence, we assumed access to version of our validation set that contains ground truth images evaluate the results of the learning rate ablation study and to pick the best performing model over the training epochs. The curve in Figure contains only the performance of the run with the best learning rate per training size.
noisy validation
Same as in our noise2noise and noisier2noise experiments we picked the best out of up to three training runs from different random initializations to account for the variance between runs for small and medium sized training sets.
C.1.5 Noise2noise self-supervised denoising with SwinIR
In Section Figure we empirically determined the sample complexity of noise2noise self supervupe and supervised Gaussian denoising with U-net and showed that as the training set size gets large model trained in self-supervised manner approaches the performance of model trained in supervised manner. However, we expect that our results for comparing supervised and self-supervised training schemes translate qualitatively other network architectures.
In this section we confirm this by re-rumning our experiments with the SwinIR 26 architecture, a recent ofteeoofthe-arr model for image denoising based on the Swin Transformer [27]
Figure 10 shows the performance of training SwinIR on the training sets of size N (0.1, 1, 10, 100} thousand image patches with a supervised and noise2noise self-supervised loss with noise level on the training targets o. = 25. As expected, the SwinIR outperforms the U-net (curves are taken from Figure 2 on all training set sizes. However the relative performance gap between the models trained in a supervised and self-supervised way similar and at 100k training examples the gap is reduced to 0.098/0.097dB for the SwinIR/U-net.
22

---

We train the default SwinlR for color image denoising from Liang et al. [26] with 1 11.5M network parameters. The networks are trained with the MSE loss, the Adam optimizer and learning rates (2e- -4-4, 1,-4,,86 - 5, 8e - } for training set sizes N E {0.1,1, 10, ] thousand training examples respectively. Depending on the training set size, the networks trained with supervised/self-oee supersseee loss are trained for (100, 87, 78, 40) (100, 87, 50, 26) epochs, where we need less epochs for self- supervised training due to overfitting similar to what was observed for noise2noise training with the U-net in Figure 2 The initial learning rate is decayed by factor of at 60%, 75% and 90% of the total number of training epochs.
C.2 Real-world camera noise denoising
In this section we provide additional details for the experiments on real-world camera noise denoising with the SID (T) dataset presented in Figure 3
We train the same U-net as described Appendix with 56 channels in the first layer resulting in
1 1.4M network parameters.
Each training set consists of non-overlapping patches of size 128 x 128 drawn randomly from all available patches. From the 160 scene instances in the medium SIDD training set we use all scenes except scenes 0199 and 0200 as the remaining scenes already constitute 125k patches and the largest training set we consider contains only 100k patches.
For sulf-supe training input and target cropped from the two noisy images given per scene in the medium SID dataset. For supervised training the noisy target is replaced with the ground truth image estimated from all noisy images of this scene described in Abdelhamed et al. (.. For the different runs with a fixed training set size displayed in Figure 3 training patches and network initializations are drawn independently between runs.
Training and testing is performed in the raw-RGB space, i.e. on single-channel images where every square of 2 x 2 pixels contains one R,G,G and B value. The order of the 4 values varies over the dataset depending on the Bayer pattern of the various cameras used. Given the Bayer pattern of an image we rearrange the squares of size 2 x 2 pixels to follow the order RGGB. Finally the raw images are packed into 4-channel representation corresponding to their RGGB values before processed by the network.
All networks are trained with the Adam optimizer with an initial learning rate of 0.00032, which halved at fixed set of epochs depending on the training set size.
D Additional material for Section 5 Self-supervised compressive sensing
D.1 Compressive sensing for natural images
For the experiments in Section 5.1 we train the same U-net as described in Appendix C.I but with 3 blocks in the encoder/decoder and with 24 channels in the first layer resulting about 1M network parameters, and unlike in Appendix C the network directly predicts the target measurement y' instead of the residual y y" during training. The input to the network consists of complex-valle valued coarse reconstruction F- F-ly in the image domain, which we normalize to zero-mean and standard deviation one before handed to the U-net. Further, the real and complex part are forwarded to the U-net as two separate input channels. The output of the network is denormalized before transformed to the frequency domain in which the (masked in case of self-supervised training) &g training loss is computed. The test performance is evaluated between the absolute value of the network output and the ground truth image.
The networks are trained using the Adam optimizer 21 with an initial learning rate of 10-3, The networks are evaluated on the validation set in every second epoch and the learning rate is decayed by factor of 10 if there is no improvement for certain number of consecutive validations. This patience parameter gradually decreased from 20 for the smallest to 6 for the largest training set size. The training is terminated after training for few epochs with the minimal learning rate 10-5 Only the best model according to the validation loss is evaluated on the test set, which we then report in Figure The validation/test set consists of 80/300 patches. The batch size is set to 1.
23

---

All experiments were conducted on NVIDIA A40 GPU. We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 500 GPU hours for the experiments in Figure
D.2 Compressive sensing accelerated MRI
The data from the multi-coil fastMRI brain dataset consists of images of different contrasts. In our experiments we use the subset of about 55k AX AXT2-weighted images corresponding to a single contrast. We use 50k those images to design training sets Sy varying size with S C Sy for i < j. 300 for validation and the remaining 4700 for testing. The images for validation and testing are taken from different subjects than for training.
The test performance for given measurement is evaluated between the magnitude of the complex valued network reconstruction and the ground truth image. Note that the estimated sensitivity maps are strictly zero outside the region in which the object that we aim to reconstruct is located. Since we compute the training loss in the k-space after applying the sensitivity maps to the network output, the network output does not receive any supervision in regions, where the sensitivity maps are zero. Hence, order to compute meaningful test scores on the magnitude of the network output we apply binary masking to remove artifacts in the background of the reconstructed images. The estimated sensitivity maps are normalized in a way such that a binary mask E gax=" can be obtained as B = 2==1 SJS,.
D.2.1 Compressive sensing accelerated MRI with U-net
For the results in Figure 6 we train the same U-net as described in Appendix D.I. but with 4 blocks in the encoder/decoder and with 64 channels in the first layer resulting in about 3TM network parameters. We use the same input/output normalization as in Appendix D.I.
The networks are trained with the RMSprop optimizer as it is the default in the fastMRI repository [46] and with an initial learning rate of 10 10-3 The networks are evaluated on the validation set after every epoch and the learning rate is decayed by factor of 10 if no improvement is observed for 10 consecutive epochs. The training is terminated after training 10 epochs with the minimal learning rate of 10-6 The best model according the validation loss is evaluated the test set.
All experiments were conducted on a NVIDIA A40 GPU, We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 600 GPU hours for the experiments presented in Figure 6
D.2.2 Compressive sensing accelerated MRI with a VarNet
In this section provide additional results supervised versus seff-supervised compressive sensing MRI, where we replace the U-net used in Figure with the state-of-the-art end-to-end VarNet (38). As our analysis pertains comparing different training schemes, we expect our results translate qualitatively to other types of network architectures.
As Figure IT shows the VarNet outperforms the U-net significantly. However, similar to the U-net the relative performance gap between models trained in a noise2noise self-supervised way and trained in supervised way is small already for small training set sizes.
For training the VarNets we follow the implementation from the official fastMRI repository [46 where the VarNet consists of a cascade of U U-nett with intermediate data consistency steps and an additional U-net that estimates the sensitivity maps used during the data consistency steps and to compute the input to the first U-net.
We use a VarNet with 6 cascades, where each U-net consists of 4 blocks in the encoder and decoder and 14 channels in the top layer resulting in total of about 9.4M network parameters. The VarNet outputs an estimate Yuuuml of the fully sampled k-space for j 1, 1...., coils. We us the pre- computed sensitivity maps S, estimated with ESPIRIT (39 instead of the sensitivity maps estimated
24

---

0.93
0.90
SSIM
0.87
VarNet supervised μ=1
VarNet self-sup. w=3033
0.84
U-net supervised μ=l
U-net sell-sup. 1=01,33
10a
103
10
Training set size N
Figure 11: Compressive sensing MRI with VarNet vs. U-net. While the VarNet outperforms the U-net absolute numbers, for both types of network architecture the performance gap between supervised and self-supervised trained models vanishes already at small training set sizes.
by the network to compute the final reconstructed image as
Σ S y
j=1
This is analogous to how we compute the reference ground truth images; See Section 5.2
Depending on the training set size N E {50, 100, 500, 1000, 5000, 10000, 50000) the model is trained for Nep (150, 130, 125, 120, 90, 75, 55} epochs. All networks are trained with the RM- Sprop optimizer with an initial learning rate of 10-3, which is decayed by a factor of 10 once for the last and again for the last training epochs.
E Details for computing estimates of the variance of the stochastic gradient
In this section we provide additional details the histograms of the normalized variance of the stochastic gradients, i.e., normalized estimates the MSE [[ VR(O))? after one epoch of training presented in Figures
and 6
We consider the gradients with respect to the network weights 0 after training epoch with the supervised loss. Note that the gradient distribution changes at each epoch.
We compute stochastic gradients (Jass) y') for = I,..., gradient of the risk V (())) is estimated with the empirical gradient of the supervised loss, i.e.,
N with N = 10000 images. The
N
v Ri
ve(Jo(y:),x.)
Finally, the normalized variance of one the i-th stochastic gradient is computed as
V(ss(fo(y:)) vi) V
V v((00]
25


Reviews:
[{"review_id": "m4SIzUzPrX", "reviewer": "Reviewer_RTiS", "summary": "The paper is an study on the sample complexity for image reconstruction in two types of methods, self-supervised and supervised. The authors studies the risk bounds for the case of self-supervised methods. They then evaluate the convergance rates in numerical and empirical experiments for two problems, denoising and compressive sensing. The authors conclude that the convergence rates are similar in the two cases of unsupervised and supervised scenarios. However, the self-supervised approach requires more iterations to reach a similar performance.", "questions": "1) In appendix 1, in the first line, I think the sign of the last term, $e$, should be negative, based on the given definition for $y\\prime$. This results in changing the sign of a few terms in following lines. But the conclusion still holds.\n    \n2) Since there is an assumption that the noise distributions for training and inference to be the same in the case of compressed sensing, is it fair to call it self-supervised?\n    \n3) As the authors also point to in their limitations segment, the experiments are limited to U-net like design for the architectures. However, they mention they do not expect using different designs would change the qualitative results. Could they ellaborate on the intuition behind this expectation?\n    \n- Typo:  \n    - Line 511: missing reference", "limitations": "The title and the claims in the paper point to image reconstruction in general. However, the experiments are only limited to denoising and compressed sensing. The behavior might be very different for some other image reconstruction tasks such as image inpainting or super-resolution. This limitation should be more pronounced in the claims.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1) The authors theoretically study and find specific risk bounds for using the self-supervised method.\n    \n2) The authors study the sample complexity empiracally by considering a range of number of parameters for the network and a range of training set sizes.", "weaknesses": "1) For many of the empirical experiments, the authors only report the best out of multiple runs. Illustrating the mean and some measure of variance among multiple runs gives a more complete picture rather than just the best case. Otherwise, the readers would wonder how reliable it is to do a single run of the self-supervised approach compared to the supervised approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and their positive evaluation of our work. In the following we address the weakness and questions pointed out by the reviewer.\n- **Weakness 1, reporting only the best runs:** For all empirical results on denoising and compressive sensing presented in Figure 2,4,5, the Figures show all conducted runs, not only the best ones. The figures show that as the number of training examples increases the variance of the runs decreases, which is why for the largest training set sizes we only conduct a single run. Instead of drawing the performance curves in Figure 2,4,5 based on the best performing models, we could also use the mean performance, which would shift all curves a little bit downwards but would not affect our findings.\n- **Question 1:** Many thanks for pointing out the typo, we fixed it.\n- **Question 2, is it fair to call the compressive sensing setup self-supervised:** Yes, we believe it is fair to call the accelerated MRI compressive sensing setup self-supervised since we do not have access to fully-sampled data during training. Specifically, during training, we only assume that we have access to 1/3 of all possible measurements, which we then split into a network input with an undersampling factor 1/4 and a corresponding training target containing the remaining measurements plus some overlap. At inference, we assume access to only 1/4 of all possible measurements. Contrary, a supervised method assumes access to fully-sampled data at training time.\n- **Question 3, why we expect the choice of network architecture not to affect our qualitative results:** Our result that the performance of a model trained in a noise2noise self-supervised way approaches the performance of a model trained in a supervised way, is based on how well two different loss functions approximate the risk, and thus the qualitative findings do not depend on the explicit network architecture. This can be also seen from Proposition 1 and 2 in which we formulate the conditions under which this result holds for denoising and compressive sensing and which do not depend on the particular choice of the network $f _ {\\mathbf{\\theta}}$. Motivated by your question we added a reference to the Propositions in the limitations segment to make this point more clear.\n- **Limitations, regarding the generality of our claims:** As pointed out by the reviewer, our results are for a class of self-supervised methods based on constructing unbiased estimates of the gradients of the supervised loss and pertain to denoising and accelerated MRI. There are other problems where such self-supervised losses can be constructed, for example for CT imaging and for some cryo-EM setups, and for those setups the qualitative results from our paper also apply. However, we fully agree that the results and statements in our paper pertain to denoising and accelerated MRI. Also, there are many problems where two independently obtained measurements cannot be easily obtained, and then the class of methods discussed in this paper does not apply. We made both points more clear in the limitations section as suggested. Thanks!\n\nThanks again for your review, and please let us know if you have any other questions.\n"}, {"review_id": "Q0RhfRyBO0", "reviewer": "Reviewer_8mky", "summary": "The work investigates the cost of self-supervised training by characterizing its sample complexity.", "questions": "Please see the weaknesses. I am willing to improve the score if the concerns are addressed well.", "limitations": "The limitations have been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is based on the given theory and carries out corresponding empirical research on self-supervised denoising and accelerated MRI.\n2. The paper shows that a model trained with such self-supervised training is as good as the same model trained in a supervised fashion, but self-supervised training requires more examples than supervised training. \n3. The paper shows that the performance gap between self-supervised and supervised training vanishes as a function of the training examples, at a problem-dependent rate, as predicted by the theory.\n", "weaknesses": "1. The main concern is that the theoretical approach of this paper seems to be similar to [1], just extending from supervised to self-supervised settings. The corresponding contribution of the theoretical approach should be further elucidated.\n2. In the results reported by some previous self-supervised denoising works (e.g., Neighbor2Neighbor), Noise2Noise generally performed the same as supervised methods. But in this work, even with a lot of training data, there is still a gap between the two, what is the reason for this?\n3. The paper is based on the setting of simple Gaussian noise. I would like to ask if the authors have done corresponding research or experiments on real-world RGB noise. Is this work still applicable to real-world situations?\n\n[1] Scaling laws for deep learning based image reconstruction. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. In the following we address the weaknesses in the order as pointed out by the reviewer.\n- **Weakness 1, 'the theoretical approach of this paper seems to be similar to [1]':** We would like to point out that the theoretical approach of this paper is different from that of [1] in that both the setup and the proof technique are substantially different. Our work analyzes the self-supervised noise2noise loss, while [1] considers a standard supervised loss. Also, [1] considers an early-stopped estimator while we do not. The proof technique used in [1] is different from ours. Ours is based on a convergence analysis of the stochastic gradient method, while [1] is based on analyzing the iterates of gradient descent. \n- **Weakness 2, noise2noise performing equivalently to supervised training in other works:** Yes, in some publications a network trained with a noise2noise loss performs as well as a network trained with a supervised loss. However this is misleading as it typically pertains to an unrealistic setup where the noise is resampled.   \nSpecifically, the original noise2noise paper [2] re-samples the noise on the noisy training targets in every training epoch, which requires the original image. Follow-up work like neighbor2neighbor [3] or noisier2noise [4] stuck to this approach. However, this is an unrealistic setup, since if we are given the ground truth images we can and should just train in a supervised manner.   \nIn our paper we consider the more realistic setup where we are given two noisy realizations of an image only. An important contribution of our work is to show for this setup, a gap between supervised and noise2noise training exists, which closes as the number of examples becomes large. \n- **Weakness 3, noise2noise for real-world image denoising beyond Gaussian noise:** \nThat is an excellent point. Noise2noise like training for denoising is applicable beyond Gaussian noise. The condition formulated in Proposition 1 only requires the noise on the training targets to be uncorrelated with the residual.  \nThere are a variety of situations in practice where noise2noise like training is applicable. To demonstrate this point, we conducted additional experiments during the rebuttal period on real-world camera image denoising to empirically determine the performance gap between models trained in a self-supervised noise2noise and a supervised manner as a function of the number of training examples. Our results on denoising the raw images in the Smartphone Image Denoising Dataset (SIDD) [6] show that an initial gap of 2.6dB in PSNR for only 100 training patches reduces to 0.2dB for 100k patches. See the attached pdf for the full results that we'll include in the paper.   \nThere are other real-world settings where our results are applicable, e.g., to real fluorescence microscopy images noise, see the paper [5].\n\nWe hope that our clarifications above address the reviewer's concerns. In particular we hope the reviewer's main concern, i.e., that the theory is similar to [1], has been addressed, and the secondary concern has been addressed with the new simulations on additional real-world-simulations for denoising. If this is the case, we would appreciate it if the reviewer would consider raising their score. We are very happy to discuss further, thanks again for your comments.\n\n[2] Lehtinen et al. Noise2Noise: Learning Image Restoration without Clean Data. ICML 2018.   \n[3] Huang et al. Neighbor2Neighbor: Self-Supervised Denoising From Single Noisy Images. CVPR 2021.  \n[4] Moran et al. Noisier2Noise: Learning to Denoise From Unpaired Noisy Data. CVPR 2020.   \n[5] Zhang et al. A Poisson-Gaussian Denoising Dataset with Real Fluorescene Microscopy Images. CVPR 2019.   \n[6] Abdelhamed et al. A High-Quality Denoising Dataset for Smartphone Cameras. CVPR 2018.   "}, {"review_id": "Avx8myabpR", "reviewer": "Reviewer_X8db", "summary": "The paper presents a theoretical analysis of the sample complexity of the problem of learning a linear denoiser with a self-supervised learning loss, and verifies the bounds on a series of experiments with linear denoising. The paper also presents an empirical study of the gap between self-supervised learning and supervised learning in the context of image reconstruction with neural networks (for denoising and compressive MRI problems).\n ", "questions": "Does the dimension of the signal set play a significant role in Theorem 1? While the dimension d appears in eq. 5, it doesn't seem to impact strongly the final bound, which is somehow surprising.\n\nWhy the paper doesn't analyze learning with a SURE-based loss? This should also be an unbiased estimator of the supervised loss for the denoising case.", "limitations": "The paper discusses the limitations related to the choice of an specific neural network. However, I think it would be good to include a discussion of the limitations of using a linear denoiser analysis to understand the dynamics of learning highly non-linear denoisers.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- A theoretical bound illustrates the role of sample complexity in the setting of self-supervised learning for image denoising. A bound scaling as 1/N where N is the dataset size is presented.\n- The gap between supervised and self-supervised learning is evaluated for various imaging tasks, showing that self-supervised losses which are unbiased estimators of the supervised loss can achieve a performance on par with supervised learning for large sample sizes.\n\n", "weaknesses": "There is little link between the theoretical analysis of linear denoisers and the empirical results on non-linear denoising and reconstruction with deep networks. It is not clear whether the theory developed in the linear case can explain the non-linear setting.\nMoreover, some assumptions in the main theorem seem unrealistic in the context of deep learning, e.g. networks are trained on multiple epochs, not on a single pass as required by the theorem.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the feedback. In the following we address the weaknesses and questions pointed out by the reviewer.\n- **Weakness and limitations; on the connection of theory and practice, and our theory pertaining to a linear estimator:** We think our theory and empirical results for the linear estimator and the empirical results for the neural networks are well connected in that the qualitative behavior found for a linear estimator theoretically is also observed empirically for neural networks. \nIn particular, our theoretical results for denoising predict that the gap between noise2noise and supervised training depends on the noise variance $\\sigma_e^2$ of the training targets and closes as the number of training examples $N$ increases, and that's exactly what we find empirically for neural networks as well.   \n*Regarding the theory pertaining to a linear estimator:* In our theory section, we focus on a linear estimator as for the linear estimator we can make a precise statement, while for a complicated neural network we cannot. However, our result also applies to a more general setup, in particular to any estimator for which the loss function is strongly smooth and has a bounded stochastic gradient as formalized in the following Theorem, which we added along with proof and discussion to the appendix:\n> **Theorem 2.** *Consider the estimate $\\mathbf{\\theta} _ N$ obtained by running the SGM for $N$ iterations on the training set $\\mathcal{D}$ with a decaying stepsize $\\alpha_k = \\frac{1}{c+k}$, where $c$ is a constant. Assume that the loss $h(\\mathbf{\\theta}) = ||f_{\\mathbf{\\theta}}(\\mathbf{y}) - \\mathbf{y}'||_2^2$ for any pair $(\\mathbf{y},\\mathbf{y}')$ is $Q$-strongly smooth and that the stochastic gradient is $(M,B)$-bounded. Then the expected generalization error, where expectation is over the random training set $\\mathcal{D}$ obeys*\n$$\n\\mathbb{E} [R(\\mathbf{\\theta} _ N) ] \\leq R(\\mathbf{\\theta}^*) + \\frac{Q}{2} \\frac{1}{N-2} \\frac{1}{m^2} (2M^2 e_0 + B^2).\n$$\n>\nThe Theorem follows from the definition of strong smoothness and Lemma 1 in Appendix B and its interpretation is equivalent to the interpretation of Theorem 1 for the linear case: We get a rate of $1/N$ and the term associated with maximizing the self-supervised loss becomes larger in the noise variance $\\sigma_e^2$ on the training targets since a larger noise variance requires a larger parameter $B$.\n- **Weakness part 2, regarding the assumption in Theorem 1 to consider a single pass of the stochastic gradient method:** With a single pass over the training set we already get an optimal risk bound (up to constants), and thus there seems little value in analyzing multiple passes. Analyzing a single pass of SGD over a training set is a standard technique in the analysis of SGD and is widely accepted, for two reasons: Often it is sufficient to get optimal bounds up to numerical constants (as in our setup), and moreover performing multiple passes significantly complicates the analysis since then we can't leverage independence of the samples as efficiently. \n- **Question part 1, the role of the signal dimension d in Theorem 1:** Good question, the signal dimension doesn't play a significant role in the theorem, since we scale the signal energy to be one in expectation, irrespectively of the signal dimension. The other energies are also scaled to be independent of the signal and ambient dimension. The signal dimension does, however, play a role in that it determines how much noise is filtered out (i.e., the factor $d \\sigma_z^2/n$), this is the standard factor that we expect from subspace denoising. \n- **Question part 2, why do we not consider a SURE-based loss?** That is an excellent question. The SURE loss gives an unbiased estimate of the loss under additional assumptions, i.e., Gaussianity of the noise. We consider a class of self-supervised  noise2noise-like losses that is significantly more general in that it applies beyond Gaussian denoising to for example real-world camera denoising (see the pdf attached to our global response) and in that it generalizes to compressive sensing as discussed in the paper. We are interested in this more general class as it is much more widely applicable.\n\nWe hope that our clarifications above address the reviewer's concerns and would appreciate it if the reviewer would consider raising their score. We are also happy to discuss further, thanks again for your comments."}]

Author rebuttal:
Dear reviewers,

Attached is a pdf containing experimental results on real-world camera image denoising as discussed in our response to reviewer 8mky (weakness 3), who asked if our results hold for real-world noise beyond the Gaussian setup studied so far in our paper. 
The results demonstrate how the performance of models trained in a noise2noise self-supervised way approaches the performance of models trained in a supervised way as a function of the number of training examples also for real-world noise analogously to our previous results for Gaussian denoising.

We hope that you find the additional material helpful and are happy to discuss further.


Meta-review: The paper presents theoretical and empirical analysis of self-supervised inverse problems for image reconstruction. The three reviewers agreed the paper was above acceptance threshold. The result that supervised and unsupervised denoising performs asymptotically similarly is of general interest to the ML community and the experiments and theory was well developed and should be of broad interest to ML researchers. Therefore, the paper would be a useful contribution to the proceedings.
Decision: Accept (poster)