Scaling Laws For Deep Learning Based Image Reconstruction
Authors: Tobit Klug
Reinhard Heckel
Venue: ICLR 2023 poster (2023)
Keywords: scaling laws
number of training examples
inverse problems
deep learning
denoising
magnetic resonance imaging
super-resolution

Abstract: Deep neural networks trained end-to-end to map a measurement of a (noisy) image to a clean image perform excellent for a variety of linear inverse problems. 
Current methods are only trained on a few hundreds or thousands of images as opposed to the millions of examples deep networks are trained on in other domains. 
In this work, we study whether major performance gains are expected from scaling up the training set size.
We consider image denoising, accelerated magnetic resonance imaging, and super-resolution and empirically determine the reconstruction quality as a function of training set size, while simultaneously scaling the network size.  
For all three tasks we find that an initially steep power-law scaling slows significantly already at moderate training set sizes. 
Interpolating those scaling laws suggests that even training on millions of images would not significantly improve performance. 
To understand the expected behavior, we analytically characterize the performance of a linear estimator learned with early stopped gradient descent. 
The result formalizes the intuition that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.

Published as a conference paper at ICLR 2023
SCALING LAWS FOR DEEP LEARNING BASED IMAGE RECONSTRUCTION
Tobit Klug & Reinhard Heckel
Dept. of Computer Engineering Technical University of Munich Munich, Bavaria, Germany (tobit klug, reinhard. ceeeaarrdeeckkleee heckeleetun. @tum.
ABSTRACT
Deep neural networks trained end-to-end to map a measurement of a (noisy) image to a clean image perform excellent for variety of linear inverse problems. Current methods are only trained on a few hundreds or thousands of images opposed to the millions of examples deep networks are trained in other domains. this work, we study whether major performance gains are expected from scaling up the training set size. We consider image denoising, accelerated magnetic resonance imaging, and super-resolution and empirically determine the reconstruction quality a function of training scaling the network size. For all three tasks we find that an initially steep power- law scaling slows significantly already at moderate training set sizes. Interpolating those scaling laws suggests that even training on millions of images would not significantly improve performance. To understand the expected behavior, we analytically characterize the performance of a linear estimator learned with early stopped gradient descent. The result formalizes the intuition that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.
size, while simultaneously
INTRODUCTION
Deep neural networks trained to map noisy measurement of image or noisy image to clean image give state-of-the-art (SOTA) performance for image reconstruction problems. Examples are image denoising (Burger al., 2012 Zhang et al. 2017 Brooks et al. 2019 Liang al. (022))
super-resolution (Dong et al. 2016) Ledig et al. 2017 Liang et al. 2022]) and compressive sensing for computed tomography (CT) (Jin al. 2017 7). and accelerated magnetic resonance imaging (MRI)
(Zbontar et al. 2018 Sriram et al. 2020 Muckley et al. 2021 Fabian & Soltanolkotabi 2022).
The performance of neural network for imaging is determined by the network architecture and optimization, the size of the network, and the size and quality of the training set.
Significant work has been invested in architecture development. For example, in the field of acceler- ated MRI, networks started out as convolutional neural networks (CNN) (Wang et al. 2016), which are now often used as building blocks un-rolled variational networks (Hammernik et al. 2018
Sriram et al.) 2020). Most recently. transformers have been adapted to image reconstruction (Lin &
Heckel 2022 Huang et al. 2022 Fabian Soltanolkotabij 2022).
However, it is not clear how substantial the latest improvements through architecture design are compared to potential improvements expected by scaling the training set and network size. Contrary to natural language processing (NLP) models and modern image classifiers that are trained billions of examples, networks for image reconstruction are only trained on hundreds to thousands of example images. For example, the training set of SwinIR (Liang al. (021). the current SOTA for image denoising, contains only 10k images, and the popular benchmark dataset for accelerated MRI consists only of 35k images (Zbontar al. 2018).
In this work, we study whether neural networks for image reconstruction only require moderate amounts of data to reach their peak performance, or whether major boosts expected from increasing

---

Published as a conference paper at ICLR 2023
32.4
(a)
32.4
(b)
PSNR (dB)
32
32
31.8
31.8
30, 9
31.4
71. 0018
31.4
31.2
31.2
10a
10
10
10
109
109
107
(c)
0.95
(d)
0.94
0.94
SSIM
0.93
0.93
0.89.%90000
0.922.000002 000020
0.91
0.91
102
10
10
10 Network parameters P
10a
Training set size
Figure 1: Empirical scaling laws for CNN-based image reconstruction. Reconstruction perfor- mance of U-Net for denoising (a) and accelerated MRI (c) as function the training set size N. In both experiments an initial steep power law R 3N transitions relatively flat one already at moderate N. Thus we expect that training on millions of images does not significantly improve performance. To obtain the scaling curves (c).C) we optimize parameters as shown (d). Colors the plots on the left and right correspond to the same training size. Since for large training sizes corresponding to the flattened scaling law, increasing the parameters does not boost performance further, the decay in scaling coefficients is a robust finding.
the number of network
the training set size. To partially address this question, we focus on three problems: image denoising, reconstruction from few and noisy measurements (compressive sensing) in the context of accelerated MRI, and super-resolution. We pick Gaussian denoising for its practical importance and since it can serve as a building block solve more general image reconstruction problems well (Venkatakrishhann al.) 2013). We pick MR reconstruction because is important instance of a compressive sensing problem, and many problems can be formulated as compressive sensing problems, for example super-resolution and in- painting. In addition, for MR reconstruction the question how much data needed is particularly important, since is expensive to collect medical data.
For the three problems we identify scaling laws that describe the reconstruction quality function of the training set size, while simultaneously scaling network sizes. Such scaling laws have been established for NL. and classification tasks, discussed below, but not for image reconstruction.
The experiments are conducted with U-Net (Ronneberger et al. 2013) and the SOTA SwinIR (Liang al. (002), transformer architecture. We primarily consider the U-Net since it widely used for image reconstruction and acts as a building block SOTA models for image denoising (Brooks) al. [202I] Zhang et al. [221 2022) and accelerated MRI (Zbontar et al. 2018 Sriram et al. (2220). We also present results for denoising with the SwinIR. The SwinIR
al. 2019:
outperforms the U-Net, but we find that its scaling with the number of training examples does not
differ notably. Our contributions are as follows:
Empirical scaling laws for denoising. We train U-Nets of sizes O.IM 46.5M parameters with training set sizes from 100 to 100k images from the ImageNet dataset (Russakovsky ct al. 2015) for Gaussian denoising with 20. 1..17dB Pakk-Sinnllto-Noiit ratio (PSNR). While for the largest training set sizes and network sizes we consider, performance continues to increase, the rate of performance increase for training set sizes beyond few thousand images slows to a level indicating that even training on millions of images only yields marginal benefit,
Fig. a
We also train SOTA SwinlRs of sizes 4M to 129M parameters on the same range of training set sizes, see Fig. aa). Albeit it performs better than the U-Net. its scaling behavior is essentially equivalent, and again after few thousand images, only marginal benefits
2

---

Published as a conference paper at ICLR 2023
expected. Scaling up its training set and network size did, however, give benefits, the largest model we trained yields new SOTA results on four common test sets by 0.05 to 0.22dB.
Empirical scaling laws for compressive sensing. We train U-Nets of sizes from 2M to 500M parameters for 4x accelerated MRI with training set sizes from 50 to 50k images from the fastMRI dataset (Knoll et al., 2020b). We again find that beyond dataset size of about 2.5k, the rate of improvement as a function of the training set size considerably slows, see Fig. c)). This indicates that while models are expected to improve by increasing the training set size, we expect that training on millions of images does not significantly improves performance.
Empirical scaling laws for super-resolution. We also briefly study super-resolution, and similarly as for denoising and compressive sensing find a slowing of the scaling law at moderate dataset sizes. See appendix
Understanding scaling laws for denoising theoretically. Our empirical results indicate that the denoising performance of neural network trained end-to-end doesn't 't increase as function of training examples beyond a certain point. This is expected since once we reach a noise specific error floor, more data is not beneficial. We make this intuition precise for linear estimator learned with early stopped gradient descent to denoise data drawn from a d- d-dimensional linear subspace. We show that the reconstruction error is upper bounded by d/N plus noise- dereedeeenddnt error floor level. Once the error induced by learning the signal model, d/N, is small relative to the error floor, more training examples N are not beneficial.
Together, our empirical results show that neural networks for denoising, compressive sensing, and super-resolution applied in typical setups (i.e, Gaussian denoising with 20.17dB PSNR, and multi-coil accelerated MRI with 4x acceleration) already operate in regime where the scaling laws for the training set size are slowing significantly and thus even very large increases of the training data are not expected to improve performance substantially. Even relatively modest model improvements such as those obtained by transformers over convolutional networks are larger than what we expect from scaling the number of training examples from tens of thousands millions.
2 RELATED WORK
Scaling laws for prediction problems. Under the umbrella of statistical learning theory conver- gence rates of 1 I/N or 1/VN have been established for a range of relatively simple models and distributions, see e.g. Wainwright (2019) for an overview.
For deep neural networks used in practice, a recent line of work has empirically characterized the performance function of training set size and/or network size for classification and NLP (Hestness) et al. 2017 Rosenfeld et al.) [2019 Kaplan et al., 2020 Bahri et al.) [021] Zhai et al. 2021 Ghorbani al. 2021 Bansal al.) 2022). In those domains, the scaling laws persist even for very large datasets, as described in more detail below. In contrast, for image reconstruction, we find that the
power-law behavior already slows considerably at relatively small numbers of training examples.
Rosenfeld et al. (2019) find power-law scaling of performance with training set and network size across models and datasets for language modeling and classification. However, because the work fixes either training set or network size, while scaling the other, the scaling laws span only moderate ranges before saturating at level determined by the fixed quantity and not the problem specific error floor. The papers Kaplan et al. (2020); Bahri et al. (202I): : Zhai et al. (2021), >); Ghorbani et al. (2021); Bansal et al. (0022) including ours scale the dataset size and model size simultaneously, resulting in an improved predictive power of the obtained scaling laws.
Hestness et al. (017)) study models for language and image classification, and attribute deviations the hyperparameters of very large networks. For transformer language models Kaplan et al. (2020) find no deviation from a plwer-law for up to training set and network size of 1B images and parameters. Further, Zhai et al. (2021) find the performance for Vision Transformers (Dosovitskiy et al. (020) for few-shot image classification to deviate from a power-law curve only at extreme model sizes of 0.3-IB parameters and 3B images.
from power-law curve lack of
The role of training set size in inverse problems. For image reconstruction and inverse problems in general, are not aware of work studying scaling laws in principled manner, covering different
3

---

Published as a conference paper at ICLR 2023
(a)
(b)
32.6
32.6
PSNR (dB)
32.4
32.4
32.2
32.2 2
32
31 1.88 90000
32
31.8
1.007
31.8
31.6
CNN bused U-Net
31.6
10a
10aa
10
10
107
10
Training set size N
Network parameters P
Figure 2: Empirical scaling laws for transformer-based image denoising. The colored curve in (a) shows the best PSNR per training size of SwinIR (Liang et al. 2021) with varying network sizes as depicted in (b). Colors in the plot on the left and right correspond to the same training set size. The SwinlR outperforms the U-Net (gray curve from Fig. [a)), but as for the U-Net the rate of improvement slows to a level that indicates that further increasing the dataset size would only marginally improve performance.
problems and model architectures. But, when proposing a new method several works study perfor- mance as function of training set and/or network size. However, those studies typically only scale the parameter of interest while fixing all other parameters, unlike our work, which scales dataset size and network size together, which is important for identifying scaling laws. Below, we review the training set sizes, and if available their scaling properties, used by the recent SOTA in image denoising and accelerated MRI.
Zhang et al. (2017) report that for DrCNN, a standard CNN, using more than 400 distinct images with data augmentation only yields negligible improvements. Chen al. (202 pre-train an image processing transformer (IPT) of 115.5M network parameters on ImageNet (1. M distinct images), and report the performance after fine-tuning to a specific task as a function of the size of the pre- training dataset. IPT's performance for denoising is surpassed by the latest SOTA in form of the
CNN based DRUnet (Chen et al. 2021), 1), the transformer based SwinIR (Liang et al. (02T) and Swin-Conv-Une ÜZang et al.) 2022) combination of the two. Those models have significantly fewer network parameters and were trained on a training set consisting of only 10kk images, leaving the role of training set size in image denoising open.
The number of available training images for accelerated MRI limited to few publicly available datasets. Hence, current SOTA (Sriram al., 22200] Fabian & Soltanolkotabi (022) are trained on the largest datasets available (the fastMRI dataset), consisting of 35k images for knee and 70k for
brain reconstruction (Zbontar al. 2018 Muckley al. 2021).
Adaptations of the MLP-Mixer Tolstikhin et al. 2021 and the Vision Transformer (Dosovitskiy al. 2020) MRI (Mansour et al. 2022 Lin & Heckel 2022) ablate their performance function of the training set size and find that non-CNN based approaches require more data to achieve a similar performance, but potentially also benefit more from even larger datasets. However, those experiments are run with fixed model sizes and only 3-4 realizations of training set size thus it is unclear when performance saturates.
3 EMPIRICAL SCALING LAWS FOR DENOISING
In this section, consider the problem of estimating image from noisy observation y where is additive Gaussian noise z ~ (0, I² Neural networks trained end-to-end to map noisy observation clean image perform best, and outperform classical approaches like BM3D (Daboy et al. 2007) and non-local means (Buades et al. 2005) that are not based on training data.
To enable studying learned denoisers over wide range of training set sizes we work with the ImageNet dataset (Russakovsky et al. 2015). Its training set contains 1.3M images of 1000 different classes. We reserve 20 random classes for validation and testing. We design 10 training set subsets SNy of sizes N E [100, 100000] (see Fig. aa) and with S, C S, for < j. To make the distributions of images between the subsets homogeneous possible, the number of images from
Datasets.
4

---

Published as a conference paper at ICLR 2023
different classes within subset differ by most one. We generate noisy images by adding Gaussian noise with variance of dg = 25 (pixels in the range from 0 to 255), i... 20. in PSNR.
Model variants and training. We train a CNN-based U-Net, a detailed description of the model architecture is in Appx. A.I We also train the Swin-Transformer (Liu al. 2021) based SwinIR model (Liang et al. 2021) y that achieves SOTA for a variety of image reconstruction tasks, including denoising. This model interesting since transformers scale well with the number of training
examples for other domains, c.g., for image classification pre-training (Dosovitskiy et al. 2020).
For U-Net and SwinlR we vary the number of network parameters as depicted in Fig. (bb) and Fig. (b) respectively. Appx.[.] and A.3 contain detailed descriptions on how the models are trained and the network size adapted.
Results and discussion. Fig. a) and Fig. a show the reconstruction performances of the best U-Net and SwinlR respectively over all considered network sizes. Our main findings are follows.
training set size of around 100 already sufficient to train a decent image denoiser. Beyond 1 100, we can fit linear power law to the performance of the U-Net with scaling coefficient 0.0048 that approximately holds up to training set sizes of about 6k images. Beyond 6k, we can fit a second linear power law with significantly smaller scaling coefficient
0.0019.
Similarly, for the SwinIR we can fit a power law with coefficient a = 0.0050 in the regime of little training data and power law with significantly smaller coefficient moderate training data, thus the two architectures scale essentially equivalently.
0.0017 in the regime of
While denoising benefits from more training examples, the drop in scaling coefficient indicates that scaling from thousands to millions of images is only expected to yield relatively marginal performance gains. While the gains are small and we expect gains from further scaling to be even smaller, our largest SwinIR trained on 100k images yields new SOTA for Gaussian color image denoising. On four common test sets it achieves an improvement between 0.05 to 0.22dB, see Appx. B. Visualizations of reconstructions from models along the curves in Fig. T and Fig. 2(a can be found in Fig. Appx. A
Comparing the U-Net scaling to SwinIR scaling (Fig. 2(a)) we see that the effect of large training sets on the performance of the U-Net can not make up for the improved modeling error that stems from the superior network architecture of the SwinIR. In fact, the simulated scaling coefficient a 0.0019 predicts a required training set size of about 150M images for U-Net to achieve the best performance of the SwinlR, and that is optimistic prediction since it assumes that the scaling does not slow down further for another 3 orders of magnitude. Thus, model improvements such as those obtained by transformers over convolutional networks are larger than what expect from scaling the number of training examples from tens of thousands to millions.
An interesting question is whether different noise levels lead to different data requirements, as indicated by different scaling behavior. We investigate the performance of U-Net for a smaller noise variance æ₂ = 15 in Appx. D.2 While the results are still preliminary, the qualitative scaling behavior is similar; the difference are that the scaling law pertaining to smaller noise is slightly steeper, and overall the performance improves by about 1.3dB in PSNR.
Finally, note that for our results we choose to re-sample the noise per training image in every training epoch, since makes the best use of the available clean images. However, fixing the noise is also interesting, since it is more similar to a setup in which the noise statistics are unknown like in real-world noise removal. See Appx. [. for additional results where the noise fixed. We found that compared to re- re-sampling the noise the performance of a U-Net drops by 0.3dB for small training set sizes. As the training set size increases the performance approaches the one of re-sampling the noise resulting in steeper scaling law that flattens at slightly larger training set sizes.
Robustness of our finding of slowing scaling laws. Our main finding for denoising is that the scaling laws slow at relatively small amounts of training data points (i.e., the scaling coefficient becomes very small). We next argue why we think that this is robust finding.
In principle it could be that the networks are not sufficiently large for the performance to improve further as the dataset size increases. However, for the U-Net and training set sizes of 10k and 30k the
5

---

Published as a conference paper at ICLR 2023
performance increase already slows, and for those training set sizes, increasing the network size does not improve performance, as shown in Fig. [b). For smaller network sizes the curves in Fig. [b)
first increase and then decrease indicating that we found network sizes close the optimum.
The parameter scaling of the SwinIR in Fig. b) indicates that for training set sizes of 10k and 100k the performance still benefits from larger networks. Hence, the power law for those training set sizes in Fig. 2a) can become slightly steeper once we further increase the network size. However, for the slope of the flat power law in (a) reach the slope of the steep power law, the parameter scaling in (b) for 100k training images would need to hold for another 2 orders of magnitude, i.e. about 12B instead of the currently used 129M network parameters, which is very unlikely considering that the current network sizes already suffice to saturate the performance for small/moderate training set sizes.
Finally, it could be that with higher quality or higher diversity of the data, the denoising performance would further improve. In our experiments, we increase the training set sizes by adding more images from the same classes of ImageNet, and we continue to test on different classes. Thus, could be that adding training examples from a different, more diverse data source leads to less of slowing of the scaling law in Fig. a We test this hypothesis by scaling our training set with 3k images to 10k images not by adding more images from ImageNet but images from the datasets that are used to train the original SwinIR, i.c. we add all images from DIV2K (Agustsson & Timofte 2017). Flickr2K (Timofte al.]|2211 (217)) and BSD500 (Arbeláez et al. 2011 and add the remaining images from WED (Ma et al. 2017). Keeping all other hyperparameters fixed the U-Net obtains for this dataset PSNR of 32.239dB, which is slightly worse than the 32.252dB it achieves with our training set with 10k images only from ImageNet. Hence, we reject the hypothesis that the drop in scaling coefficients can be explained with how the training sets are designed in our experiment.
EMPIRICAL SCALING LAWS FOR COMPRESSIVE SENSING
We consider compressive sensing (CS) to achieve 4x accelerated MRI. MRI is an important medical imaging technique for its non-invasiveness and high accuracy. Due to the physics of the measurement process, MRI is inherently slow. Accelerated MRI aims at significantly reducing the scan time by undersampling the measurements. addition, MRI scanners rely on parallel imaging, which uses multiple receiver coils to simultancously collect different measurements of the same object.
This leads to the following reconstruction problem. We are given measurements y; E C of the image E C y; MFS,x + noise; for i = 1,.., C, and our goal is to reconstruct the image from those measurements. Here, C' is the number of receiver coils, the diagonal matrices S, E C the sensitivity maps modelling the signal strength perceived by the i-th coil, F C is the discrete Fourier transform (DFT), and M E Cm is a binary mask implementing the undersampling.
Classical CS approaches (Lustig al. 2008) first estimate the sensitivity maps with method such as ESPIRIT (Uecker et | (0144), and second estimate the unknown image by solving regularized optimization problem, such as total-variation norm minimization. Recently, deep learning based methods have been shown to significantly outperform classical CS methods due to their ability to learn more complex and accurate signal models from data (Knoll et al. 2020a Muckley et al. 202T).
Datasets. To explore the performance of learning based MR reconstruction over a wide range of training set sizes, we use the fastMRI multi-coil brain dataset (Knoll et al., 2020b), which is the largest publicly available dataset for MRI. The dataset consists of images of different contrasts. To ensure that the statistics of the dataset are as homogencous as possible, we take the subset of the dataset corresponding
single contrast resulting in 50k training images. We design training set subsets SN of size N E [50, 50000] (see Fig. [ (c)) with S; C S, for i S j. For more information on selection, division and subsampling of the dataset see Appx. A.2
Model variants and training. We train the same U-Net model used in Section 3 and described in Appx. A. but with 4 blocks per encoder/decoder. The network is trained end-to-end to map a coarse reconstruction ÅCR to the ground truth image x. The coarse reconstruction is obtained as 1/2
*CR = F=
,where F-1 is the inverse DFT, and yı.FF are the undersampled measurements of the i-th coil, where missing entries are filled with zeros. We vary the number of

---

Published as a conference paper at ICLR 2023
network parameters depicted in Fig. | (d). Appx. model is trained and the network size is adapted.
A.2 contains detailed descriptions how the
Results and discussion. For each training set size Fig. (c) shows the reconstruction performance in structural similarity (SSIM) of the best model over all simulated network sizes. There are 2 main findings. First, we can fit a linear power law with a scaling coefficient a = 0.0072 that holds up to training set sizes of about 2.5k images. Second, for training sizes starting from 5k can fit second linear power law but with significantly smaller scaling coefficient (x 0.0026.
Similar to denoising in Section we conclude that while accelerated MRI slightly benefits form more training examples, the drop in scaling coefficient indicates that it is unlikely that even training set sizes of the order of hundreds of millions of images result in substantial gains. Visualizations of reconstructions from models along the curve in Fig. (c) can be found in Fig. Appx. A.2
Robustness of our finding of slowing scaling laws. Next, we argue why the drop in scaling coefficient for accelerated MRI a robust finding.
In Fig. (d) we demonstrate that the network size does not bottleneck the performance of our largest training sets. For each training set size the performance as function of the number of network parameters is relatively flat. Even small training sets benefit from large networks before their performance slightly decays for very large networks. Hence, we expect that for large training sets the performance as function of network parameters would not increase significantly before decaying.
Finally, is there different type of training data that would improve the scaling coefficient? We don't think so, since all experiments are for the very specific task of reconstructing brain images of one particular contrast, which means that the examples that we add to increase the size of the training sets is already the data most suitable to solve this task.
UNDERSTANDING SCALING LAWS FOR DENOISING THEORETICALLY
We study simple linear denoiser that is trained end-to-end reconstruct a clean image from a noisy observation, in order to understand how the error as a function of the number of training examples for inverse problems is expected to look like.
We define a joint distribution over a signal and the corresponding measurement (x, y) as follows.
Consider a d < n dimensional subspace of R" parameterized by the orthonormal basis E Raxd We draw signal approximately uniformly from the subspace x Uc, where ~ (O, I), and an associated noisy measurement as y = x + z. where z ~ (O, o²I) ) is Gaussian noise. The subspace is unknown, but are given a training set {(x1, ..... (XN, yN)), consisting of examples drawn iid from the joint distribution over (x.y).
We assume that the signal lies in low dimensional subspace. Assuming that data lies in low- dimensional subspace or more general, a union of low-dimensional subspaces, is common, for example it underlies the denoising of natural images via wavelet thresholding (Donoho & Johnstone 1995. Simoncelli & Adelson 1996 Chang et al. 2000). Mohan et al. (2020) found that even deep learning based denoisers implicitly perform a projection onto an ceaativell--seectee low-dimensional subspace that captures the features of natural images.
We consider linear estimator of the form fw (y) = Wy, and measure performance in terms of the expected mean-squared reconstruction error, defined as R(W) JE |W - x|2] where expectation is over the joint distribution of y).
The optimal linear estimator. The optimal linear estimator (i.e., the estimator that minimizes the risk R) is given by W - U The estimator projects the data onto the subspace and shrinks
I+07
towards zero, depending the noise variance. The associated risk is R(W*) = = / +z)
An estimator based subspace estimation. The optimal estimator W' requires knowledge of the unknown subspace. We can learn the subspace from the noisy data by performing principal component analysis on Y [y1,..., yN]. Specifically, we estimate the subspace as the d-leading
7

---

Published as a conference paper at ICLR 2023
singular vectors of the empirical co-variance matrix YY? E R denoted by by Ü Raxd, If the measurement noise is zero (i.e., o² 0), Ü is an orthonormal basis for the d-dimensional subspace U provided we observe at least many linearly independent signals, which occurs with probability one if we draw data according to the model defined above, and if the number of training examples obeys N > d. There is a vast literature on PCA; see Vershynin [201]] 2018); Rudelson & Vershynin (0010)
Tropp 2012) for tools from nigh-dimensional probability to analyze the PCA estimate.
Now consider the estimator WPCA = 1+o2 Ú y based on N noisy training points. The estimator assumes knowledge of the noise variance, but is not difficult to estimate it relatively accurately. The following result, proven in Appx. [ characterizes the associated risk.
Theorem 1. Suppose that the number of training examples obeys (d + no?)) log(n) S N. For numerical constant with probability at least 1 - -nn -1 3e- + t the risk of the PCA-estimate is bounded by
PCPa) R(W") + c(d + no?) log(n)//n
Thus, as long as the number of training examples N is sufficiently large relative to (d + no?), the risk of the PCA-mstimator is close to the risk of the optimal estimator.
Estimator learned end-to-end. We now consider an estimator learned end-to-end, by applying gradient descent to the empirical risk L(W) Σ |[Wy xill² and we regularize via early- stopping the gradient descent iterations. The risk of the estimate after & iterations of gradient descent, W*, is bounded by the next result. This estimator mimics the supervised training we consider throughout this section, with the difference that here we consider simple neural network, and in the previous sections we trained neural network.
Theorem 2. Let W be the matrix obtained by applying k iterations of gradient descent with stepsize y starting W O the loss L(W). Consider the regime where the number of training examples obeys (d + no²2) log(n) I N S Ed/o², for an arbitrary & and N log(N) S n. For an appropriate choice of the stepsize 1) there exists an optimal earty stopping time Kopt at which the risk of the estimator fw(y) = Wk is upper-bounded with probability at least
1 -N/8 -N188 5n-9 5e 5e- 26- -N 2e 20/2 by
R( R(W kopt) S (8 + 24) + (1+ no?
(d + no?) log(n) + V
(d+ + o²n) logn
log n
N
N
where e is a numerical constant.
The proof is provided in Appx. C Similar to Thm. Thhe first term in the bound corresponds to the noise-dependent error floor and the second two terms decrease in (d + no²)/N and represent the error induced by learning the estimator with a finite number of training examples. Once this error is small relative to the error floor, more training examples do not improve performance. See Appx. for a more detailed discussion on the estimator and the assumptions of Thm.
Discussion. In Fig. 3 we plot the risk as function of the training size for the early-stopped empirical risk minimization (ERM) and the PCA based estimator. Starting at training set size of N 100, we see that the risk minus the optimal risk follows approximately a power law, i.c.. log( (g(R(W)) - R)(W)) log( (N). as suggested by Thms. [ [andd
In practice, however, we don't know the risk of the optimal estimator, R(W*). Therefore, in the second row of Fig. and throughout our empirical results we plot the risk log R (W)) as a function of the number of training examples log(N) and distinguish different regions by fitting approximated scaling coefficients to log( R W)) R -a log(N)) N). In our theoretical example, we can identify three regions, coined in Hestness et al. (0117) The small data region in which the training set size does not suffice to learn a well-performing mapping and the power-law region in which the performance decays approximately as Na with a < O, and an mrolemm-peccife irreducible error region, which is R(W*) here. Note that the power-law coefficient in the risk plot R(W) are smaller then those when plotting R(W)- - R(W*) an are only approximate scaling coefficients.
Already in this highly simplified setup of an inverse problem the true scaling coefficients heavily depend on model parameters such as the noise variance as can be seen in Fig. 3 The scaling coefficients further vary with the signal dimension d and ambient dimension (see Appx. E.1
8

---

Published as a conference paper at ICLR 2023
early stopped ERM
PCA
10
R(W) - R(W")
10 -0
=0
0 1 1.0.0=0.00
10
1000,=02
10
-00.-0.1
1.0,0=00005
100
R(W)
10
- 0.8, 0.2
-0-00.0 0.7, 0.2
10"4
+aa-11m 1.3-111,000 1.500=001
0.9. 0000.1
= 2.4 0.005
1.2, -0.05
10
10
10a
10a
10
100
10
102
10a
10a
Training set size N
Training set size N
Figure 3: Subspace denoising. Simulated risks of the early stopped ERM estimator Whys (left) and the PCA estimator WICa (right) over the training set size N. The signal and ambient dimensions are = 10, n = 1000 and we vary the noise level σz. We linear scaling laws ~ Na to the power law regions. As the noise level decreases the scaling coefficients increase. Further, the learned estimator exhibits steeper scaling than the PCA estimator. Error bars are over independent runs.
CONCLUSION
We found that the performance improvement of deep learning based image reconstruction function the number of training examples slows already at moderate training set sizes, indicating that only marginal gains are expected beyond few thousand examples.
Limitations. This finding is based on studying three different reconstruction problems (denoising, super-resolution, and compressive sensing), two architectures (U-net and SwinlR), and for each setup we extensively optimized hyperparameters and carefully scaled the networks. Scaling gave new state-of-the-art results four denoising datasets, which provides some confidence in the setup.
Moreover, our statements necessarily pertain to the architectures and metrics we study. is possible that other architectures another scaling of architectures yield larger improvements when scaling architectures. It is also widely acknowledged that image quality metrics (such as SSIM and PSNR) do fully capture the perceived image quality by humans, and could be that more training data yields improvements in image quality that are not captured well by SSIM and PSNR
Most importantly, our findings pertain standard in-distribution evaluation. i.c., the test images are from the same distribution as the training images. To achieve robustness against testing on images out of the training distribution recent work indicates a positive effect larger and more diverse training sets (Miller et al., 2021 Darestani et al. 2022. Nguyen et al.) 2022). Thus it is possible that the distribuiotiii performance of image reconstruction methods improves when trained on larger and more diverse dataset, even though the in-distribution performance does not improve further.
Future research. We focus on supervised image reconstruction methods. For image denoising and other image reconstruction problems self-supervised approaches are also performing very well (Laine) et al. 2019 Wang et al. 2022 Zhou et al. 2022). even though supervised methods perform best if clean images available. It would be very interesting to investigate the scaling laws for self- supervised methods; perhaps more training examples are required to achieve the performance of a supervised setup. We leave this for future work.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY
The repository
https:/ggituub.com//LL- MaweLLI-----/Seee Lab/Scaling..... Laws_ For Deep_ Learning Based__mage__aaaaaaaaaaaaaaaaeee mmase__Reconstructiiieo contains the code to reproduce all results in the main body of this paper.
ACKNOWLEDGMENTS
The authors acknowledge support by the Institute of Advanced Studies at the Technical University of Munich, the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) 4556455471, 464123524 the DAAD, and the German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts. The authors of this work take full responsibility for its content.
REFERENCES
Eirikur Agustsson and Radu Timofte. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1122-1131, 2017.
Pablo Arbeláez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour Detection and Hier- archical Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):898 916, 2011.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining Neural Scaling Laws. D.Xiv:2102.06700 [cond-mat, stat], 2021.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data Scaling Laws NMT: The Effect of Noise and Architecture. 22XX::2202201994 [cs], 2022.
Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T. Barron. Unprocessing Images for Learned Raw Denoising. IEEE Conference on Computer Vision and Pattern Recognition, pp. 11036- 1005, 2019.
A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pp. 60-65, 2005.
Harold C. Burger, Christian Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with BM3D? IEEE Conference Computer Vision and Pattern Recognition, pp. 2392-2399, 2012.
Tony Cai, Zongming Ma, and Yihong Wu. Optimal estimation and rank detection for sparse spiked covariance matrices. Probability Theory and Related Fields, 161(3):71-815, 2015.
S.G. Chang, Bin Yu, and M. Vetterli. Adaptive wavelet thresholding for image denoising and compression. IEEE Transactions on Image Processing, 9()): -546, 2000.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, et al. Pre-Trained Image Processing Transformer. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 12299-12311 2021.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image Denoising by Sparse 3-D Transform- Domaii Collaborative Filtering. IEEE Transactions on Image Processing, (6(8))2080-2095 2007.
Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing. In Proceedings of the 39th International Conference on Machine Learning, pp. 4754-4776. PMLR, 2022.
Chandler Davis and W. M. Kahan. The Rotation of Eigenvectors by a Perturbation. III. SIAM Journal Numerical Analysis, 701):1-66, 1970.
10

---

Published as a conference paper at ICLR 2023
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning Deep Convolutional Network for Image Super- Reser-Resolution. In European Conference on Computer Vision, pp. 184-199, 2014. ISBN 978-3-319-10595
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image Super-Resolution Using Deep Convolutional Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307, 2016.
David L. Donoho and lain M. Johnstone. Adapting to Unknown Smoothness via Wavelet Shrinkage. Journal of the American Statistical Association, 90(432):1200-13 1224,, 1995.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. arXiv:2010. 1929 [cs], 2020.
Zalan Fabian and Mahdi Soltanolkotabi. HUMUS-Net: Hybrid unrolled multi-scale network archi- tecture for accelerated MRI reconstruction. arXiv:2203.08213 [cs, eess), 2022.
Rich Franzen. Kodak Lossless True Color Image Suite. http://fOkkusggraahhcc/kkokkk 1999.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling Laws for Neural Machine Translation. arXiv:2109.077400 [cs]. 2021.
Shuhang Gu, Nong Sang, and Fan Ma. Fast image super resolution via local regression. In Proceedings of the 21 International Conference on Pattern Recognition (ICPR2012), pp. 3128 3131,, 2012.
Javier Gurrola-Ramos, Oscar Dalmau, and Teresa E. Alarcón. A Residual Dense U-Net Neural Network for Image Denoising. IEEE Access, 131772-317544 2021.
Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael Recht, Daniel Sodickson, Thomas Pock. and Florian Knoll. Learnnng a variational network for reconstruction of accelerated MRI data. Magnetic Resonance in Medicine, 79(6):305533071 2018.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kian- inejad, Md Mostofa Ali Patwary, et al. Deep Learning Scaling is Predictable, Empirically, 11X1v:1712.00409 [cs, stat], 2017.
Bia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197-5206. 2015.
Jiahao Huang, Yingying Fang, Yinzhe Wu, Huanjun Wu, Zhifan Gao, Yang Li, Javier Del Ser, et al. Swin Transformer for Fast MRI. arXiv:2201.03230 [cs, eess), 2022.
Kyong Hwan Jin, Michael T. McCann, Emmanuel Froustey, and Michael Unser. Deep Convolutional Neural Network for Inverse Problems in Imaging. IEEE Transactions Image Processing, 26(9): 4509-4522, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, et al. Scaling Laws for Neural Language Models. arXiv:2001.0836/ [cs, stat], 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International
Conference on Learning Representations, 2014.
Florian Knoll, Tullie Murrell, Anuroop Sriram, Nafissa Yakubova, Jure Zbontar, Michael Rabbat, Aaron Defazio, et al. Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge. Magnetic Resonance in Medicine, 84(6): 3054-3070, 2020a.
Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew J. Muckley, Mary Bruno, Aaron Defazio, Marc Parente, et al. fastMRI: A Publicly Available Raw k-Space and DICOM Dataset of Knee Images for Accelerated MR Image Reconstruction Using Machine Learning. Radiology: Artificial Intelligence, 2(1):e190007 2020b.
11

---

Published as a conference paper at ICLR 2023
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. Tugh-Quulity Self Supervise Deep Image Denoising. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. IEEE Conference on Computer Vision and Pattern Recognition, pp. 103-114, 2017.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. SwinIR: Image Restoration Using Swin Transformer. In IEEE International Conference on Computer Vision Workshops, pp. 1833-1844, 2021.
Kang Lin and Reinhard Heckel. Vision Transformers Enable Fast and Robust Accelerated MRI. In Medical Imaging with Deep Learning, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In IEEE International Conference on Computer Vision, pp. 10012-10022, 2021.
Michael Lustig, David L. Donoho, Juan M. Santos, and John M. Pauly. Compressed Sensing MRI. IEEE Signal Processing Magazine, 25(2):72-82, 2008
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang. Hongwei Yong. Hongliang Li, and Lei Zhang. Waterloo Exploration Database: New Challenges for Image Quality Assessment Models. IEEE Transactions on Image Processing, 2602):004-1106, 2017.
Youssef Mansour, Kang Lin, and Reinhard Heckel. Image-to-Image MLP-mixer for Image Recon- struction. arXiv:2202.0221 02018 [cs, eess], 2022.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV, volume 2, pp. 416-423 vol.2, 2001.
Tomer Michaeli and Michal Irani. Nonparametric Blind Super-resolution. In Proceedings of the IEEE International Conference Computer Vision, pp. 945-952, 2013.
John P. Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, et al. Accuracy on the Line: On the Strong Correlation Between Out-of- Diitri-Diitiitttinn and In-Distribution Generalization. In Proceedings of the 38th International Conference on Machine Learning, pp. 7721-7735 PMLR, 2021.
Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Fernandez-Granda. Robust And Interpretable Blind Image Denoising Via Bias-Free Convolutional Neural Networks. In International Conference Learning Representations, 2020.
Matthew J. Muckley, Bruno Riemenschneider, Alireza Radmanesh, Sunwoo Kim, Geunu Jeong, Jingyu Ko, Yohan Jun, et al. Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction. IEEE Transactions on Medical Imaging, 4009):2306-2317, 2021.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP, 2022.
Roberto Imbuzeiro Oliveira. Concentration of the adjacency matrix and of the Laplacian in random graphs with independent edges. arXiv:0911.0600 [math], 2010.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention. 2015.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A Constructive Pre- diction of the Generalization Error Across Scales. In International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2023
Mark Rudelson and Roman Vershynin. Non-asymptotic Theory of Random Matrices: Extreme Singular Values. In Proceedings of the International Congress of Mathematicians, pp. 1576-1602. 2010. ISBN 978-981-4324-33
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3)211 2015.
E.P. Simoncelli and E.H. Adelson. Noise removal via Bayesian wavelet coring. In Proceedings of 3rd IEEE International Conference on Image Processing, volume 1, pp. 379-382 vol.1, 1996.
Amuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C. Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-End Variational Networks for Accelerated MRI Reconstruction. In Medical Image Computing and Computer Assisted Intervention, pp. 64-73, 2020. ISBN 978-3-030-59713-9,
Radu Timofte, Vincent De, and Luc Van Gool. Anchored Neighborhood Regression for Fast Example- Example-Based Super-Resolution. In 2013 IEEE International Conference on Computer Vision, pp. 1920-1927, 2013.
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, al. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results. In IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 1110-1121, 2017.
Iyy Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, et al. MLP-Mixer: An all-MLP Architecture for Vision. arXiv:2105.0166 [cs], 2021.
Joel A. Tropp User-Friendly Tail Bounds for Sums of Random Matrices. Foundations of Computa- tional Mathematics, 12(4):389 -344,,444444 2012.
Joel A. Tropp. An Introduction to Matrix Concentration Inequalities. Foundations and Trends in Machine Learning, 8(1-2):1-230, 2015.
Martin Uecker, Peng Lai, Mark J. Murphy, Patrick Virtue, Michael Elad, John M. Pauly, Shreyas S. Vasanawala, and Michael Lustig. ESPIRiT an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA. Magnetic Resonance in Medicine, 7103);990-1001, 2014.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv:1607. 1607.0802 [cs]. 2017.
Singanallur V. Venkatakrishnan, Charles A. Bouman, and Brendt Wohlberg. Plug-and-Pla priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 945-948, 2013.
Roman Vershynin. Introduction the non-asymptotic analysis of random matrices. arXiv:1011.3027 [cs, math), 2011.
Roman Vershynin. Digh-DDimesional Probability: An Introduction with Applications in Data Science. Cambridge Series Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018. ISBN 978- 118-1-900-41519944
Martin J. Wainwright. High- Vigh--Diensional Statisticy: Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2019, ISBN 978-1-108-49802-9.
Shanshan Wang, Zhenghang Su, Leslie Ying, Xi Peng, Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang. Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 514-517, 2016.
Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. Blind2Unblind: Self-Supervised Image Denoising With Visible Blind Spots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2027-2036, 2022.
13

---

Published as a conference paper at ICLR 2023
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J. Muck- ley, Aaron Defazio, et al. fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. arXfv:1811.08839 [physics, stat], 2018.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transformers. arXiv:2106.04560 [cs], 2021.
K. Zhang, W. Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Transactions on Image Processing. 2017.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and- Play Image Restoration with Deep Denoiser Prior. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 2021.
Kai Zhang, Yawei Li. Jingyun Liang, Jiezhang Cao, Yulun Zhang, Hao Tang, Radu Timofte, and Luc Van Gool. Practical Blind Denoising via Swin-Conv-UNe and Data Synthesis. arXiv:2203.13278 [cs, eess], 2022.
Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging ELECTRON IMAGING, 20, 2011.
Bo Zhou, Jo Schlemper, Neel Dey, Seyed Sadegh Mohseni Salehi, Chi Liu, James S. Duncan, and Michal Sofka. DSFormer: A Dual-domain Self-supervised Transformer for Accelerated Multi-contrast MRI Reconstruction. arXiv:2201.10770 [cs, eess), 2022.
14

---

Published as a conference paper at ICLR 2023
A DETAILS OF THE EXPERIMENTAL SETUPS
A.1 EXPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR DENOISING WITH A U-NET
In this Section, we give a detailed description of the experimental setup that led to our results for and Section
Gaussian denoising with U-Net presented in
In addition, Fig. shows examples of reconstructions from different models along the performance curve in Fig. (a). In these examples the improvement in perceived image quality from increasing the training set size from 100 to 1000 is larger than from increasing from 1000 to 10000 or from 10000 to 100000. This correlates with our quantitative findings in Fig. (a).
Next, we describe the experimental details. We train U-Nets with two blocks in the encoder and decoder part respectively and skip connections between blocks. Each block consists of two convolu- tional layers with LeakyReLU activation and instance normalization (Ulyanov et al. 2017) after every layer, where the number of channels is doubled (halved) after every block in the encoder (decoder). The downsampling in the encoder is implemented as average pooling and the upsampling in the decoder transposed convolutions. As proposed by Zhang al. (2017) we train residual denoiser that learns to predict x instead of directly predicting x, which improves performance.
We scale the network size by increasing the width of the network by scaling the number of channels per (transposed) convolutional layer. For denoising we trained U-Nets of 7 different sizes. We vary the number of channels in the first layer in {16, 32, 64, 128, 192, 256, 320), which corresponds to {0.1, 0.5, 7.4, 16.7, 30.0, 36.5) million parameters. We do not vary the depth, since this would change the dimension of the informational bottleneck in the U-Net and thus change the model family itself.
The exact training set sizes we consider are
1,3, 6, 10, 30, 60, 100) thousand images from ImageNet. We center crop the training images to 256 x 256 pixels. We also tried a smaller patch size of 128 x 1 128 pixels, but larger patches showed to have a better performance in the regime of large training set sizes, which is why the results presented in the main body are for patch size 256 x 256. For a comparison between the scaling behavior of the two different patch sizes see Fig. 9 Appx. D.3
We do not use any data augmentation, since it is unclear how to account for it in the number of training examples. For validation and testing we use 80 and 300 images respectively taken from 20 random classes that are not used for training.
We use -uaree-uuard loss and Adam (Kingma & Ba 2014) optimizer with 81 0.9, B2 0.999. For moderate training set sizes up to 3000 images we find that heuristically adjusting the initial learning rate with the help of an automated learning rate annealing performs well. To this end, we start with a learning rate of 10-4 and increase after every epoch by a factor of until the validation loss does not improve for 3 consecutive epochs. We then load the model checkpoint from the learning rate that was still performing well and continue with that learning rate. We observe that this scheme typically picks an initial learning rate reduced by factor 2 for every increase in the number of channels C. For training set sizes larger than 3000 we directly apply this rule to pick an initial learning rate without annealing as we found this to give slightly better results. In particular, for number of channels C {128, 192, 256, 320} we used initial learning rates 1) {0.0032, 0.0016, 0.0008, 0.00043.
For small training sets up to 1000 images we found that batch size of works best. For larger training sets we use batch size of 10 and found that further increasing the batch size does not improve performance.
We do not put limit on the amount of compute for training. We use automated learning rate decay that reduces the learning rate by 0.5 if the validation PSNR has not improved by at least 0.001 for 10 epochs 6 epochs for training set sizes starting from 6000 images. Once the learning rate drops to 10 5 we observe near to no gains in validation loss and stop the training after 10 additional epochs.
For training set sizes up 1000 we train 3 random seeds and pick the best run. As the variance between runs compared to the gain in performance between different training set sizes decreases with increasing training set size only run one seed for larger training set sizes.
15

---

Published as a conference paper at ICLR 2023
The experiments were conducted four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA Quadro RTX 6000 GPUs We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 1800 GPU hours for the experiments in Fig. [a).(b).
EXPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR COMPRESSIVE SENSING A.2 WITH A U-NET
In this Section, we give a detailed description of the experimental setup that led to our results for compressive sensing MRI presented in Fig (c) and Section 4
In addition, Fig. shows examples of reconstructions from different models along the performance curve in Fig. (c). In these examples the improvement in perceived image quality from increasing the training set size from 500 to 2500 is larger than from increasing from 2500 to 10000 or from 10000 to 50000. This correlates with our quantitative findings in Fig. (c).
The first in row in Fig. [ shows an example which all models including the one trained on the largest training set fail to recover fine detail. This is known problem in accelerated MRI and has been documented in both editions of the fast MRI challenge (Knoll et al.] [020aa] MMucley et al. 2021 in which for all methods examples could be found in which fine details have not been recovered. However, the question remains if more training data or better models would help or the details are simply not there since the information is lost due to the large undersampling factors considered in the fastMRI challenges and in this work.
Next, describe the experimental details. For compressed sensing in the context of accelerated MRI we trained U-Nets of 14 different sizes. We vary the number of channels in the first layer in {16, 32, 48, 64, 96, 112, 128, 144, 160, 176, 192, 208, 224, 256), which corresponds to {2, 8, 18, 31, 70, 95, 124, 157, 193, 234, 279, 327, 380, 496] million network parameters.
The exact training set sizes we consider are {0.05, 0.25, 0.5, 2.5, 5, 10, 25, 50} thousand AXT2 weighted images from fastMRI multi-coil brain dataset (Zbontar et al., 2018), where AXT2 corre- sponds all images of one type of contrast. We focused on images only from this type to make the statisties of our datasets as homogeneous as possible. We do not use any data augmentation, since it unclear how to account for in the number of training examples. We use 4732 and 730 additional images for testing and validation.
We consider an acceleration factor of meaning that we only measure 25% of the information. We obtain the 4 times undersampled measurements by masking the fully sampled measurement with an equispaced mask with 8% center fractions meaning that in the center of y we take all the measurements and take the remaining ones at equispaced intervals.
We use structural similarity (SSIM) loss and RMSprop optimizer with a = 0.99 as this is the default in the fastMRI repository (Zbontar et al. 2018) and we found no improvement by replacing it with Adam. We do not put limit on the amount of compute invested into training. We deploy an automated learning rate decay that starts at learning rate of 10-3 and decays by a factor 0.1 if the validation SSIM has not improved by at least 10-4 for 5 epochs. Once the learning rate drops to 10 we stop the training after 10 additional epochs. Only for the largest training set sizes 25k,50k we found that an additional drop to 10 resulted in further performance gains. We use a batch size of 1.
For training set sizes up to 5k we train three models with random seeds and pick the best. For larger training set sizes we only run one seed, since the variance between runs decreased.
The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA Quadro RTX 6000 GPUs We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 4250 GPU hours for the experiments in Fig. (c).d))
A.3 EXPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR DENOISING WITH THE SWINIR
In this Section, we give a detailed description of the experimental setup that led to our results for Gaussian denoising with SwinIR presented in Fig 2 (b) and Section 3
In addition, Fig. shows examples of reconstructions from different models along the performance curve in Fig. (a). Despite of the best SwinIR clearly outperforming the best U-Net in terms of PSNR
16

---

Published as a conference paper at ICLR 2023
Clean/Noisy
100
1000
10000
100000
U-Ner
PSNR:
30.20
30.92
31.27
31.42
SwiniR
PSNR:
31.07
31.48
31.77
31.88
U-Net
PSNR: DC
28.37
28.83
29.07
29.17
SwinT
PSNR:
28.91
29.15
29.30
29.36
Figure 4: Reconstructions along the scaling law for denoising with U-Net and SwinIR. The two examples illustrate how the reconstruction quality improves as the training set size N increases. First rows: ground truth and reconstruction, Second rows: residuals w.r.t. the ground truth.
17

---

Published as a conference paper at ICLR 2023
Ground Truth
N
500
N 2500
N 10000
N 50000
SSIM:
0.824
0.8869
0.8936
0.8945
SSIM:
0.919
0.9268
0.9305
0.935
Figure 5: Reconstructions along the scaling law for compressive sensing MRI with a U-Net. The two examples illustrate how the reconstruction quality improves as the training set size N increases. For each example the first row shows the ground truth and the reconstruction with the respective quantitative performance in SSIM and the second row shows the residual of each reconstruction with respect to the ground truth.
18

---

Published as a conference paper at ICLR 2023
Table I: Batch size and maximal number of steps for every experiment in Fig. 2 Each exper- iment can be identified by the number of training examples N in thousands and the network size small/S//idddee//l/eie///gggggg
N/P
0.1/S
0.1/M
0.1/L
0.3/S
0.3/M
0.3/L
1.0/S
Batch size # epochs (-109)
8
20
8
20
8
11.52
11.52
11.52
5.76
5.76
5.76
3.84
N/P
1.0/M
1.0/L
10/M
10/L
100/M
100/L
100/H
Batch size # epochs (-10a)
20
20
8
20
4
4.32
3.855
1.232
1.28
0.128
0.128
0.128
Table 2: List of used network configurations of the SwinIR Liang al. (2021) We consider four network sizes Maaas///mmmdeeeeM/a/ae////////
by varying the number of residual Swin
Transformer blocks, the number of Swin Transformers per block, the number of attention heads Swin Transformer, the number of channels in the input embedding and the width of the fully connected layers in a Swin Transformer.
Size
blocks
# transformers
heads
# channels
MLP width
learning rate
S
5
120
240
2 10 4
M L
6
6
180
360
2 10-4
8
6
240
720
1.10-1
H
11
8
360
720
50-5
it is difficult for the naked eye notice large differences in the quality of the reconstructions. This indicates that for Gaussian denoising both models already operate in regime, where improvements to be made are only marginal.
Next, we describe the experimental details. To obtain Fig. 2 we train the SwinIR for color denoising from Liang et al. 2021) on the same training sets mentioned in Appx. A.I with {0.1, 0.3, 1, 10, 100) thousand images from ImageNet. Instead of center cropping the training images to 256 x 256 we have crop 128 x 128 pixels larger input patches would make it computational infeasible for us to train large versions of the SwinIR. The largest SwinIR alone took over 2 months to train on NVIDIA A40 GPUs.
We train 4 different network sizes with (3.7, 11.5, 41.8, 128.9} million parameters. We denote the four network sizes Amall(S//idddeeMM//ageee////M
The training details and network configurations are as follows. The default SwinIR for denois- ing (Liang al. 2021 proposed for training set size of about 10k images, 11.5M network parameters and was trained with a batch size of 8 for T' 1220 epochs, where the Icarning rate is halved at [0.57. 0.7. 0.875I, 0.93751"] epochs. We keep the learning rate schedule but adjust the maximal number of epochs T according to the training set size. Table shows batch size and maximal number of epochs for every experiment in Fig. 2 We did not optimize over the choice of the batch size but picked the batch size as prescribed by the availability of computational resources.
See Liang al.) (202T) for detailed description of the SwinIR network architecture. We vary the network size by adjusting the number of residual Swin Transformer blocks, the number of Swin Transformers per block, the number of attention heads per Swin Transformer, the number of channels in the imput embedding and the width of the fully connected layers in a Swin Transformer. Table contains a summary of the settings. When scaling up the network size, we invested in the parameters that seemed to be most promising in the ablation studies in Liang al. (202T).
The experiments were conducted on four NVIDIA A40 and four NVIDIA RTX A6000. We used about 13000 GPU hours for the tranformer experiments in Fig. 2 For training the models in parallel on multiple GPUs we utilize the torch. di st ibut ed package with the glow backend, instead hardware that time.
distribut
of the faster neel backend, which was unfortunately not available
19

---

Published as a conference paper at ICLR 2023
Table Benchmarking results for Gaussian color image denoisng. Average PSNR on common test sets of our best U-Net (46.5M parameters, 100k training images) and three different versions of the SwinIR (see Appx. A.3). Values for the original SwinIR and SCUnet are taken from Liang et al. (2021) and ZndZhang et al. (2022). Best and second best performance are in red and blue colors respectively.
Noise level
SwinIR original
SCUnet original
SwinIR 10/M
SwinIR 10/L
SwinIR 100/H
Dataset
U-Net
CBSD68
25
31.57
31.78
31.79
31.72
31.78
31.84
Kodak24
25
32.78
32.89
32.92
32.97
33.05
33.14
McMaster
25
33.01
33.20
33.34
33.20
33.32
33,44
Urban (0)
25
32.17
32.90
33.03
32.63
32,90
33.21
B BENCHMARKING OUR MODELS FOR GAUSSIAN IMAGE DENOISING
In this Section, evaluate the models trained for image denoising in Section on four common test sets form the literature and show that the largest SwinIR trained on the largest dataset achieves new SOTA for all four test sets and the considered noise level.
In Fig. in the main body we compared the performance of the U-Nets and the SwinIRs trained on subsets of ImageNet for image denoising. The models are evaluated on a test set sampled from ImageNet. We observed that while SwinlRs significantly outperform U-Nets, the performance gain from increasing the training set size slows already at moderate training set sizes for both architectures equally. However, there is still moderat performance gain in scaling the models, and thus we expect the largest SwinlR trained on the largest dataset to outperform the original SwinIR from Liang et al. (2021) Our results in this section show that this is indeed the case.
In Table we evaluate on the standard test sets for Gaussian color image denoising CBSD68 (Mrrtin let al. 2001), Kodak24 (Franzen 1999), McMaster (Zhang et al.) (0011) and 1rban100 (Huang 2015). We observe a significant performance difference between the best U-Net (46.5M parameters, 100k training images) and the other transformer based methods. As expected, our largest SwinIR trained on the largest dataset SwinIR 100/H outperforms the original SwinIR, but also the
iet al.
SCUnet (Zhang al., 2022 later SOTA model that has been demonstrated to outperform the original SwinIR.
We also depict the gains for the SwinIR from just scaling up the network size and then from scaling up network size and training set size. While this led to a new SOTA for Gaussian image denoising, note that on the downside training the SwinIR 10/M, which comparable to the original SwinIR, took about 2 weeks on 4 NVIDIA A40 gpus, while training the SwinIR 100/H took over 2 months.
C EMPIRICAL SCALING LAWS FOR IMAGE SUPER-RESOLUTION WITH A U-NET
In this Section, we consider the problem of super-resolution, i.e., estimating an high-resolution image from low-resolution version of the image. This can be viewed as compressive sensing problem, since can view the super-resolution problem reconstructing a signal x E R² from downsampled version Ax, where the matrix A mm implements a downsampling operation like bicubic downsampling, or blurring followed by downsampling. As first shown in the pioneering work of Dong et al. (2014) data driven neural networks trained end-to-end outperform classical
model-based approaches (Gu et al. 2012 Michaeli & Trani 2013 Timofte al. 2013).
Dong et al. (014) reports that for super-resolution with a simple three-layer CNN, the gains from very large training sets do not seem to be as impressive as in high-level vision problems like image classification. Liang al. (202I) 1) plot the super-resolution performance of the SOTA SwinIR model as a function of the training set size up to 3600 training examples for a fixed network size. When plotting their results logarithmic scale, observe that the performance improvement follows llw. is unclear, however, whether this power law slows beyond this relatively small number of images.
20

---

Published as a conference paper at ICLR 2023
In this Section, obtain scaling laws for super-resolution for U-Net wide range of training set and network sizes, similar as for denoising and compressive sensing in the main body.
Datasets. We use the same training, validation and test sets as described in Appx. A.1 with training set sizes N [100, 100k] images from ImageNet. On top we add two larger training sets of size 300k and 600k. Instead of center cropping the training images to 256 x 256 we follow the super-resolution experiments in Liang et al. (2021) and train on images cropped to 128 x 128 pixels. We consider super-resolution of Tactor 2 so the low-resolution images have size 64 64. The low-resolution images are obtained with the bicubic downsampling function of Python's 's PIL. Image package.
Model variants and training. We train the same U-Net model used in Section 3 and described in Appx. A.I but with only one block per encoder/decoder as this resulted in slightly better results than with two blocks. The network is trained end-to and-to-end to map coarse reconstruction, obtained through bicubic upsampling, to the residual between the coarse reconstruction and the high-resolution ground truth image.
We vary the number the channels the first layer (8, 32, 64, 128, 192, 256, 320, 448, 564), which corresponds to {0.007, 0.025, 0.10, 0.40, 1.6, 3.6, 6.4, 10.0, 20.0, 31.2] million network pa- rameters. We do not use any data augmentation, since it is unclear how to account for in the number of training examples.
We use the losss and Adam optimizer with its default settings. For all experiments we find good initial learning rate with the same annealing strategy deseribed in Appx. However, instead of picking the largest learning rate for which the validation loss does not diverge, we pick the second largest, which leads slightly more stable results. We start the annealing with learning rate of 10-5 In the few cases, where our heuristic leads to a degenerated training curve, typically due to picking a significantly too small or too large learning rate, starting the annealing with smaller learning rate of 10- resolves the problem.
We do not put limit on the amount of compute invested into training. To this end, we deploy an automated learning rate decay that reduces the learning rate by 0.5 if the validation PSNR has not improved by at least 0.001 for 8 epochs. We stop the training once the validation loss did not improve for two consecutive learning rates. For training sets up 10000 images we found that batch size of 1 works best. For larger training sets we use a batch size of 10.
For training set sizes up to 10000 we train 3 random seeds and pick the best run. As the variance between runs compared the gain performance between different training set sizes decreases with increasing training set size only run one seed for larger training set sizes.
The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA Quadro RTX 6000 GPUs. We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 1500 GPU hours for the experiments in Fig.
Results and discussion. For each training set size, Fig. 6 a) shows the reconstruction performance in PSNR of the best model over all simulated network sizes in Fig. bb... Since the curves per training set size in Fig. b)) are relatively flat, further scaling up the network size is not expected to significantly improve the performance on the studied training sets. Here are the two main findings:
A linear power law with scaling coefficient a 0.0075 holds roughly up to training set sizes of about 30k images, and for training set sizes starting from 60k this slows to linear power law with significantly smaller scaling coefficient of 1.6B images would be required to increase performance by another 1dB (assuming the relation 32.05.V0.000 persists, which is likely to slow down even further).
0.0029. With this slowed scaling law training set size
While slowing already at few tens of thousands of training images, the scaling laws for super- resolution do not slow early as those for denoising and compressive sensing (see Fig D.. This could be partially due training on image patches of size 128 x 128 as opposed to size 256 x 256 used for denoising.
21

---

Published as a conference paper at ICLR 2023
(b)
PSNR (dB)
32.8
32.6
32.6
32.4
32.4
32.2
32.2
32
32
31.8
30.55
31.8
31.6
0.005
32.05
31.6
31.4
31.4
102
10a
10a
10
104
10
10
107
Training set size
Network parameters P
Figure 6: Empirical scaling laws for super-resolution with U-Net. The curve in (a) contains the best reconstruction performances per training set size over the different network sizes depicted (b). Colors in the plot on the left and right correspond to the same training size. While the scaling laws for super-resolution do not slow as early as those for Denoising and Compressive sensing (see F they are likely slow further at larger number of training examples
D ADDITIONAL EMPIRICAL SCALING LAWS FOR GAUSSIAN DENOISING
In this Section, we extend our results for Gaussian denoising with a U-Net from Section 3 with two additional setups. Section D.I considers fixing the noise sampled in each training epoch and Section D.2 investigates reducing the noise level from 0₂ = 25 to a= = 15.
D.1 EMPIRICAL SCALING LAWS FOR DENOISING WITH FIXED NOISE
Our main results for denoising with a U-Net trained end-to-end discussed in Section 3 follow a setup in which the noise per training example is re- remsad in every training epoch. We choose this setup since makes the best use of the available clean images. However, fixing the noise is also interesting since it is closer to a denoising setup in which the noise statistics are unknown, which is the case in noise removal problems. such a problem, we would be given pairs of noisy and clean image, and could not synthesis new noisy images from the clean images.
some
In this section, we follow the same experimental setup from Appx. AA.I simulate the performance of U-Net for denoising with fixed noise for up 100k training images (see Fig. 7 Compared to re-sampling the noise, we observe a drop in performance of about 0.3dB for small training set sizes. The performance difference at 10k images is reduced to 0.2dB resulting in slightly steeper scaling law for moderate training set sizes. However, at around 10k training images the scaling of the performance of training with fixed noise also starts flatten it approaches the performance of re-sampling the noise during training. This indicates that if the noise statistics are unknown, more data is required achieve the same performance when they are known. However, in both cases the scaling with training set size slows already down at moderate training set sizes.
D.2 EMPIRICAL SCALING LAWS FOR DENOISING WITH A SMALLER NOISE LEVEL
Our results for Gaussian denoising in Section are for a fixed noise level of a; 25. In this Section, with smaller
we repeat the experiments for Gaussian denoising with U-Net described in Appx. with noise level of a= 15, in order to see how the scaling laws change. The results for both noise levels are depicted in Fig. 8
We observe an improvement about 2.3dB in PSNR, which is expected since the irreducible error decreases for smaller noise levels. We also observe that the scaling coefficient for the smaller noise level 0 = 15 (i.e., a = 0.0026) is slightly steeper than that for the larger noise level a = 25 (i.e., 0.0019). This coincides with the qualitative behavior of the curves for subspace denoising in Figure 3 Apart from that the curves are qualitatively similar, in that initially steep power law is replaced by slower one around 6000 training images.
22

---

Published as a conference paper at ICLR 2023
(b)
32.2
32.2
PSNR (dB)
32
32
31.8
31.8
31.2
Fix: 30.47.
31.2
Re sample: 00.91.10000.
31
31
102
10aa
10
10
10a
10
107
Training set size
Network parameters P
Figure 7: Empirical scaling laws for denoising with fixed noise. The colored curve in (a) contains the best reconstruction performances per training set size over the different network sizes depicted in (b) for fixed noise realizations during training. Colors in the plot on the left and right correspond to the same training set size. The gray curve is taken from Fig. a) and shows the performance, when the noise is re-sampled during training. The initial drop in performance due to fixing the noise during training reduces as the training set size increase resulting in a slightly steeper scaling compared to re-sampling the noise. Yet, we expect also the scaling of the performance training with fixed noise to flatten as it approaches the performance of re-sampling the noise.
34.8
(a)
34.8
(b)
34.6
PSNR (dB)
34.4
34.4
34.2
34.2
333 06.36.7 0.000
33.8
33.8
33. 33.78.0 33..88.8..00 0y
33.6
102
10a
104
10
10
10
107
105
32.4
32.4
(C)
(d)
PSNR (dB)
322
32
31.6
31.6
30.91 900000000000
31.7.0000030
31.2
31.2
10a
10a
10
10
105
10a
107
10a
Training set size N
Network parameters P
Figure 8: Comparison of empirical scaling laws for denoising with noise level 15 and 25. The colored curves in (a) and (c) contain the best reconstruction performances per training set size over different network sizes depicted (b) and (d) for noise level 15 and 25 respectively. Colors plot the left and right correspond to the same training set size. The curves in (c),d) are taken from Fig. (b)).
23

---

Published as a conference paper at ICLR 2023
32.4
32.4
(a)
32.2
32.2
PSNR (dB)
31.8
31.8
31.6
31.6
31.4
31.4
0.006
10a N Number of training patches of size 128 x 128
102
10a
107
Network parameters P
Figure 9: Empirical scaling laws for denoising with a smaller patch size. The colored curve in (a) contains the best reconstruction performances per training set size over the different network sizes depicted in (b) for training patches size 128 x 128. Colors the plot on the left and right correspond to the same training set size. The gray curve is taken from Fig. a and shows the performance for training patches of size 256 x 256. Note that the x-axis shows the number training patches of size 128 128. Hence, patch of size 256 x 256 is worth patches of size 128 128. We see that in the regime large training set sizes larger patches performs better than more but smaller patches.
D.3 EMPIRICAL SCALING LAWS FOR DENOISING WITH SMALLER PATCH SIZE
Our results for Gaussian denoising in Section 3 with U-Net were obtained for constant training patch size of 256 x 256 pixels across all network and training set sizes. In this Section, we repeat the experiments for Gaussian denoising with U-Net described in Appx. A.T with smaller patch size of 128 X 128 pixels.
The results for both patch sizes depicted in Fig. We observe that in the regime of large training set sizes, that we are primarily interested in, training on N patches of size 256 x 256 is more beneficial than training 4.V patches of size 128 128. We therefore focus on patch size 256 256 in the main body of this work.
E UNDERSTANDING SCALING LAWS FOR DENOISING THEORETICALLY SUPPLEMENTARY RESULTS
In this section, we provide additional details on the statements in Section 5 understanding scaling laws for denoising theoretically by studying a linear subspace denoising problem theoretically, and provide additional numerical results.
Recall that we consider linear estimator the form fw(y) = Wy, and measure performance in terms of the expected mean-squared reconstruction error (normalized by the latent signal dimension d):
R(W)
E [w x|[] d d (W - IJU + a 2
d
Above, expectation over the joint distribution of (x,y), and the second equality follows from using that x = Uc, where e (O, is Gaussian, and = x + z, where the noise z ~ N'0 oTI is Gaussian.
The optimal linear estimator. The optimal linear estimator (i.e.. the estimator that minimizes the risk defined in equation (I) is given by W T4az UU", This follows from taking the gradient of the risk (I). setting it to zero, and solving for W. The estimator projects the data onto the subspace and shrinks towards zero, depending on the noise variance. The associated risk is R(W*)= = o2/(1 (1 + o2).
24

---

Published as a conference paper at ICLR 2023
Early-stopped empirical risk minimization. descent to the empirical risk
We consider the estimator that applies gradient
((W) Σ Wy; x112 IWY X
(2)
where X, Y E RaxN contain the training examples as columns, and early-stops after iterations for
regularization.
We next discuss the early-stopped estimator W more detail. Fig. 10 Onmmerically demonstrates the
regularizing effect of early stopping gradient descent, where w = -XY is the converged learned estimator (see Appx. Eq. (5). We see that regularization is necessary for this estimator to perform
well.
We next discuss Theorem and the associated assumptions in more detail. Theorem considers the following regime: (i) the number of training examples obeys (d + no?) log(n) N and (ii) N < Ed/o", for an arbitrary &, and (iii) N (N) < n. For this regime, the theorem guarantees that the risk of the optimally early-stopped estimator obeys, with high probability,
R kopt) ) S (8 + 2g) + <(1 no² d
log n (d + no?) log(n)
+ + / (d+ (d++2n)) o2n) logn
N
N
The theorem looks similar to that for the PCA estimate (Theorem D in that the risk is constant away from the optimal risk, with an error term that becomes small as (d + / becomes small. However, the error bound does not converge to R(W") the number of training examples, N, converges to infinity. This is probably an artifact of our analysis, but it is unclear, at least to us, how derive a substantially tighter bound. In our analysis (see appendix G), we balance two errors: One error decreases in l: and is associated with the part of the signal projected into the subspace, and the second error increases with and is associated with the orthogonal complement of the subspace. We choose the carly-stopping time to optimally balance those two terms, which yields the stated bound (3).
Now with regards to the assumption: Assumption (iii) N (og(NN)= S means we are in the high- dimensional regime; we think this somewhat closer to reality (for example for denoising 512 x 512 image, this would require the number of training examples to be smaller than 250k), but we can derive an analogous bound for the regime N log(N) 2 n,, where the number of training examples larger than the ambient dimension.
log(n) S N relatively mild, as it necessary to being able to somewhat accurately estimate the subspace; this assumption is also required for the PCA estimate.
Assumption (i) (d + no]))og(()
Assumption (ii) S Eddoo", for an arbitrary ç, is not restrictive in that & can be arbitrarily large, we make this assumption only so that the theorem can be stated in convenient way. However, assumption (ii) reveals a shortcoming of Theorem 2 which that we cannot make the bound go to zero as N oc, since increasing & increases one term in the bound, and decreases another one.
E. ADDITIONAL NUMERICAL SIMULATIONS
In this Section we provide further numerical simulations for the PCA subspace estimator and the esti- mator learned with early stopped gradient descent discussed in Section 5 Theorem [I and Theorem Similar to Fig. 3 Fig. 11 shows the risks R( R(Wkkp))) and R( ((WP))) as a function of the number of training examples N for varying values of the signal and ambient dimension d and n, while fixing all other model parameters.
the power law region we fit linear power laws with negative scaling coefficients a. We observe steeper power laws (larger |a|) for smaller ambient dimensions and larger signal dimensions d. Also the scaling coefficients of the learned estimator consistently excel the coefficients from the PCA estimator.
F PROOF FOR THEOREM I RISK BOUND FOR PCA SUBSPACE ESTIMATION
We provide bound on the risk of the estimator f(y) = WPCAY with WICA = ~ŮUT and Itog. Recall that U goxd contains the singular vectors corresponding the d-leading
25

---

Published as a conference paper at ICLR 2023
100
W W W
R(W)
10
10 10 10a
10a
10a
Training set size
Figure 10: Effect of early stopping the learned estimator. The risk of the early stopped learned estimator Whose and converged estimator W as function of the training set size N measured in simulations. While both estimators approach the optimal performance (R(")) for large N, early stopping is critical for performance in the regime N R n. We consider the setup d 10, n 100, o. = 0.05. Error bars are over 5 independent runs.
early stopped ERM
PCA
109
qpe 1- 1000
1.2, n= 0000
1000
m= 10000
n= 1000
Ope 2.4,
100
R(W)
10
-2
100
d=100
1.5, d=100
a=-2 d-50
1.2.=-00
1.5, d=10
1.300,9d=10 2-9,
R(
10
10
102 10a3
10
10
10a
102
10a
104
Training set size N
Training set size N
Figure 11: Additional numerical results for subspace denoising. Left and right show the simulated risk of the early stopped empirical risk minimizer Whopps and the PCA estimator WPCA as a function of the training set size N. In the upper part fix the signal dimension and noise level d 10, o₂ 0.1 and vary the ambient dimension n. In the lower part we fix the ambient dimension and noise level 1000, a. 0.1 and vary the signal dimension d. We fit linear scaling laws No to the power law regions. Similar to Fig. we observe that with varying model parameters the scaling coefficients a change. Further, the learned estimator exhibits steeper scaling than the PCA estimator over all settings. Error bars are over 5 independent runs.
26

---

Published as a conference paper at ICLR 2023
singular values of YY". We define Ú_ 1=x==-d the orthogonal complement of U. Starting from the risk expression given in equation
we obtain
(FÜ - nJ ||= + -0
R (CC))
0
( - 1)000 + U 07 J
+ +
( 1)² 0 + uJu
+ +
7 1)a P ( - ()) + 2 U +
(1 ( - 1)
+ ( 1)2 + 00?
1+ 20
o²
UTU
(1+
1+
1 20
(1+
1 1
(4)
1 20²
(d no²)) log(2n)
0
(1 o?)²
N
1+0
- (d + no?)) log(n)
o²
(4)
N
1 oz
Here, incquality (ii) follows from Section H.1 equation (34 and holds in the regime (d no?) log(n) I N, for some constant and with probability at least 1 n-10 3e e This concludes the proof.
G PROOF OF THEOREM 2 RISK BOUND FOR EARLY STOPPED EMPIRICAL RISK MINIMIZATION
This section contains the proof of Theorem 2 The theorem characterizes the performance of the learned, linear estimator W that is obtained by applying iterations of gradient descent with stepsize 7 starting W = 0 to the loss L(W) in equation 2
We start by deriving a closed form expression for the estimator W The gradient of the loss
VwC(W)) = (WY - X)
and thus the iterations of gradient descent are
W W W - X)Y?
-w YY + "XYT.
Let Y = U,E, VT E ROxN and X U.E.V V RAxN be the singular value decompositions of Y and X respectively and assume that the singular values non-zero and descending, i.c., n,1 2 0g22 2 .... We have, with Wo = and 2 that
K-
W = XXY
YYT
0=
nVV, U
= -,
Σ (1 - no UT
XV, DU
27

---

Published as a conference paper at ICLR 2023
where we defined D3 E RNXN diagonal matrix with i-th diagonal entry given by (1 (1 (oo)))) and where we used the geometric series to obtain
k-1 Σa- 2.1) (u--noy.)
1 - (1 no v.a)
110 y.
Note that for - oo and choosing 7 such that 1 - 1-10 n/²
for all we get
W ===xxv XV Uy xY!.
(5)
Evaluating the risk from equation
1 at the estimator W gives
R(W*) - (XV Dull
+ a XV,
(6)
To shorten notation we define
(d + nd? log(n)
=
(7)
N
+ no log(n)
(8)
We next provide bounds for the two terms on the right-hand-side of equation (6. proven later in this section.
Bound the first term in equation 6 In Section G.1 we show that provided N (N) I n,
9d < N and (d + no?) log(n) N, for some constant e, with probability least 1 - -N11
3n
3e-d d e e-N 222 the following bound holds:
3e
7 (XV, D.U. IIU s C j
(o + 7o m.ax))
1 (1 (1 U.i
(1
11 7,1)
d i=1
by
n.max y.k
(1
(1 no 122 cy.
(9)
i=d+1
Bound on the second term in equation
6
In Section G.2 we show
xv
XVUDD y
mana (1 (1 - no ".)
(10)
With equations 99 and 10) place and by splitting up the sum in (T0p we can bound the right hand side of equation 66 as
C d
R(W*) < d (coIn o
mas +
2 a.max) Σ
1 (1- (1 2
d (1
no? g.i)" y2
1=1
i=1
N
02
+ +oy+ + á (V7 + 02)
Σ
n.maz
(1 no y.i) 1/62
(11)
smd+1
Since the singular values of Y are in descending order 0y21 2 0v.2 2 ..., this is, for any iteration k bounded
R(W*) < (co + cyy
+ 2
o2 2
+ c(1 - "oll y.a" 22
N
C + + + + j (107 + +z2) 2
ozmazz
(1 (1 - 10ト
(12)
x=
28

---

Published as a conference paper at ICLR 2023
This is good bound if we' re at iteration sufficiently large that (1 - (1-no 1 is small.
In (12 we have (1 (1- aaa ) that is decreasing in the number of gradient descent steps &s and also decreasing in the stepsize as long as noo S for i = 1,.... d. Further, we have (1 (1-(1- (1-(1-70), ) that is increasing in k and also increasing 7). The first term corresponds to the signal that want to fit sufficiently well, whereas the second term corresponds to the noise from which we want to fit as little as possible. Hence, there exist optimal choices for k, 7) that trade-off the sum of the two terms.
In our setup the d leading singular values of Y corresponding to the signal are large and concentrate around N + o²). while the remaining singular values are small and concentrate around No.. Hence. we can apply a single step of gradient descent k = 1 to already fit a large portion of the signal, while minimizing the portion of the noise that fitted. For that we choose the stepsize y) as large as possible such that no y. I 1 for i = .,....d d still holds.
Next, suppose the following events hold
E fo S N (o3
(13)
E2 = foy. 2 N(1+ (1-())
(14)
Ε for S 4
(15)
& for I N(1 + c) + o
(16)
In Section H.4 show that
[ 2 1 1 2e N/8
(17)
In Section H.4 we also show that, provided that N 2c + o?n) log n, for some constant C and E E (0, 1),
P [[ [() P(E.) 1 - e n
(18)
We next bound the terms in equation 12
As discussed, we set k 1 and the stepsize as large as s 1/0 which holds on event &g. Finally, on event &g
possible, i.c. 1) = 1/ ( )
we obtain
c(1 - no )* S c(1 - n1 + ()) = o ( 1 See
Next we bound the sum in equation with its linear approximation at the origin. We compute the derivative at the origin as
Towards this goal, we upper bound each term in the sum
i (1 (1 (1--g))+ =2 (n1))
lim -
0q q
Thus, we have
N
9,441 S + eo?
(1-
(1-
Σ
12
1212002 o - Nh2
(1+ (1+(( S c(o² + 2c),
i=d+1
v.5
i=d+1
on the event E and for o² < 1, k =
and n = 1/ (N(1+ + oo)). Putting this together and on
the events Eg, Es we get the bound
R(W*) <
o²
2h + c + +oy + c(1 - noo g.a)
- (077 + o?) N
Σ
N
1
(1 (1 n10
1 1+o7
i=d+1
3.4
a2
cN
oZ oanloon (d + no?)
log
<8
+ oz x2 +
+ 0²
(02 + 2c)
1 + o2
d
N
N
(d + no? log(n)
(ld+ no?) log logn)²
of.
< <8R(W*)
c62
(0² 2e)-
N
dN
(19)
29

---

Published as a conference paper at ICLR 2023
The bound holds provided 3Ce 3C d2n) log n s N and N [og((N) I n, for some constants C, C
and with probability least 1 2e -e- N/8 2e N/18 - 5n 5e 2 - c N - -2e n/2
To maximize the benefit from early stopping we set e as small as possible with respect to the condition 3C' (d + o.n) logn - N
3C + o2n))ogg log
(20)
With that equation
19
becomes
R(W*) s 8R(W") + (d+ (d)) (o2))
((d ((d+ 2o)) log 1)g
og.)
(
(o2 +
(d + oīn log
+ d
N
dN
N
= -R((W*) + + (7(1+
no?
o.N)
log n +
(0] + V1)
d
d
< + (1+10) _
log NoN (o v7).
(21)
We now consider the regime where N S E for numerical constant &, simplify the statement further. For this regime, we have
R RWW & R(W (8 (E) + oy (1+
no²
log n + C
d
This concludes the proof of Theorem
G.1 PROOF OF EQUATION 9
In this Section, we derive bound for (XV, D.U, - -Ij the first term in O To this end, we introduce further notation. Recall that Y U, V y RnxN and X U,E, V RAXN are the SVDs of Y and X respectively. All derivations below hold regardless of whether N <n or N > n. Exemplarily, we will show them for N s n. Thus, U, R=NN, Ey NNN Vy R N and Ux Rexd Σg Raxd V NNx.
Let yy, Uz1 E R be the leading left singular vectors of Y and X (note that U₂1 U2), let U y2, U,2 Raax-d be their orthonormal complements. Let U, [Uyl Uy2] E Ruxx and
Ú IU Uz2] E Rann Analogous definitions can be made for the leading right singular
vectors of Y and X.
Recall that D2 = diag (..., (1 (1 - )+) g........). E NRN Let D = [Dx 0] E Nxn
and define D&1 Idddd and D22 N-xn- such that
D 0
D =
(22)
D22
With these definitions in place we can write
IXV.De 1)U
U (XV, D. NJ U
UU
U -DUÇU F
XV Dai U X
12 D k2
I
0 I
U
U
U XV
222 k2
0
n2
X D IJ 31
+ U XV D U U
F
U XV
22
U
U
(23)
30

---

Published as a conference paper at ICLR 2023
The first term in (23) is bounded in the regime 2 N, for constant and with probability at least 1 2e-n/2 by
[ 1n XV DI D y1 /
Σ
1
Σ
110 2 )
U,
(24)
See Section G.I.I for proof. The second term in (33 is bounded for some constant c, with 2e-d e and in regime (d + noz) log(n) s N by
probability at least 1 2n 10
U XV D
k1
U
U
sov
Σ
z,maz (1 T
no (1, M.)..
(25)
See Section G.1.2 for a proof. The third term in 23 least 2e N/18 3n 10
bounded for some constant e, with probability
3e-d
N and the regime (d + no?) log(n) S N,
N (o) S and 9d
N by
02 (1 no + ov (26)
UUXVV
U V2
U
S Se,
a.aax
+
i=++1
y,i
See Section G.1.3 for proof.
Combining those results
bound equation 23 in the regime N log(NN n, 9d N and
(d + no?) log(n) < N, for some constant and with probability at least 2c -N/18 30 3e- e e N
n/2 nm/
co d
d
2 [|VV D.U - D -
+ d
jozzmaz? 7
Σ
2 (1 (1 no 4,
no y,s) )
1=1
1=1
o2
bby
a,man
(1- (1
no g.r) )/) ) cy.
(27)
d
Mi
This concludes the proof of equation
G.1.1 PROOF OF EQUATION 24
In the regime 2 N, for some constant and with probability at least 1 2e-n/2 the first term in bounded by
23
[UT
XV Ve DA
NU
U
V
DA
I
D
Ey1 D
+
Σg1 D - 1
V
Ena ||| D
+
Ey D 1
d
d
C0
V
Σ
1
(1
n70 2.
)092
+
Σ
(1 no
1=1
(28)
Inequality (i) uses
| 41 U| S 1. To obtain inequality (ii), we used that DA = (........(
(1
(1 - (1-100 )r ) E ddd and that
U U
UN
E V
En1l
U En V U,1 E, v
yi
<
U.E. VT - U.E.V
|Z]l
vn.Vn.
(29)
31

---

Published as a conference paper at ICLR 2023
Here, the last inequality holds in the regime 2 N, for some constant and with probability at least 1 2e-n/2 and follows from Section H.4. equation (64)
G.1.2 PROOF EQUATION 25
For some constant c, with probability at least - 2n-10
4e e e
and in regime (d
no?) log(n) & N the second term in 23 is bounded by
U y2 XV
U
UUU 12 1
U yi
U|
|
D
21
|
Uni
|2
D&L
(0)
d
o2
c/7
mama
(1 no )²
(30)
1=1
Here, inequality (i) follows from
V V m1 S 1 and U U| S 1. For inequality (ii) we used yi
that D diag (..., (1 (1 no /)/ /oy....... E Raxd and the bound on | y2 Un1 from
Section H.1 equation (35) that holds in the regime (d + no?) log(n) S N, for some constant and
with probability at least 2n 2n-10 2c
G.1.3 PROOF EQUATION 26
For some constant e, with probability at least 2e- -N/18 3n
3e -d
-N and in
e
the regime (d + no?) ( log(n) S N and 9d < N we obtain
U UU XV De2
0
T
UT X D2
82 U
I
02 XV U2
U Un E, V D U
U
U Un En V D U
U Ul
(VT + y)v / v Emil | + cv/d
02
c uby
R.mor (1 (1 - "0, + oo (31)
i=d+1
Here, inequality (i) follows from | U21 I 1 and the bound in Section H.1 equations 34 35
and (36) and holds the regime (d + no 2) log (m) S N, N log(NN S n and 9d N, for some
constant and with probability at least 2e - N/18 3n-10 3e-d e- e Inequality (ii)
holds the regime (d + ( log(n) S N, which implies y < 1 and we used the definition of D
from equation
G.2 PROOF OF EQUATION 10
Recall that ; = diag (..., (1 (1 - (1-110 M) can be bounded as
y..)....) //
NxN
The second term in equation (6)
2
XV D U
U E V V D 1
d
V. V y
2
D
2
N
I.max yi
(1 - (1 - no " )2
(32)
where we used that
[ VVVV V
and the definition of D3 from equation
22
32

---

Published as a conference paper at ICLR 2023
H AUXILIARY PROOFS
In this Section we provide summary of auxiliary proofs that used to prove the main results in Sections F and G
H.1 APPLYING THE SIN-THETA THEOREM TO BOUND THE DISTANCE BETWEEN SUBSPACES
In this Section, we use the following variant (Cai et al. 2013 Prop. I) of the sin-theta theorem Davis & Kahan 1970) to bound the distances between subspaces occurring in the proofs in Section F and
Proposition 1. Let Q and Q be n x nt symmetric matrices. Let r < n be arbitrary and let U and U be formed by the leading singular vectors of Q and Q. Then
l00 -UT
2
Q o/|
(33)
(. -0r+1 (Q)
Recall that Y = U E R and X = U.E.V? E R are the SVDs of Y and X respectively. Let Uy,, U21 E Rexd be the d leading left singular vectors of Y and X, let U22. U22 E Raxn-d be the orthonormal complements. Let U, [U,] Uy21 E T=** and U 1 U22] E Ruxn Analogous definitions can be made for the leading right singular vectors of Y and X.
We start by applying Proposition I to bound the distance between the subspaces spanned by the leading left singular vectors U,I of Y and the subspace model U
U
n2
U y U UU'
YY UU - -
N
c(d + no?) log(n) N
c(d + no?) log(n)
+
N
(00)
(d+ no? log(n)
(34)
N
where inequality (i) follows from Proposition I and inequality (ii) holds with probability at least
1 n 2e -d e " and follows from Section H.2 equationn (45). Inequality (iii) holds in the regime (d + no?) log(n) < N.
Next, we establish bound for the distance between the subspaces spanned by the d leading left singular vectors of X and Y. We have that
T 22
unll
||
U21 U || u.
y1
VA
UT
U U U
y1
1A
YY UU" -u"-o'1 +
1 xxx -xx"_Uu"
UUT
1A
c(d + no?) log(n)
c(d + no?) log(n)
cdlog(n)
oog())))
N
N
N
c(d log(n)
c(d + no?) log(n)
N
N
(12)
(d + no? log(n)
(35)
N
where inequality (i) follows from Proposition I and inequality (ii) holds for some constant e and with + e and follows from the results in Section H.2 equations (45). 46).. Inequality (iii) holds in the regime (d + +na? log(n) N.
probability at least 1 - 2n 10 2e 4d
33

---

Published as a conference paper at ICLR 2023
Last, we establish a bound for the distance between the subspaces spanned by the right singular vectors of X and Y. We have that with probability at least 1 2e 2e-N/18
2n 10 e-" - e-
V 21 V well ||
Vai
VT
Vy1
V
21
2
Y
X'X
0 x,d+1
2,d
(1)
Y
xx
1 CTUTZ
S 6
Z
20
N
N
8 0
ZZ"
+ +
+ 20
c"UZ
N
N
680
cno log(n)
ena? log(n)
+ co2 log(N)
N
N
n nlog(n)
n log(n)
N2N log(N)²
co3.
+ co
vV
N
N
N
(10)
n log(n)
pgnlog(n)
co
2
+ev
N
N
(v)
no? log(m)
(36)
N
Inequality (i) follows from Proposition
that I S o2 holds in the regime 9d < N, with probability at least 1 - 2e N/18 a. ",d+1 0 and
Inequality (ii) follows from the fact that o2
and for some
constant c (see Section H4.4] equation (63)).
Inequality (iii) holds for some constant c and with probability at least 1 2n -
and follows from the results in Section H.2 equations (47) (38 Incquality (iv) holds in the regime S N. To abbreviate notation from
N log(N) < n. Inequality (v) holds in the regime no log(n) n
now on we define
no log(n)
(37)
N
H.2 BOUNDING A SUM OF INDEPENDENT RANDOM MATRICES
The Matrix Bernstein inequality (Oliveira 2010 Tropp 2012) can be used to bound a sum of independent, bounded and centered random matrices. We state the theorem below and then show how we applied it to bound several terms occurring in Sections Fj G
Theorem 3 (Matrix Bernstein), Let Si,..., Sn be independent, centered random matrices with common dimension d x d, and assume that each one is uniformly bounded
E [ = and ||&&|| < L for each k = ,,...,n.
Introduce the sum
Z - Σ S
k=
and let v(Z) denote the matrix variance statistics of the sum:
u( (Z) max
{ E
ZZ T
E
|
maxx Σ E
S
71
1
ΣE
S T
Then with probability at least
8 and 8
Z
< 3
L( + log(2d)) + (2 + log((d))
34

---

Published as a conference paper at ICLR 2023
Recall our signal model to be Y x UC Z, with entries i.id. c.j NN(0,1 1) and
Z.j (O o²) and dimensions Y,X, RT*N C E R The subspace matrix U E Raxd has orthonormal columns.
We start by applying Theorem to establish the following bound. With probability 1 N-10 e-N N c-d and for some unspecified numerical constant e we have
+ VIIC u² S co2 log(N).
(38)
We define 2 U E R Note that the entries in Z are independent and identically distributed like the entries in Z, i..., 2 ~ NN of). Next we check the conditions of applying Theorem 3 to bound cTT
Proof of equation 38);
Note that
CT
Σ c2
(39)
2=1
with c; 2[i] RRN being the rows of C, Z. Since c; has zero mean,
E
[c,2
0,
for all i = I.... d. Further, have
||,,2
c,,2," wl12 max
1|w 1
1 w w/llcill2
max ||w [wlla=1
|2:112 ||c:ll2
|
|| ll ||cl:ll2
I c0,,,
(40)
where the inequality follows from equations (56) and holds with probability at least 1 -e- Finally we need to compute the matrix variance statistic vi( v(C 2). Note that
c,2 (c,22
)
E 2 2.c.c."]
= N
= E (cx"t) c.2!!
(41)
and therefore
v( (C max
Σ c (c,z²
Σ E (c,z) c,
dN.
(42)
With equations 440
in place ready to apply Theorem obtain with probability least and some constant
1
N
(og()
cf UZ
+ co (Ng(N)
N
N
00, log(N))
(43)
where the last inequality holds since d < N. This concludes the proof of equation (38)
In the remainder of this Section we apply the example in Tropp 2015 Sec. 1.6.3) that illustrates how use Theorem 3 to bound the distance between sample true covariance matrices. With probability at least 1 10 and an unspecified numerical constant e we have
N
cB
laa?]
log(n)
+ cB log(n)
a,a;" E [aa"] <
(44)
N
N
N
35

---

Published as a conference paper at ICLR 2023
where assume that the la norm of the random vector R is bounded ||al = B.
In the following we show how to apply (44) to establish the following bounds. With probability at least 1
10
2e te and for some unspecified numerical constant c
1 N
1 YYT -U - - N
Σ y.y.
-B [yy"] =
c(d + no? log(n)
c(d + no?) log(n)
(45)
N With probability least n- e- and for some unspecified numerical constant c
N
x T E Xx""
xx UUT
ed (on) cdloo(())
(46)
With probability at least 1 - n - e- and for some unspecified numerical constant
N
2,z -E 2z 21
zz o,'
cno log(n)
ena log(n)
(47)
N
N
the proofs of for ||al| < B and | [aa")) respectively.
In the notation of the general example in (44).
consist of deriving expressions
Proof of equation
45
We have
||y11 = c 4-22 2z Uc z'z c
$ (5 + v5)d + 5no S c(d + no?),
(48)
for some constant and where the first inequality holds with probability at least 2e-d e shown in Section H.3 Further, have
as
|E [yy"] ]|I
"ou""o"''' - = 1
(49)
Since practical noise levels satisfy of < 1, there exists some constant such that
y E [yy"]] c(d + n
(50)
with probability at least 2e
Inserting and (50) in the general form in 44 concludes
the proof of equation (45)
Proof of equation (46):
We have
| 2 c" c S cd
(51)
for some constant c and where the inequality holds with probability at least - -d as is shown in Section
H.3 Further, we have
| [xx"]] = |00 1
(52)
Inserting 51 and 52) in the general form in 44 concludes the proof of equation (46).
Proof of equation
We have
| & cno
(53)
for some constant and where the inequality holds with probability at least - as is shown in
Section H3 Further, have
| 2z
o²-
(54)
Inserting 53 and (54) in the general form in 44 concludes the proof of equation 47
36

---

Published as a conference paper at ICLR 2023
H.3 TAIL BOUNDS FOR INNER PRODUCTS OF GAUSSIAN VECTORS
Recall that cv (0,, E R and z ~ (0, o E IR are the columns of C, Z respectively. Also U E Raxd has orthonormal columns. In this section we state some relations on the concentration of the inner products between those vectors. The chi-squared distributed c c can be bounded with probability most
cTc 2
2d2++20 + 21.
(55)
Substituting with the signal dimension gives
cº c 2 5d,
(56)
with probability at most e 4, Using the same result
write
z² z z² z 2 5no
(57)
with probability at most em and where z' ~ (0, I). Next we show that with probability at most 2e 2end
UC 2 V5d.
(58)
To this end, note that for any (deterministic) vector we have x UC ~ N/ |lel1l and we apply simple tail bound for Gaussian random variables to get
z Z' 2 ||c||t.
(59)
with probability at most -t" It is straightforward to see that substituting with vd, applying (66
to bound ||e|| and combining the results with union bound results into the bound 58 We can combine the results from this section to bound the sum
C 2z zJ (5 v5)d + 5no
(60)
with probability at most 2d + -en
H.4 BOUNDING THE EXTREME SINGULAR VALUES OF GAUSSIAN RANDOM MATRICES AND EMPIRICAL COVARIANCE MATRICES
In this Section we state results for the extreme singular values of some of the matrices occurring in Sections F and Specifically, we establish equations 17
29 and 36) (ii).
A standard deviation inequality for the extreme singular values of some matrix A E M with independent and identically standard normal distributed entries implies that (Rudelson & Vershynin 2010 equation (2.3))
VM vm S Trinn (A) S Tmax (A) S VM + Wm+4
(61)
with probability at least 2c -2- for / 2 0.
12/2
We start with the largest singular value a,maa of the feature matrix X R= Recall that we have X UC with orthonormal U e Ra and C d with i.i.d. entries c.j ~ (O 1). Hence, the singular values of UC are the singular values of C.
Applying equation 6T) the matrix C under the assumption that the hidden signal dimension d and the number of training examples N fulfill 4d < N and choosing v /N/2 we obtain
VN
Ozmmax < N + d
+
2
VN
VN
2
2
N,
(62)
37

---

Published as a conference paper at ICLR 2023
which holds with probability least 2e-N/S This establishes equation 17 For z.min the smallest non-zeru singular value of X we have J,min = D,, = Je.d- In the regime 9d N and choosing = VN /3 we obtain
VN
VN-
vd
3
N
VN
S Oz,min
3
3
VN
N
S a.mins
(63)
which holds with probability at least - 2e N//88 and for some constant c. This establishes (36 (ii).
Next, state bound on the largest singular value ||z]| of the noise matrix Z E <R with i.i.d. entries Zi.j ((00,)) Note that Z Zz where the entries of Z follow Σي (N Applying equation (1] with vN + /n yields
Z
S co. + vn)
co,VVT
(64)
with probability at least 1 - 1-2e-mm 2e/2 and some constant For the last inequality assumed n 2 N.
This establishes equation (29).
Finally, we derive bounds for some of the squared singular values of E RaxN, particular, we show that
jy 2 (1 - c)
(65)
and
0² gdt11 S N (0. + e(1 + +o?))
(66)
and finally
22 - N ((1 () e) +())
(67)
both hold with probability at least 1 c cm and for some constants C' and E (0, in the regime 2 3Cc-- o②n) log n. This establishes equation (18).
To this end, rely the following Corollary (Vershynin, 2011 Corollary 5.52).
Corollary 1 (Covariance estimation for arbitrary distributions). Let x be random vector in IR" supported some centered Euclidean ball whose radius we denote ym. Consider N independent samples x; arranged as columns of the random matrix A E TRaNN Denote EN AA as the sample covariance matrix and E as the true covariance matrix. Let e (0,1) and Z 1. Then the following holds with probability at least n -*:
N C(t/c) | m log then [N Σ|| < <||||
Here C an absolute constant.
We apply Corollary to the random vectors y Uc z. Note that E [yy
-UUT + L Further
note that E ||x| d nb We have
llyll2 < ||c||2 + ||z||2
< /5d + / /5no?
< <</ d + no?
(68)
where the second inequality follows from Section H. (56).
(57)
and holds with probability at least
1 e-d e-n
38

---

Published as a conference paper at ICLR 2023
Now apply Corollary [
to make the following statement. For some constant C and E (0,1)
if Ctc (d + o2n) log n. then with probability at least e-d
YY E [yy"]
< [yy"]l c(1 oz).
(69)
Consequently the singular values 0il N YYT) and (B [yy²]) differ by at most c(1 + oz)) and we can bound
b.d Noal 1 YYT) N
-(1 + o)))) 2 Niaa(E [yy") "
-N((1 (1 +
-1-c))((1+
(70)
and further
oy
hd++ No
-
1
rYT)
N
N (dy,t+1 (E (yy")) + E(1 + +??))
N (o + ((1++
= N ((1 +e)o² + e),
(71)
which concludes the proof of equations (65 and (66) In the same manner bound
1
HYT)
0/11 Noy.
N
S N (or, (E [yy") 'I + +c(1 + +o?))
N (1+ + + + +0)))
N ((1 (c +c)())))))
(72)
which concludes the proof of 667
39


Reviews:
[{"review_id": "wEFp-74PVjl", "reviewer": "Reviewer_aCDc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigated on the relationships between the performance in deep learning based inverse problems and the size of the training dataset. Denoising, compressive sensing MRI and super resolution problems were studied for U-Net and SwinIR. Theoretical analyses were also presented under simplified conditions. The conclusion of this work was that more training images beyond a certain point do not improve performance once the error induced by learning the signal model is small relative to the error floor, which was intuitively considered.", "summary_of_the_review": "While this work attempted to investigate an interesting problem on the relationship between the performance of inverse problems in imaging and the size of training sets, the presented experiments seem limited and there are a number of potential issues in them. Thus, I can not recommend its acceptance in its current form.", "strength_and_weaknesses": "Strengthes:\n- Comprehensive empirical results are presented.\n\nWeaknesses:\n- Experiments seem to be limited to a very small set of all possible cases so it is unclear if the conclusion is well supported by the current experiments (see below for more details).\n- It is unclear how the proposed conclusion can help researchers in inverse problems. Its usefulness and practical value are not really clear.\n- The assumption on \"Current methods are only trained on a few hundreds or thousands of images as opposed to the millions of examples deep networks are trained on in other domains\" seems somewhat biased in my view. Tasks are different and the way of using one image is different (e.g., one image can not be divided into two or more independent images for a label in classification task, but it is common that multiple independent patches are extracted for training in denoising or super resolution tasks). Moreover, see (Brooks et al., 2019) that used 1M training samples for training a denoiser. Thus, it is hard to agree with the motivation of this work.\n- The analysis with a linear estimation learned with early stopped gradient descent seems quite far from modern deep learning based methods that are highly non-linear and fully converged. Thus, all the conclusions made in this work seem over-claimed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "v92VYSiabc", "reviewer": "Reviewer_nBGL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied whether performance gains of neural network approaches for inverse/regression problems are obtained from scaling up the training set size. They reported their claims in three fields: image denoising, accelerated MRI, and super-resolution.\nThey calculated reconstruction quality as a function of training set size, while optimally scaling the network size. The authors found that an initial performance increasement slowly downs already at moderate training set sizes(~10^3). They confirmed that even training on millions of images would not significantly improve performance over small-sized training. \nThey concluded that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.", "summary_of_the_review": "The paper analyzes the scalability of training dataset in linear inverse problems and this is quite needed property from this society. I'm convinced of their claims and the authors prepared well-designed experiments to back up their claims. I believe their work is a good work. This paper will help relevant societies a lot.", "strength_and_weaknesses": "(strength)\n+ This paper analyzes the scalability of training dataset in linear inverse problems (denoising, super resolution, MRI acceleration). This aspect has never been analyzed significantly, so I believe this manuscript is first attempt to give a focus on the scalability of dataset in image restoration problem. This aspect is quite important in terms of intuitive instructions of building database. \nFurthermore, the authors try to elaborate such the scalability property with rigorous theories which are supported in supplementaries in details.\n\n(weakness)\n- The way of writing from section 5 until conclusion is a bit distracted. This manuscript has a target to anlayze the data scalability, but their explanation from section 5 seems to be irrelevant with their claim. It would be easier  than the current. For example, in Fig. 3, W^\\infty is not explained in caption and section, rather authors provide a link to supplementery which is not self-contained. I believe there is a need to be compact writing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mwqSJHjaen8", "reviewer": "Reviewer_duwb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies scaling of performance of deep learning-based methods for different image reconstruction tasks with respect to available training set sizes. Tasks investigated are U-net and SwinIR transformer for image denoising (and super-resolution in the appendix), as well as U-net for MRI compressive sensing. Experiments are pertinent and convincing. Discussions for the found power laws provide suitable insight. In addition, a theoretical study on linear subspace estimation from noisy data is provided, supporting the empirical observations by explaining/reproducing their general behavior in this simplified setting.", "summary_of_the_review": "Good paper delivering insight above performance. The paper would be stronger with a clearer message towards noise depencence of the findings.", "strength_and_weaknesses": "Strengths: The paper is very well written in an easy to read language. Contents are novel and interesting to the expected audience of the conference. Main findings are presented very clearly and supported by well-designed easy to digest figures, that would allow to grasp the main findings from the figures even without reading the text in depth.\n\nNew SOTA in some of the tasks come as a side product of scaling but are only given as a side remark in the main story.\n\nWeaknesses: As with most experimental deep-learning studies the findings depend on hyperparameter settings that may have more or less dramatic effects on the results. The implicit assumption, that hyperparameter choices are in all cases in a suitable range cannot be supported without solid ablation studies. They are not given in the current paper and thus results presented need to be taken with a grain of salt.\n\nA weakness of the paper is that experiments on image denoising do not vary noise levels. Thus, in the experiments it remains open, how the power laws change with different noise levels or models. This would be quite interesting in practice, as the selected noise level sigma = 25 is unusually high. It remains unclear (at least to this reviewer) how the found tipping points, where the steep power laws flatten considerably, change with noise level. This is quantitatively investigated in the theoretical part of the paper and shown in Figure 4. There it seems that lower noise levels correspond to smaller needed data set sizes. For image denoising this may be different, due to the typical signal distribution in Fourier space, where low-frequent structures have higher amplitudes than high-frequent ones and thus high noise levels ‘occlude’ high-frequent structures more than coarser ones. Lower noise levels would therefore allow to recover larger parts of the image signal and thus larger datasets may be needed before reaching the noise floor. Additional experiments would be needed to clarify this question.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Suma4taxyX7", "reviewer": "Reviewer_pVJy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper conducts an empirical study of how the image reconstruction accuracy changes with the number of training observations for low-level tasks such as image denoising and compressed sensing. For that, multiple models with different numbers of parameters are trained for each training set, and the best test accuracy is considered for the final plot. The plot reveals that the accuracy has an initial zone of high improvement after which a zone with more modest improvement is observed. The paper also obtains the scaling plot for a vision transformer for image denoising. Finally, the paper obtains some theoretical guarantees for a related problem: low-rank linear denoising.", "summary_of_the_review": "The paper shows promise in obtaining scaling laws for denoising, but it is not ready for publication at this time. ", "strength_and_weaknesses": "Strengths:\n- Obtains some empirical results on the best accuracy that could be obtained for different numbers of training examples for two image reconstruction tasks.\n- Obtains some theoretical guarantees for another image reconstruction task.\n\nWeaknesses:\n- The scaling of the transformer on the MRI reconstruction is missing\n- The literature review of the theoretical guaranteed for SVD reconstruction is weak. The authors fail to discuss how the results from Theorems 1 and 2 compare to the already existing theoretical results from the literature.\n- Theorem 2 is a very loose bound that has a rate worse than Theorem 1 because of the 1/sqrt(N) term plus (7+2 xi)R(W*). From Theorem 2 it would result that early stopping is worse than PCA, which is inconsistent with the experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8ygCH6LqaG", "reviewer": "Reviewer_ksGx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the impact of training set size for imaging applications. The paper shows that increasing training size will not yield similar gain in terms of performance. Quantitative experimental results conducted on image denoising and reconstruction aligns with the claims.", "summary_of_the_review": "The paper investigates the impact of training data size. While this work might be useful for designing training data size for some imaging applications, I feel the approach proposed in this paper is not well-motivated. Moreover, generalizations of the results to different imaging applications are questionable since only some quantitative metrics which may not be reflective of the performance are provided.", "strength_and_weaknesses": "Strengths:\n\n1. The paper shows that deep learning-based imaging tasks does not require immense amount of data unlike other domains such as NLP. \n2. The experimental results can navigate designing the amount of training data for imaging applications \n\nWeaknesses:\n1) Authors motivate their study by contrasting with another domain such as NLP. While it is true that imaging applications require far less data compared to NLP, authors do not really explain/motivate the reason behind it.  Authors try to explain the performance in terms of network size and training size. However, in imaging applications, every pixel in an image can be considered as a sample. Thus, the available amount of data is scaled exponentially.  Indeed, in many recent denoising applications, a set of pixels are used as a label rather than entire image (e.g. Noise2Void, Noise2Self, …). In imaging applications such as image denoising or reconstruction, a moderate performance can be achieved with even 1 sample/image (e.g., self2self). In this perspective, in imaging applications, available data for the task of interest is much larger than the number of images.  \n\n2) In denoising applications, quantitative results might be used as a metric to back-up the results. However, this cannot be the only metric for MRI reconstruction shown in the study. A high SSIM does not necessarily translates to an artifact-free reconstruction. A method can achieve a very high SSIM while suffering from artifacts (e.g., Muckley et al. 2021). Thus, in MRI or similar medical imaging applications, the amount of training data can be crucial in removing coherent artifacts which worsen with higher acceleration rates. Thus, I find the claims in section 4 questionable.  Extension to such domains could be a separate study that should be supported with extensive qualitative results.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster