Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies scaling of performance of deep learning-based methods for different image reconstruction tasks with respect to available training set sizes. Tasks investigated are U-net and SwinIR transformer for image denoising (and super-resolution in the appendix), as well as U-net for MRI compressive sensing. Experiments are pertinent and convincing. Discussions for the found power laws provide suitable insight. In addition, a theoretical study on linear subspace estimation from noisy data is provided, supporting the empirical observations by explaining/reproducing their general behavior in this simplified setting.
strength_and_weaknesses: Strengths: The paper is very well written in an easy to read language. Contents are novel and interesting to the expected audience of the conference. Main findings are presented very clearly and supported by well-designed easy to digest figures, that would allow to grasp the main findings from the figures even without reading the text in depth.

New SOTA in some of the tasks come as a side product of scaling but are only given as a side remark in the main story.

Weaknesses: As with most experimental deep-learning studies the findings depend on hyperparameter settings that may have more or less dramatic effects on the results. The implicit assumption, that hyperparameter choices are in all cases in a suitable range cannot be supported without solid ablation studies. They are not given in the current paper and thus results presented need to be taken with a grain of salt.

A weakness of the paper is that experiments on image denoising do not vary noise levels. Thus, in the experiments it remains open, how the power laws change with different noise levels or models. This would be quite interesting in practice, as the selected noise level sigma = 25 is unusually high. It remains unclear (at least to this reviewer) how the found tipping points, where the steep power laws flatten considerably, change with noise level. This is quantitatively investigated in the theoretical part of the paper and shown in Figure 4. There it seems that lower noise levels correspond to smaller needed data set sizes. For image denoising this may be different, due to the typical signal distribution in Fourier space, where low-frequent structures have higher amplitudes than high-frequent ones and thus high noise levels ‘occlude’ high-frequent structures more than coarser ones. Lower noise levels would therefore allow to recover larger parts of the image signal and thus larger datasets may be needed before reaching the noise floor. Additional experiments would be needed to clarify this question.

clarity,_quality,_novelty_and_reproducibility: Clear and novel study of high quality but limited coverage. Should be reproducible from the details given in the text and especially the appendix.
summary_of_the_review: Good paper delivering insight above performance. The paper would be stronger with a clearer message towards noise depencence of the findings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper conducts an empirical study of how the image reconstruction accuracy changes with the number of training observations for low-level tasks such as image denoising and compressed sensing. For that, multiple models with different numbers of parameters are trained for each training set, and the best test accuracy is considered for the final plot. The plot reveals that the accuracy has an initial zone of high improvement after which a zone with more modest improvement is observed. The paper also obtains the scaling plot for a vision transformer for image denoising. Finally, the paper obtains some theoretical guarantees for a related problem: low-rank linear denoising.
strength_and_weaknesses: Strengths:
- Obtains some empirical results on the best accuracy that could be obtained for different numbers of training examples for two image reconstruction tasks.
- Obtains some theoretical guarantees for another image reconstruction task.

Weaknesses:
- The scaling of the transformer on the MRI reconstruction is missing
- The literature review of the theoretical guaranteed for SVD reconstruction is weak. The authors fail to discuss how the results from Theorems 1 and 2 compare to the already existing theoretical results from the literature.
- Theorem 2 is a very loose bound that has a rate worse than Theorem 1 because of the 1/sqrt(N) term plus (7+2 xi)R(W*). From Theorem 2 it would result that early stopping is worse than PCA, which is inconsistent with the experiments.

clarity,_quality,_novelty_and_reproducibility: There is novelty in both the scaling empirical plots and the theoretical guarantees but it is not clear whether Theorem 1 has already been proved in the literature or not and the authros fail to discuss it.
The error bars in Fig 3 are hidden behind the line symbols, which are quite useless. 

summary_of_the_review: The paper shows promise in obtaining scaling laws for denoising, but it is not ready for publication at this time.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the impact of training set size for imaging applications. The paper shows that increasing training size will not yield similar gain in terms of performance. Quantitative experimental results conducted on image denoising and reconstruction aligns with the claims.
strength_and_weaknesses: Strengths:

1. The paper shows that deep learning-based imaging tasks does not require immense amount of data unlike other domains such as NLP. 
2. The experimental results can navigate designing the amount of training data for imaging applications 

Weaknesses:
1) Authors motivate their study by contrasting with another domain such as NLP. While it is true that imaging applications require far less data compared to NLP, authors do not really explain/motivate the reason behind it.  Authors try to explain the performance in terms of network size and training size. However, in imaging applications, every pixel in an image can be considered as a sample. Thus, the available amount of data is scaled exponentially.  Indeed, in many recent denoising applications, a set of pixels are used as a label rather than entire image (e.g. Noise2Void, Noise2Self, …). In imaging applications such as image denoising or reconstruction, a moderate performance can be achieved with even 1 sample/image (e.g., self2self). In this perspective, in imaging applications, available data for the task of interest is much larger than the number of images.  

2) In denoising applications, quantitative results might be used as a metric to back-up the results. However, this cannot be the only metric for MRI reconstruction shown in the study. A high SSIM does not necessarily translates to an artifact-free reconstruction. A method can achieve a very high SSIM while suffering from artifacts (e.g., Muckley et al. 2021). Thus, in MRI or similar medical imaging applications, the amount of training data can be crucial in removing coherent artifacts which worsen with higher acceleration rates. Thus, I find the claims in section 4 questionable.  Extension to such domains could be a separate study that should be supported with extensive qualitative results.



clarity,_quality,_novelty_and_reproducibility: The paper is well-written with moderate contributions. 
summary_of_the_review: The paper investigates the impact of training data size. While this work might be useful for designing training data size for some imaging applications, I feel the approach proposed in this paper is not well-motivated. Moreover, generalizations of the results to different imaging applications are questionable since only some quantitative metrics which may not be reflective of the performance are provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work investigated on the relationships between the performance in deep learning based inverse problems and the size of the training dataset. Denoising, compressive sensing MRI and super resolution problems were studied for U-Net and SwinIR. Theoretical analyses were also presented under simplified conditions. The conclusion of this work was that more training images beyond a certain point do not improve performance once the error induced by learning the signal model is small relative to the error floor, which was intuitively considered.
strength_and_weaknesses: Strengthes:
- Comprehensive empirical results are presented.

Weaknesses:
- Experiments seem to be limited to a very small set of all possible cases so it is unclear if the conclusion is well supported by the current experiments (see below for more details).
- It is unclear how the proposed conclusion can help researchers in inverse problems. Its usefulness and practical value are not really clear.
- The assumption on "Current methods are only trained on a few hundreds or thousands of images as opposed to the millions of examples deep networks are trained on in other domains" seems somewhat biased in my view. Tasks are different and the way of using one image is different (e.g., one image can not be divided into two or more independent images for a label in classification task, but it is common that multiple independent patches are extracted for training in denoising or super resolution tasks). Moreover, see (Brooks et al., 2019) that used 1M training samples for training a denoiser. Thus, it is hard to agree with the motivation of this work.
- The analysis with a linear estimation learned with early stopped gradient descent seems quite far from modern deep learning based methods that are highly non-linear and fully converged. Thus, all the conclusions made in this work seem over-claimed.
clarity,_quality,_novelty_and_reproducibility: - This work presented limited experimental results only using U-Net and SwinIR. The trends of these results look similar, but they are different in details. Thus, it will be also different for other networks. Can we say that there is a common phenomenon by looking into two different examples? More comprehensive experiments with a lot more networks may help to find this common aspect. Moreover, the most worried concern that I have is this question: "Are U-Net or SwinIR scalable?" It may be possible that this work simply investigated two non-scalable networks and there might be a scalable network for inverse problems. Another possible bad scenario is that the observed trends may not be the result of large training data, but the result of difficult global convergence for high dimensional spaces! Thus, in my view, it seems that this work may not be ready to draw the conclusion that the title and the conclusion claimed. 

- The experiments in this work seem to be limited in many aspects besides the network types. This work covers the size of the training set up to 100K, but as mentioned above, there was a work using 1M samples (Brooks et al., 2019). It will be more convincing to cover larger settings than current settings to show really interesting areas to look into. Moreover, the way of increasing the size of networks is important (e.g., increasing depth, width and/or channel), but this work did not discuss this aspect at all. While it was claimed that the optimal parameter number was found for each case, but it is unclear if it is really optimal in terms of the way of increasing the network size. how about using EfficientNets since they are providing different network sizes?
summary_of_the_review: While this work attempted to investigate an interesting problem on the relationship between the performance of inverse problems in imaging and the size of training sets, the presented experiments seem limited and there are a number of potential issues in them. Thus, I can not recommend its acceptance in its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied whether performance gains of neural network approaches for inverse/regression problems are obtained from scaling up the training set size. They reported their claims in three fields: image denoising, accelerated MRI, and super-resolution.
They calculated reconstruction quality as a function of training set size, while optimally scaling the network size. The authors found that an initial performance increasement slowly downs already at moderate training set sizes(~10^3). They confirmed that even training on millions of images would not significantly improve performance over small-sized training. 
They concluded that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.
strength_and_weaknesses: (strength)
+ This paper analyzes the scalability of training dataset in linear inverse problems (denoising, super resolution, MRI acceleration). This aspect has never been analyzed significantly, so I believe this manuscript is first attempt to give a focus on the scalability of dataset in image restoration problem. This aspect is quite important in terms of intuitive instructions of building database. 
Furthermore, the authors try to elaborate such the scalability property with rigorous theories which are supported in supplementaries in details.

(weakness)
- The way of writing from section 5 until conclusion is a bit distracted. This manuscript has a target to anlayze the data scalability, but their explanation from section 5 seems to be irrelevant with their claim. It would be easier  than the current. For example, in Fig. 3, W^\infty is not explained in caption and section, rather authors provide a link to supplementery which is not self-contained. I believe there is a need to be compact writing.
clarity,_quality,_novelty_and_reproducibility: The overall flow of manuscript is well written and quality of writing is also fine.
The novelty of the work is appealing and authors submitted their code in supplementary.
summary_of_the_review: The paper analyzes the scalability of training dataset in linear inverse problems and this is quite needed property from this society. I'm convinced of their claims and the authors prepared well-designed experiments to back up their claims. I believe their work is a good work. This paper will help relevant societies a lot.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper empirically derives scaling laws for three image reconstruction problems (inverse estimation, denoising, and super-resolution) and two neural network architectures (CNN and transformer). Additionally, there is some theoretical justification provided. The empirical finding that performance saturates rather quickly is useful to know, and might spur research towards overcoming this limitation.
justification_for_why_not_higher_score: By its nature, the most important scaling results are empirical, and limited to the particular task studies. Thus their applicability is limited.
justification_for_why_not_lower_score: It is valuable for the community to learn about case studies such as this one, and it can spur future research into methods with superior scaling.
Decision: Accept: poster