Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper analyze the soft sampling method for training DNNs, which selects a subset uniformly at random with replacement from the full data set in each epoch. Analysis of convergence rate and coverage are conducted.
strength_and_weaknesses: Strength:
1. The theoretical analysis of convergence and coverage of the sampling method could be a useful supplement to the methodology.

Weaknesses:
1. There is no major technical contribution in this paper. Algorithm 1 seems to be a standard practice of DNN training.
2. The theoretical results of convergence might be useful. It is not clear how the analysis coverage and occupation could be useful.
3. The evaluation part is weak. A larger scale dataset, like ImageNet is required for evaluation for validating the effectiveness.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read. 
summary_of_the_review: As the technical contribution of this paper is weak, I recommend reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
1. The paper argues in favor of random sampling (termed as soft sampling) over other methods - specifically coreset based. The argument is well understood and to summarize 
   a. The advantage of coresets in accuracy over random sampling is limited in some (maybe authors want to argue over many / most? But the evidence is provided on 3 datasets, 1 of which is very large) large scale datasets. 
   b.  On the other hand, the computational cost of coresets is usually very high which is well known and accepted. 
To extend the argument, the time spent in coreset computation can actually be used to train models for longer times with random sampling 

2. The analysis seems fairly straightforward which is not a bad thing in my opinion. However, it definitely requires more scrutiny. Specifically, how is the convergence analysis different from mini-batch SGD analysis with bounded variance and smooth and lipschitz continuous ? More specifically, what new thing do we learn here ? Also, sample coverage and occupancy computations are important to understand the dynamics of random sampling in the context of training. However, to get fair understanding of the contribution, you might want to compare against a mini-batch SGD and talk about number of batches vs occupancy / coverage. (instead of epochs)

3. In the results, authors use same number of epochs across different methods. However, when sampling, the number of epochs really loose their meaning. Instead, we can keep number of iterations the same . Because it is difficult to compare between, say, 10% and 20% sampling rate if the 20% is runs twice as many iterations. 

4. The argument that random sampling is competent for a lot of cases is a well accepted belief in the community. I am not sure if there has been a extensive study on the topic though. However, it should be noted that there will be cases where coresets will strongly outperform random sampling w.r.t accuracy and showing that random sampling beats coresets on a handful of datasets does not support using random sampling is better than coresets for training. In order to make a strong argument, i would suggest the following two directions - 

a. Use a lot of real datasets from different domains with learning and do an extensive study confirming what people believe. Having such a study would be very useful.
b. Else, maybe you want to make an argument in favor of random sampling even in adversarial cases. In such a case, hand design data distributions where random sampling is bad (so in terms of #iterations, coresets beat random sampling) . But in training scenarios, using the advantage of fast sampling random sampling is competent with coresets in terms of #seconds to converge. 

5. Also, it is not clear what the novelty of the paper is specifically because random sampling for training is the norm. It would be useful if authors would elaborate on the novelty. One interesting direction that i see in random sampling itself is this trade-off between soft-hard sampling. For example, the following two are natural extremes in random sampling, 
[softest] you pick each sample with replacement and form batches from these samples
[hardest] you shuffle the entire dataset and make batches from this shuffled dataset. This is also what is current practice.
Your approach is then somewhere in the middle, where you select a subset (which obviously has no repetitions) and then make batches from the subset. However the two different subset selections can have overlaps. 

You can analyse effect of this granularity that you introduce w.r.t occupancy, coverage and variance of gradient estimate for convergence (in theory and simulations) and in training convergence performance (# iterations required to converge). 


strength_and_weaknesses: The paper is well written. However, it has several weaknesses. Please see the points 2, 3, 4, 5
clarity,_quality,_novelty_and_reproducibility: Clarity in paper is very good. However the novelty seems limited see 5 above
summary_of_the_review: The major concern is that the paper does not provide clear evidence to the effect that random sampling should be always preferred in training neural networks.  See points above in summary
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method called "Soft Sampling" which is a data selection method for training of deep learning models. They also show some theoretical convergence guarantees and experimentally show that their method achieves similar accuracy as GradMatch (a SoTA method) while using less resources (as measured by wall-clock training time). The datasets used are CIFAR10 (image classification), Librispeech (audio to text), and their in-house Payload dataset (audio to text).

The way I understand it, Soft Sampling simply selects a random subset with replacement given the budget at each epoch, as such the subset is not pre-determined/fixed.
strength_and_weaknesses: Strengths:
- training time improvements on the baseline (GradMatch) are significant (e.g. when comparing GM_R1 and SS_R1 on CIFAR10)
- theoretical guarantees are provided

Weaknesses:
- the method name is very confusing. Looking at algorithm 1, I would simply call this method a variant of random sampling. I can understand that the "soft" part is trying to show that the data selection is not fixed (i.e. hard), so "soft random sampling" (with replacement) could be a better name
- given the above, I see no novelty in this method. Random sampling* is one of the first and most important baselines for data selection. As such, I'm surprised to see this being proposed as a novel method. The strength of this baseline has also been reported in multiple earlier works, for example, DeepCore (Chengcheng Guo et. al.) concludes: "... although various methods have advantages in certain experiment settings, random selection is still a strong baseline.". I believe this is a fatal flaw.


*: I understand that "random sampling" can refer to different but related methods: hard vs soft (or online vs offline) as well as with/without replacement.
clarity,_quality,_novelty_and_reproducibility: Clarity: I believe the method is clear, though the name is confusing.

Quality: Experiments seem sound and show clear improvement on multiple datasets, but only one baseline is used. Admittedly, I did not verify the theoretical proof (I saw little point in doing so).

Novelty: I see no novelty in this method.

Reproducibility: the experiments section mentions the hyper-parameters and the GPU used. However, no code seems to be available.
summary_of_the_review: Unless I'm missing something, which I'm hoping the authors can point out, the proposed method is not novel and has been one of the baselines that most data selection methods are always compared to. The strength of this baseline has also been observed many times before (e.g. DeepCore). As such, even though I believe the problem (data selection) to be highly important and knowing that I believe the method to be valuable, I must reject this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed efficient training of neural networks by selecting a subset uniformly at random with replacement from the full data set in each epoch. The authors derive a theoretical convergence guarantee for the proposed random sampling method on non-convex objective functions and provide convergence rates. They also evaluate the performance of the proposed method through experiments on a few datasets. I do not see the novelty of the proposed method to be significant, which is in-line with the reviewers' comment. Besides, it would be more useful if the authors evaluate the performance of their method on datasets with different characteristics (e.g. imbalanced classes), and characterize the use-cases where it works and the use-cases where it does not work. Hence, I do not recommend acceptance.
justification_for_why_not_higher_score: Lack of novelty and insights.
Decision: Reject