Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates the use of importance sampling in budgeted training.  Four importance sampling techniques from prior works are applied within the context of fixed training budgets, and compared under different conditions of training set selection, learning rate schedule and data augmentations.  Each aims to sample more useful examples more frequently, by using the loss or gradient magnitude as an importance measure.  Uniform sampling with and without replacement are used as baselines, and experiments are performed on cifar-10 and cifar-100.  The final conclusion is that importance sampling with budgets as low as 20% the original training schedule offer little if any improvement over uniform sampling, while additional data augmentations work well to make up lost validation accuracy.

While these are a thoroughly executed set of experiments on the cifar datasets, it's hard to know exactly what to make of negative results on just one dataset family.  Does importance sampling not work for budgeted training, or is cifar data not amenable to the technique?  Additional datasets or exploration of why it didn't work here would make this a lot stronger.

Similarly, I'm not sure exactly what to take away from the data augmentation experiment.  The conclusion states "in budgeted training ... data augmentation surpasses state-of-the-art importance sampling" --- but this seems to be the case in non-budgeted as well (SGD lines of table 3).  So is the conclusion that augmentation tends to work, for the training schedule lengths explored?

The budgets go down only to 20% of a 200-epoch training scheme (40 epochs), causing a baseline error increase from 5% to 8% in the worst case, still a fairly long schedule.  Evaluating smaller budgets, even down to near-chance performance level, might reveal areas with different behavior.

Overall, I find the experiments that were performed were well executed, and the negative result in this regime is a useful datapoint in assessing these methods.  Still, the experimental setups are a little too narrow (one dataset family, budgets starting at medium-length schedules) to draw any larger conclusions.


Additional comments:

- I feel the title is a little misleading, it implies that importance sampling is important, whereas the findings are the opposite.  Although maybe not technically wrong when read in hindsight (it merely raises the topic), I think it would make more sense if the title better reflected the findings, or was phrased as a question (eg "how important is ... ?"), which would leave a negative result more open

- related work section:  I think this it would also make sense to relate to few-shot learning, which is in a sense an extreme case of budget with just a handful of examples

- fig 1:  says scan-SGD but corresponding text in 4.4 says unif-SGD

review_title: well executed but narrow

final_score: rating: 3: Clear rejection
review_content: Overview:

The paper studies the effect of important sampling approaches in the context of budgeted training. They empirically show that important sampling does not provide consistent improvement over uniform sampling. Instead they find that the budgeted training benefits from variety in the sampled introduced by data augmentation.


Strengths:

++ Both the average accuracy and the standard deviation are reported across 3 runs. This makes the experiment results more statically convincing.

++ The literature survey on budgeted training as well as importance sampling is detailed and clear.


Weaknesses:

-- Inappropriate title. The paper argues that " under budge restrictions, importance sampling approaches do NOT provide a consistent improvement ..." and the useful part is the data augmentation. Then why the title is the importance of importance sampling (instead of data augmentation)?

-- The motivation of this paper is unclear.  The majority of the introduction looks like a duplicate related work to me. 

-- The paper is not well organized. For example, the reason for including core-set selection is unclear. In Sec 4, the authors conduct experiments by adapting importance sampling approaches in the setting budgeted training. Therefore, I don't see any necessity to spends two paragraphs in both introduction and related works on discussing core-set selections.

-- Flaw in the formula: For example, Eq (2) is wrong: If the prediction $ h^k_\theta $ is exactly the same as the gt $ y_i $, then $ p^t_i $ is being maximized but this is the easiest example (therefore $ p_i $ should have be minimized). Indeed, in Chang et. al. (2017), what they used was $ P_S(i \vert H, S_e, D) \propto 1 - \bar{p}(y_i \vert X_i) +\epsilon_D $. There is a negate in the front.
review_title: Reviewer #3

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Summary
This paper studies the interplay of budgeted training, data augmentation, learning rate schedules and importance sampling and finds data augmentation to play a key role.

# Score and reasons
Overall I vote for rejecting, but encourage the authors to extend their study to more learning rates and more datasets, as well as provide more statistics.
While I appreciate the idea of comparing the sampling methods and learning rates in a budgeted training setting, such a study should be more comprehensive or suggest a new method as a conclusion.


# Strong / Weak points
## Pros
This paper aims to connect important research directions, i.e. given a specific training budget, i.e. epochs, how can the best performance be achieved.
The paper provides an interesting comparison of the speed/accuracy trade-off of several methods.

## Cons
The paper is merely a comparison of existing methods, which is as such interesting, a broader range of datasets would be beneficial.
Stating to study the influence of learning rate schedules, while using only two such schedules is not enough. The authors should enhance their study with further examples, e.g. cyclic learning rate [https://arxiv.org/pdf/1506.01186.pdf]
Clarity of the presentation could be improved.
E.g. introduction and related work are not clearly separated
Much of the paper is spend on explaining contributions by others, while remains elusive what the novel contribution of this paper is.

# Questions to the authors
Please provide more details on the data augmentation you used, e.g. how many samples are shown to the network during training, including the variations through data augmentation.

How does data augmentation compare in terms of wall-time?

# Detailed comments
Abstract could be more concise.
Introduction is missing a common leitmotif connecting the paragraphs and seems to anticipate some content that would better be placed in the related work section.

Related work
The statements on curriculum learning are not complete.
Not all curriculum methods require pre-training a model, a simple curriculum is the focal-loss Lin, T., Goyal, P., Girshick, R.B., He, K., & Dollár, P. (2020). Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42, 318-327. [http://arxiv.org/pdf/1708.02002]. Methods like MentorNet jointly learn the curriculum from data, Jiang, L., Zhou, Z., Leung, T., Li, L., & Fei-Fei, L. (2018). MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. ICML. [http://proceedings.mlr.press/v80/jiang18c/jiang18c.pdf]

Methods for estimating the importance are also highly used in active learning, relating to that, e.g. margin sampling, ensemble variation etc. would be beneficial, for a survey of established methods see e.g. Settles, B. (2009). Active Learning Literature Survey. [http://axon.cs.byu.edu/~martinez/classes/778/Papers/settles.activelearning.pdf]

Section on learning rate schedules:  "have proven to be useful alternatives for faster convergence (Smith, 2017;
Smith & Topin, 2019; Li et al., 2020b). The authors in (Smith & Topin, 2019; Smith, 2017)" --> clutters the text to cite the very same papers twice in a row

Avoid vague language where possible, e.g. section 3: "DNN are usually trained" --> if you need that formula for further explanations state it as an assumption, if not, why have it at all? If they are only trained "usually" with a loss function, what are the alternatives?
DNN is missing an "s" here

Early stopping is probably not an optimal name, as this usually refers to the regularization method of "early-stopping", meaning terminating a training when the validation accuracy drops.

Deriving statistics from three runs seems to be not very informative, I would suggest to use a least 5 runs.

Highlighting the best results in your tables could guide the reader. 
review_title: Interesting comparison, that should be extended

final_score: rating: 4: Ok but not good enough - rejection
review_content: This experimental contribution concludes that importance sampling strategies do not improve training with budgeted constraints. Data Augmentation seems to be a better strategy in this case. 
The paper is easy to read and the experiments are clearly stated. 
Data augmentation and sampling strategies and two different technics. Comparing them in a jointly way seems not natural. Figure 1 showing the variability of the data may be a commun link between these two methods since they have some effect on entropy.
Table 3 shows that the effect of Data augmentation  is close for both SGD model and Sampling strategies. Why SGD-based accuracy is not reported for budget = 0.2 an 0.5? It seems to be important to validate that data augmentation is beneficial for both SGD and Sampling technics. 
Additional experiment should be achieved in largest and more challenging datasets (at least imagenet) to show if the conclusions are the same.  

The budget aware version proposed by Li and Al should be detailed and as in the paper, it should be interesting to report some training/val loss vs epoch to compare the convergence of the tested strategies
review_title: This paper studies how several importance sampling strategies jointly applied with data-augmentation technics impact the performances of deep budgeting training

Meta-review: This work investigates how importance sampling strategies can improve training with budgeted constraints., with a focus on the benefits from variety provided by data-augmentation samples.

Initial clarification issues raised by the reviewers were taken into account such as a new title, clarification of some explanations and corrections of typos.

However, the reviewers still agree that the paper is not ready for publication for several reasons:
- the comparison with the literature is still insufficient and should be better organised,
- experiments too narrow to conclude general benefits from the paper as it is, since there is a single type of tasks that is studied from the same dataset family. Questions related to very small budgets, below 20% also remain open and would require a new submission.
Decision: Reject