Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work presents an interesting exploration of learning optimal data sampling probability. It is formulated with a population based training strategy. To this end, authors propose to record the data frequencies through all previous steps and thereby generate the sampling policy for the next step.

It has been a recent trend to make almost all key factors of deep networks learnable, such as differentiable data augmentation. This work is taking a novel aspect for effectively training neural networks, since rare effort has been devoted to make data sampling learnable. Authors frame the task in PBT and take incremental modification to PBT to adapt to the new problem. I have no complaint regarding the task per se. However, the empirical validation seems finished in rush and not sufficient. 

The chosen baselines are essentially standard sampling scheme or variants of the proposed method. Authors should compare with a few state-of-the-art data-sampling or data-reweighting methods, such as Focal Loss proposed by He et al. In addition, Tables 2 and 3 should be clarified in the rebuttal. It seems that very small margins are observed in comparison with random exploration. The follow-up analysis attributes the success to the emphasis of hard examples during training, which is consistent to previous common practice. However, authors are encouraged to go deeper for the analysis, beyond the selected illustrative samples in Figure 3. For example, it may be meaningful to check the alignment of data weights and classification scores.

review_title: a work that explores data-sampling scheme in PDT framework

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors mainly concentrate on data sampling. To address the issue of optimizing high-dimensional sampling hyper-parameter in data sampling and release the requirement of prior knowledge from current methods, the authors introduce a searching-based method named AutoSampling. This method is comprised of exploration step and exploitation step which are conducted alternatively. The exploitation step train multi child models with current sampling strategy and save the best model for next iteration. while the exploration step estimates the sampling distribution according to the sampled data in exploitation step and rectifies it to sample all data possibly. The authors have conducted sufficient experiments to verify the superior of their method, especially for the effectiveness and generalizability. 

Advantages:
l	The exploitation step and exploration step in AutoSampling is interesting, it is straightforward that this method can work well as the sampling strategy is updated dynamically according to the current state of model.
l	The proposed AutoSampling is simple and effective, one can implement it without much effort.
l	This method owns great generalizability and does not require any knowledge prior.
l	This paper is well organized and written.

Disadvantages:
l	In Table 1, the number of workers does have an influence on performance, and this influence is positively correlated in my opinion, however, we can see a performance degradation for DenseNet121. The authors did not explain this.
l	The transferability of the gained optimal sampling schedule is discussed in Section 4.4, a simple experiment is recommended. 


review_title: A paper marginally above average

final_score: rating: 3: Clear rejection
review_content: This paper discusses an approach to perform importance sampling by reviewing performance over past batches and suggesting future batches.

First I will comment on the listed contributions:

> • To our best knowledge, we are the first to propose to directly learn a robust sampling
schedule from the data themselves without any human prior or condition on the dataset.

From this description I believe this paper has done it before: "CASED: Curriculum Adaptive Sampling for Extreme Data Imbalance". 

The paper seems to go beyond this and focuses on multi node training so this may be a way to refocus the paper in contrast to the existing work.

> • We propose the AutoSampling method to handle the optimization difficulty due to the high
dimension of sampling schedules, and efficiently learn a robust sampling schedule through
shortened reward collection cycle and online update of the sampling schedule.

It is not clear to me what the intuition of P(x) is. If H* is the "optimal sampling schedule" and then P is defined to be the ratio of x in H* over all x in H* then it is not clear to me what you are converging to. It would be better for the reader to make this as easy to understand as possible. Also, I think this would be the place where this approach is different from the CASED approach.

On the experiments to demonstrate that this method is better: The replicate runs seem to almost have no variance. It is the case that only the model init was varied between runs with fixed train/valid/test sets? If so this does not seem sufficient to confirm that this method works because the data you sample from is always the same. This should be randomized at least for CIFAR.

Typo: "Algorithm 2: Search based AuoSampling" 

review_title: Unclear novelty and method, as well as inconclusive experimental results.

Meta-review: The work focuses on a new method for sampling hyper-parameter based on an "Population-Based Training" schedule that tend to sample more often configurations that gave good performances in the past. The authors have conducted experiments to verify the superior of their method, especially for the effectiveness and generalisability.

Pros:
- simple method that can be implemented without much effort,
- good empirical performances on Imagenet,
- paper well organised and written.

Cons:
- lack of explanation about the DensNet121 performance degradation [partially addressed in the rebuttal],
- additional simple experiments in Section 4.4 were recommended to evaluate the generality of the method [addressed in Table 5],
- empirical validation seems not sufficient [partially addressed in the rebuttal],
- similarity with respect to prior art, such as the focal loss [partially addressed in the rebuttal],
- clarification of the randomisation strategy in experiments [addressed in the rebuttal].

Despite most of the issues being addressed, the reviewers decided that this paper would benefit more work to be accepted for the conference this year.
Decision: Reject