Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes Recency Bias, an adaptive mini batch selection method for training deep neural networks. To select informative minibatches for training, the proposed method maintains a fixed size sliding window of past model predictions for each data sample. At a given iteration, samples which have highly inconsistent predictions within the sliding window are added to the minibatch. The main contribution of this paper is the introduction of sliding window to remember past model predictions, as an improvement over the SOTA approach: Active Bias, which maintains a growing window of model predictions. Empirical studies are performed to show the superiority of Recency Bias over two SOTA  approaches. Results are shown on the task of (1) image classification from scratch and (2) image classification by fine-tuning pretrained networks. 

+ves:
+ The idea of using a sliding window over a growing window in active batch selection is  interesting.
+ Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty (sliding window instead of a growing window), the method has been well motivated by pointing out the limitations in SOTA methods. 
+ The results section is well structured. It's nice to see hyperparameter tuning results; and loss convergence graphs in various learning settings for each dataset. 

Concerns:
- The key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. Despite the paper stating that there have been earlier work (Joseph et al, 2019 and Wang et al, 2019) that attempt mini-batch selection, the paper does not compare with them. This is limiting. Further, since the proposed method is not specific to the domain of images, evaluating it on tasks other than image classification, such as text classification for instance, would have helped validate its applicability across domains. 

- Considering the limited results, a deeper analysis of the proposed method would have been nice. The idea of a sliding window over a growing window is a generic one, and there have been many efforts to theoretically analyze active learning over the last two decades. How does the proposed method fit in there? (For e.g., how does the expected model variance change in this setting?) Some form of theoretical/analytical reasoning behind the effectiveness of recency bias (which is missing) would provide greater insights to the community and facilitate further research in this direction. 

- The claim of 20.5% reduction in test error mentioned in the abstract has not been clearly addressed and pointed out in the results section of the paper. 

- On the same note, the results are not conclusively in favor of the proposed method, and only is marginally better than the competitors. Why does online batch perform consistently than the proposed method? There is no discussion of these 	inferences from the results.

- The results would have been more complete if results were shown in a setting where just recency bias is used without the use of the selection pressure parameter. In other words, an ablation study on the effect of the selection pressure parameter would have been very useful.

- How important is the warm-up phase to the proposed method? Considering the paper states that this is required to get good estimates of the quantization index of the samples, some ablation studies on reducing/increasing the warm-up phase and showing the results would have been useful to understand this.

- Fig 4: Why are there sharp dips periodically in all the graphs? What do these correspond to?

- The intuition behind the method is described well, however, the proposed method would have been really solidified if it were analysed in the context of a simple machine learning problem (such as logistic regression). As an example, verifying if the chosen minibatch samples are actually close to the decision boundary of a model (even if the model is very simple) would have helped analyze the proposed method well.  

Minor comments: 
* It would have been nice to see the relation between the effect of using recency bias and the difficulty of the task/dataset. 
* In the 2nd line in Introduction, it should be "deep networks" instead of "deep networks netowrks". 
* Since both tasks in the experiments are about image classification, it would be a little misleading to present them as "image classification" and "finetuning". A more informative way of titling them would be "image classification from scratch" and "image classification by finetuning".
* In Section 3.1, in the LHS of equation 3, it would be appropriate to use P(y_i/x_i; q) instead of P(y/x_i; q) since the former term was used in the paragraph. 

=====POST-REBUTTAL COMMENTS========
I thank the authors for the response and the efforts in the updated draft. Some of my queries were clarified. However, unfortunately, I still think more needs to be done to explain the consistency of the results and to study the generalizability of this work across datasets.  I retain my original decision for these reasons.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes an interesting heuristic of batch construction from samples. Instead of the usual random sampling, the authors to sample based on some measures of the ``uncertainty”. To be specific, the uncertainty is measured as a normalized entropy estimated from a window of historical predictions. 

I like the idea of designing more sophisticated ways to encourage more exploration over the samples that the model is not good at. The thought is similar as active learning. It is interesting to see how similar thought can be used to improve the performance of the algorithm in the general batch gradient descent setting.

On the other hand I am not quite convinced the proposed way is truly better. The main concern is the experiments do not quite show the state-of-the-art result at all. It is not even close on MNIST, CIFAR-10 and CIFAR-100. Also those datasets are relatively small one. Can authors add results on larger datasets such as tiny image net?

Besides this main concern I also have some worries about the design of the algorithm. I listed them below:
1. The vanilla stochastic gradient descent can be roughly justified since the expectation of the stochastic gradient is the true gradient of the loss. Now with the proposed heuristic will this still be true?
2. Is there any guarantee the algorithm can converge? It is not clear to me as the optimization proceeds the ``uncertainty” may oscillate. Is there any condition when the convergence is guaranteed?
3. As the number of classes grows the estimation of the entropy itself is a tough problem. Is there any way to mitigate this issue other than increase the window size?

Another minor comment:
Could the authors add more explanation on equation (4)? For example, $\delta$ is related to the maximum entropy led by a uniform distribution, and the summation term in (4) is related to the empirical entropy.
 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper explores a well motivated but very heuristic idea for selecting the next samples to train on for training deep learning models. This method relies on looking at the uncertainty of predictions of in the recent history of statements and preferring those instances that have a predictive uncertainty over the recent predictions.  This allows the training method to train on instances that are neither too hard nor too easy and focus on reducing the uncertainty whenever it has the greatest potential gain to do so.

There are two extra components that make this method work: 
- Windowing: only looking at the recent history of the instances which has two effects: firstly, the current state of the model is explored which gives a more recent assessment relative to the current state of the model. Secondly, it makes the algorithm faster by reducing the overhead of analyzing the prediction history of samples.
- Annealing the selection bias: as the training goes on the selection becomes more random and less biased.

This approach is evaluated in on three simple data-sets: MNIST, CIFAR-10 and CIFAR-100. Although this is a very limited subset of models, the results are consistent and statistically significant, although their effect is not really huge.

The paper gives very little theoretical justification or analysis of the results but gives only the presented empirical evidence which seems to support the hypothesis on the efficacy of the approach.

Another drawback of the approach is that it introduces new hyperparameters: those governing the annealing schedule for the selection bias.

Since the approach seems efficient in a relatively constrained setup, it can be reasonably expected that it might be helpful in more general situations, therefore. On the other hand, since it is only evaluated on three very similar tasks, it limits the conclusiveness of the results.

That's why I would for weak accept. In the presence of more empirical (or even theoretical) evidence, I would vote for strong accept.


review_title: Official Blind Review #3

Meta-review: The authors propose a new mini-batch selection method for training deep NNs. Rather than random sampling, selection is based on a sliding window of past model predictions for each sample and uncertainty about those samples. Results are presented on MNIST and CIFAR.

The reviewers agreed that this is an interesting idea which was clearly presented, but had concerns about the strength of the experimental results, which showed only a modest benefit on relatively simple datasets. In the rebuttal period, the authors added an ablation study and additional results on Tiny-ImageNet. However, the results on the new dataset seem very marginal, and R1 did not feel that all of their concerns were addressed. I’m inclined to agree that more work is required to prove the generalizability of this approach before it’s suitable for acceptance.

Decision: Reject