Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Paper Summary:

This paper proposes a new method that uses uncertainty estimation to do ensemble active learning on image classification tasks. The proposed method mainly consists of two steps: 
1. select a subset of data based on the uncertainty estimation from an ensemble model. The ranking of data point is calculated via an acquisition function, which measures the model uncertainty.
2. train a “subset model” on the selected subset.
The paper then considers 3 initialization methods, 4 acquisition functions and 4 ensemble configurations and makes an empirical study on different combinations of these three. The experiment is done progressively. The paper first fixes the acquisition function to Mutual Information and finds the build-up initialization most promising. It then fixes the build-up initialization and finds Variation Ratios and Mutual Information result in better performance. Finally, with build-up initialization and Mutual Information acquisition, the paper compares different ensemble configurations and shows checkpoints ensemble can scale up better (less computation burden) and result in better performance when the number of ensembles is large. In addition, the paper also shows the obtained subset can help models with richer capacity as well. Results on deeper architectures outperform their corresponding performance when trained on the entire training set.

Strengths:
- The idea of using checkpoints as free samples of models is new and practical.
- Experiments are comprehensive and convincing in terms of the performance.

Concerns:
- Although saving checkpoints is “cheap” and shows empirical good performance, the models are somehow dependent on each other, particularly in experiments where consecutive checkpoints are saved. I am not sure whether this fits well in the bayesian framework of uncertainty estimation.
- The build-up initialization method lacks details for reproducibility if the Algorithm 1 of Chitta et al., 2018a is used.
- It is interesting that large number of ensembles increases the accuracy of acquisition model by a lot, but doesn’t boost the performance of the subset model too much (according to table 2 and 3). Does this imply that it is the ensemble rather than the active learning that helps?

Minor issue:
- It might be better to clarify which ensemble is used in the ensemble configuration experiment since previous experiments can have two different ensembles (acquisition and subset).
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: Review Summary
--------------
Overall, I'm not quite convinced this method would be worth the trouble to implement. On the more realistic benchmarks, they need to keep ~80% of the total dataset size and the claimed "improvement" is rather small (less than 0.6% absolute gain in accuracy, e.g. from 81.86% to 82.37% on CIFAR100 and from 72.33% to 72.78% on ImageNet). There is no runtime comparison, there are missing baselines, and most of the method development seems guided by trying out many options instead of taking a principled approach. Without these, the paper is just not ready for a top conference like ICLR.

Paper Summary
-------------
The paper considers a new take on active learning for image classification: given a large fully labeled dataset, identify a subset of the data that, when training on that subset alone, yields similar performance as training on the (much larger) full dataset. The paper focuses on "ensembles" of deep neural network classifiers as the prediction model, following Lakshminarayanan et al. (2017). 

The presented method is summarized in Algorithm 1. Given a suitably initialized "acquisition model", the model makes predictions on each example in the full dataset, then ranks examples using an acquisition function to find the subset of size N_s (top N_s examples by rank) where there is most "disagreement" among the model ensemble. This subset is then used to train a "subset" model (again, an ensemble of DNNs).

Experiments consider several possible initializations, acquisition functions, and ensemble sizes. Evaluation is done using the validation sets of three prominent image classification benchmarks: CIFAR10, CIFAR100, and ImageNet (1000 classes). 


Significance
------------
I don't think a successful case has been made that the proposed solution would generate significant widespread interest, because the gains demonstrated here are too minimal. Looking at the primary results in Table 2, it's really only when using 80% of the total images of imagenet or cifar100 (the most realistic benchmarks) that there is a small (<1%) absolute gain in accuracy over the simpler approach of just using the full dataset. Thus, the approach is not going to significantly reduce computational burden but adds a lot of complexity.

Novelty
-----------
The method seems new to me.


Experimental Concerns
---------------------
## E1: Need to consider runtime in evaluation

None of the figures/tables that I can see report elapsed runtimes for the different methods. To me this is the fundamental tradeoff: not how many fewer examples can I learn from, but how much faster is the method than the "standard" of using the full dataset? Showing curves of validation progress over wallclock time would be a better way to present results. 

The important thing here is that even *full dataset* makes progress after each minibatch. You'd need to show progress at checkpoints for each epoch in  0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, .... 

## E2: Potential missing baseline: Random subset with balanced classes

When you select a random subset, are you ensuring class balance? If not, that seems like a more refined baseline. Perhaps won't make too much difference for cifar10, but could be important for ensuring rarer classes in ImageNet are represented well. 


Presentation Concerns
---------------------

## P1: Initialization description confusing

I didn't understand the "build up" method as described in Sec. 3. How large is the subset used at each phase? How do you know when to stop? This could use a rewrite to improve clarity.

## P2: Missing some details for reproducibility

How was convergence assessed for all models? How were learning rates set? Many of these are crucial to understanding the runtime required for different models. (Sorry if these are in the appendix, but some short summary is needed in the main paper)

## P3: Title Change Recommended

I don't think the presented method is really doing a "Distribution Search"... I would suggest "Training Data Subset Selection with Ensemble Active Learning"

Minor Method Concerns
---------------------

## M1: What about regression?

Acquisition functions seem specialized to classification. What to do for regression or structure learning? Any general principles to recommend?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This work makes use of uncertainty estimation methods from active learning to select a subset of training data that produces models with similar (or better) performance compared to models trained on the full training set. It proposes a way to improve the Monte Carlo estimation of model uncertainty by including multiple checkpoints that are generated "for free" during a training run, thereby increasing the number of samples from 5-10 in previous work to 100 in this work. It compares several initialization schemes for the subset model using mutual information as the acquisition function, finds that a "build-up" approach (based on Chitta et. al 2018a) works best, and uses that for the rest of the studies. It then compares several acquisition functions, using the build-up approach, finds that variation ratio performs best, and uses that for the rest of the studies. Next, it compares the Top-1 accuracy on ImageNet obtained by evaluating the ensemble models produced by different ensembling schemes, and finds that ensembling 20 checkpoints from 5 training runs with different random seeds work best. Then, it uses acquisition models that use ensembles from each ensembling scheme to select subsets of the ImageNet data to be used for training the subset model, and then compares the performance of the subset models. Finally, it demonstrates this method of selecting a subset of the training data works even if the subset is used to train a model with a different architecture from the acquisition model.

Strengths:
- Algorithm is likely to be useful in practice. Training dataset can be "compressed" using a smaller architecture like ResNet-18, then used to train larger architectures like DenseNet-121, thus saving the amount of compute per training epoch. Using training checkpoints in the ensemble is very practical but not obvious (to me), since my first intuition would be that checkpoints from the same training run would not provide enough diversity to improve the acquisition function. I am glad that there were thorough experiments to address this concern and demonstrate that it works.
- Experiments answer key questions about the method proposed, and the sequence of experiments have a clear logical flow. Good baselines. Clear notation and problem set-up.

Weakness that affected the score:
- Missing detail on the build-up initialization scheme. The work referred to Chitta et al., 2018a, but that algorithm requires selecting a growth parameter. This growth parameter determines the number of times the subset model needs to be retrained, which can affect the viability of this method in practice. I would like to see the build-up initialization scheme described in greater detail.

Clarifications:
- In Table 2 and Table 3, are the results in the "Single (1)",  "Checkpoints (5)", and "Checkpoints (20)" columns obtained by averaging over the 5 random seeds?
- In the last column of Table 2, Top-1 accuracy of ~84% from an ensemble of 100 ResNet-18s (Table 2) seem very high. In comparison, ResNet-50 and AmoebaNet-A (2019) obtained a Top-1 accuracy of 77.2% and 83.9% respectively. What do the authors think about this?
- Why would one expect high accuracy of the ensemble of NNs in the acquisition model to indicate good sampling quality of the acquisition model? (caption for table 2)

Minor issues:
- Algorithm 1: first two steps should be kept un-italicized, like the rest of the steps.
- Page 7, first paragraph: "This shows that the checkpoints are obtained with no additional computational cost at train time can be used to generate diverse ensembles." The first "are" in this sentence is unnecessary.
- Build-up was chosen as the initialization scheme for the rest of the studies, as it performed best when the acquisition function was fixed at *mutual information*. However, the acquisition function that was finally chosen for the rest of the studies is *variation ratio*, since it performed best when the initialization scheme was fixed at build-up. It would be more convincing if figure 1 also includes variation ratio.
review_title: Official Blind Review #3

Meta-review: This paper proposes an ensemble-based active learning approach to select a subset of training data that yields the same or better performance. The proposed method is rather heuristic and lacks novel technical contribution that we expect for top ML conferences. No theoretical justification is provided to argue why the proposed method works. Additional studied are needed to fully convincingly demonstrate the benefit of the proposed method in terms computational cost. 
Decision: Reject