Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to use snapshot ensemble (SE) to replace deep ensemble (DE) for modeling uncertainty in active learning. SE takes predictions from different epochs of training of a single model and thus is more efficient than DE. The authors compare SE with DE and MC-dropout on several basic AL acquisition functions including Variation Ratio, Max-entropy and BALD, and show that SE is comparable. They also use SE with continual learning where the model is not trained from scratch but is continuously finetuned after each acquisition. Such training strategy achieves comparative result with vanilla AL when using VR as acquisition.
strength_and_weaknesses: Strength

- The authors propose a simple modification of ensemble-based AL learning by using Snapshot ensemble and continual learning, which achieves comparable or better result compared to deep ensemble while reducing the computational cost, when used with simple uncertainty-based acquisition functions. 
- The finding that snapshot ensemble produces more diverse predictions than deep ensemble is interesting. 

Weakness

- The novelty of the work is limited, as both the snapshot ensemble and continual active learning have been introduced previously, and there is no novel acquisition function or AL algorithm. 
- The authors only consider very basic AL methods (VR, Entropy, BALD) which are relatively old and has shown to be inferior than recent SOTAs. Moreover, the empirical experiments are incomplete, with full comparison among MC-dropout, DE, SE and continual-SE only available for VR, making the conclusion on superiority of SE unconvincing. 
- The authors use a batched AL setting; however, the chosen acquisition functions are not suitable for batched AL as it just greedily takes the top m examples (in fact BALD can be worse than random in batched setting). It would be more compelling if result on algorithms designed for Batched AL is shown.
- The claim that prediction diversity is more important than model performance could be misleading, as overly diverse predictions could result in under-confidence, making choices of samples almost random. The right amount of uncertainty should be calibrated and informative of actual model uncertainty. No theoretical analysis is provided on why SE always produce better diversity. 

clarity,_quality,_novelty_and_reproducibility: The paper is overall clear and easy to follow. The major limitation is on the novelty and the comprehensiveness of the empirical results. 
summary_of_the_review: The paper proposes a simple modification to ensemble-based AL by relacing DE with SE and shows some preliminary results using basic AL strategies. Although the result on VR is promising, more thorough comparison with recent AL techniques is needed to prove the significance of the result.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a simple idea of using the snapshot ensembles instead of deep ensembles for active learning for uncertainty based methods. The paper compared their method with traditional DE and MCDO on three different dataset (CIFAR and Tiny ImageNet) with three differenty acquisition functions and showed that the SE + FT method achieved both a high accuracy as well as a low computation requirement.
strength_and_weaknesses: Strength:

The idea is simple. Combining SE to AL is not a new idea as I believe a lot of AL practitioners (like myself) training multiple models (DE) would think of possibly training less, like using FT. The problem is majority of us stop short when we see the fine-tuned accuracy is much lower than retraining from scratch, not realizing that the final accuracy is what we want....

Just like the title, this simple idea is powerful as demonstrated in the experiments. The paper is well-written and easy to understand. The details are sufficient for the readers to follow and reproduce the results.

Also, just want to point out that Figure 1 is crucial for helping the reader as it conveys an important point that before the final episode, the model is at low accuracy. I couldn't understand why other fine-tuning efforts failed until I see Figure 1.

Weakness / Suggestions:
The final analysis part is a bit weak (possibly due to the limitation of article length and the desire to squeeze things into 9 pages). I would suggest prioritizing Fig 2 and put Fig 3 (and more analyses like this) into the appendix.
The table 1 and 2 might require some caption to explain the 10%, 15%, 20% meaning. 
best-performing results should be highlighted in Tables
Minor typo on page 4: "are to acquire acquire samples"

clarity,_quality,_novelty_and_reproducibility: The paper is clear, and novel in the sense that it points out SE should be carried out with the final few episodes retrained from scratch. Reproducibility is good with enough details disclosed.
summary_of_the_review: I think this is a very interesting paper that made a simple idea that other people have tried actually working. The experiments are sound and the clarity is good, overall a good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a more time efficient active learning method based upon epistemic uncertainty by incorporating snapshot ensembles rather than using the traditional method of deep ensembles. The experimental results demonstrate comparable recognition performance with a significantly lower runtime.
strength_and_weaknesses: The strength of the paper is that is well written, presents a reasonable somewhat novel approach for active learning, and the experimental results do show the potential of the approach.  The weakness of the paper is that both the deep ensemble and snapshot ensemble approaches are controlled by many various hyperparameters (e.g. number of samples, burn in time N_thresh, jumps between snapshots J, etc.).  These particular hyperparameters do not seemed to be discussed in the context of any hyperparameter search method in the paper or the appendices.  Choosing these hyperparameters judiciously seems important to appreciate the reduction in runtime. Also, the SE with fine tuning seems to be a significant advance over SE, but it is not fully evaluated in the experimental results.  It seems that the authors ran out of time to run the experiments, and the results are incomplete. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the approach has novelty and the significant details of the approach are presented so that the paper appears reproducible.

I may quibble that some of the maximum entropy and variation ratio are not selecting the samples with highest epistemic uncertainty. In these cases, this reviewer believes these are estimates of aleatoric uncertainty while BALD is indeed a measure of epistemic uncertainty.  Nevertheless, these issues are not central to the main result of the paper which is the use of SE rather than DE.

This reviewer would like to see more assessment of the reduction in run-time via SE+FT over SE. This reviewer would like to see some discussion of selection of the SE specific hyperparameters. 

It also seems that what really matters is the runtime to select the next m samples to label rather than the total runtime?  What really matter is that this runtime is comparable (less than) the time for the annotators to actually label the batch of m samples.

In the bottom paragraph of page 6, SE and DE are discussed in relation to baselines.  What are these baselines?

The table captions should explain what is being shown.  Are the results the probability of correct classification?  Are the percentages in the column header the percentage of unlabeled data that are eventually labeled?

I could not see where Figure 1 is referenced in the narrative. 

I did not follow Figure 3.  It seems that the upper left in lowest VR and highest VR scores is equally confused between cat and dog for all 5 snapshots. In that case, it would seem that max p would be comparable.  If so, why the discrepancy between VR scores. I am also not clear what the probability map actually represents.

Nevertheless, I do like how the paper tries to show how the SE method provides ensembles with more diverse opinions.

It would seem that MCMC methods such as Langevin dynamics would provide more diverse ensembles in a single pass than SE methods.  Perhaps some discussion of why this might not be the case or if this should be explored as future work would seem appropriate.



summary_of_the_review: The work appears novel and the experiments demonstrate the effectiveness of the method. The results seem somewhat incomplete at the SE+FT seems like the significant results that could be explored more.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper combines two ideas: snapshot ensembles and sampling-based (uncertainty and mutual information) active learning. Snapshot ensembles are shown to empirically perform better than standard (deep) ensembles and Monte Carlo dropout, when used in conjunction with uncertainty sampling (and BALD). Furthermore, warm-starting (instead of training from scratch) is shown to work well empirically for data collection with active learning as long as the model is evaluated with a model trained from scratch.
strength_and_weaknesses: Strengths:
 - The combination of techniques used in the experiments dramatically improves runtime with equal or better runtime.
 - The observation that warm-starting works for data selection (though not for the final model) is very interesting and important.
 - This paper's combination of recent ideas in parameter sampling with old ideas in active learning is very timely.


Weaknesses:
 - It seems that some rows are missing for the experiments. For example, for CIFAR10, BALD with SE+FT, ME with MCDO, and ME with SE+FT are missing. Furthermore, it appears that MCDO and SE+FT are missing for the pre-trained models.
 - I'm curious why margin uncertainty sampling (best vs second best or BvSB) is not used. In the literature, this method usually outperforms entropy uncertrainty sampling.

clarity,_quality,_novelty_and_reproducibility: - This paper is quite clear.
- Figure 3 is a bit hard for me to understand. Perhaps cut it, or add more explanation.
- The right side of Figure 2 is a bit mysterious. What are the percentage values in the bottom?
- The experimental work is original, though all techniques appear in previous work (not necessarily in active learning).
summary_of_the_review: While "only" combining existing ideas, this paper experimentally finds two very practically useful observations that I think the research community will appreciate.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a deep active learning method where the sampling is based on uncertainty estimates based on parameter "snapshots" from a single optimization path. The authors show that this method is effective, using a single learning trajectory for the active learning episodes as opposed to having to retrain the network. This method solves an important problem, relevant in the continual learning setting. Previous methods do not maintain a single model, use it to sample and successfully train it continuously, whereas previous attempts at this failed - the authors pointed this out in addressing reviewer WKSq's comments about novelty. The reviewer did not reply to the author's explanations of the differences to prior work, however, their explanations make sense to me. I urge the authors to make these differences to prior work clearer in the camera ready. Reviewer qtq1 pointed out that the analysis part in the main paper should be expanded. I fully agree with this, and their suggestions about moving the figures, because the method itself is straightforward (albeit effective), so gaining some insight into why it works is especially important. The authors have also addressed the questions of reviewer s1VV about hyper parameter tuning, learning rate and runtime, and performed additional experiments, as requested.
To summarize, this paper presents a simple method to perform active learning and train a deep model continuously, demonstrating its effectiveness on several datasets.
justification_for_why_not_higher_score: More theoretical results would have been needed to rate the paper higher.
justification_for_why_not_lower_score: As the title suggests, the method is simple and useful, the claim made were verified experimentally.
Decision: Accept: poster