Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes an approach for updating predictions on an unlabeled dataset as new, pretrained classifiers become available. Calling it the "Prediction Update problem", the authors seek to 1) make updates which improve accuracy, 2) avoid updates which introduce new errors, and 3) update predictions on data points using a limited budget (relevant in the massive dataset setting). The authors propose a Bayesian approach in which a posterior belief over true labels for each sample are updated as new classifier predictions become available for that sample. The authors assume (label-conditional) independence of the classifiers, allowing the likelihood to be computed easily using normalized confusion matrices of the classifiers. Samples are chosen for prediction on the basis of maximum posterior entropy, and different prediction update strategies are considered. These include naive baselines (e.g., newest classifier's prediction and majority vote) as well as MAP estimates and cost-aware strategies (which tradeoff large belief increases against the possibility of incorrectly flipping a prediction). The authors extensively discuss limitations and opportunities for future improvements. Detailed experiments using imaging benchmark datasets examine the properties of different strategies on standard (i.e., train and test data from same dataset/distribution) and transfer learning (i.e., train and test data from different datasets) tasks.
main_review: This paper considers an interesting and, to my knowledge, new problem. Existing work on backwards compatibility in ML has focused on training new models whose predictions are consistent with the old model (as discussed by the authors in Section 5). Instead, this work considers the perspective of an owner of, e.g., a proprietary unlabeled dataset for which they need labels to provide downstream services. It is increasingly common for large, state-of-the-art pretrained models (e.g., winners of benchmark challenges) to be publicly released. This work is aimed towards those who want to capitalize on the increasing availability of high quality pretrained models to perform labeling on their own data. It is a very interesting use case and I'm glad to see the authors begin a thorough investigation here.

The proposed approach is a very straightforward, but elegant, Bayesian approach for tracking and updating posterior belief about the true labels in a backwards compatible way. The proposed framework lends itself to a number of natural extensions (e.g., synthesizing probabilistic predictions instead of classification predictions, new selection strategies, adaptive budgets, etc.). I thought Section 4 on extensions and limitations was very well thought out and it addressed a number of the minor questions/concerns that I thought of while reading the paper. Finally, I thought the experiments were well executed and were a convincing demonstration of the proposed approach. The experiment studying the order of the classifiers is particularly interesting and important---I think the authors should draw more attention to it since I doubt new classifiers will truly be strictly better than older ones.

I have a few suggestions for improvement:
- When first reading the introduction I was a little confused by the setting, and had a hard time imagining possible use cases. The authors first discuss the problem in general and only then introduce the photo tagging example on Line 79. I think this should be reversed: first provide the concrete motivating example and then explain the general problem/principles that will drive the method and the rest of the paper.
- Is there any way to test Assumption 1 or measure the magnitude of violations? In Section 4 the authors mention this assumption is a limitation, but could you provide suggestions or possible avenues for addressing it, measuring it, or performing sensitivity analysis?
- Is my understanding correct that no adjustments to the method were made for the ImageNet --> ObjectNet experiment? That is, the updates were performed assuming no distribution shift? I think it would be beneficial for the authors to expand their discussion in Section 4 of how distribution shift affects the prediction update problem (and also discuss beyond just the covariate shift case), since this is a problem that is going to come up a lot in practice.


UPDATE:
Thanks to the authors for their response. I remain very positive about this work and will be keeping my score the same. I think slightly more detailed discussion of Assumption 1 and distribution shifts in the main paper will help round it out nicely.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Paper proposes a new problem *Prediction Update Problem* with a complementary perspective originating from the increasing availability of pre-trained and regularly improving state-of-the-art models. In particular, they focus on (i) given a limited budget, deciding which data points should be re-evaluated using the new model; and (ii) if the new predictions differ from the current ones, deciding which predictions should be updated. The authors present an efficient probabilistic approach that answers both questions. Empirical evaluation shows the efficacy of their proposed approach over baselines. 
 
main_review: The paper introduces a very important problem and, in general, is very clearly written and is very easy to follow. The proposed method is simple and intuitive. Limitation of the current work is also presented succinctly. 

Comments: 
- Three desiderata are postulated in a very clear manner. Figure 1 illustrates the approach very clearly. 
- The proposed problem and solution are contrasted nicely with the related work. 
- Empirical results are strong in experiments across multiple common benchmark datasets. 

Limitations: 
- As mentioned by the authors, the assumption of conditionally independent classifiers is strong as usually the improvements are built on prior architectures. 

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper formulates the “Prediction Update Problem”: as new (better) models are trained, you want to integrate them into your system, and improve accuracy. However, you want to do so without making too many previously correct predictions incorrect (negative flips), as this would reduce user trust. The authors provide a natural Bayesian procedure for doing this. Additionally, they consider the case where the dataset may be too large to evaluate each model on the full dataset, and propose an entropy-based subset selection procedure. Experimentally, they show that this method can beat simple baselines at keeping negative flips low without losing too much in accuracy.
main_review: -	This problem seems very interesting and well motivated, and the authors describe it well
-	The proposed methods section provides a reasonable working-through of what a sensible, simple approach to this problem might be
-	There are a few assumptions that seem limiting (e.g. independent predictions given label, reliable estimation of confusion matrix) but the authors mostly do a good job mentioning these
-	The experiments tell a clear story although the presentation is quite cramped, it’s hard to really make out what is happening in the figures and charts. But the improvement over the baselines is clear
-	One thing that would be helpful in the experiments is more discussion of the Pareto front between accuracy and negative flips
-	The “limited evaluation budget” part of the story seems a little bit separate – in some sense I’m not sure you really need it as part of the proposed problem
-	BTC + BEC should definitely be defined in the main body!!!
-	The experiment with the models ordered by when the broke SOTA is very cool
-	One baseline I would find interesting would be using the oracle only for estimation of the confusion matrix – that seems like an important source of noise/uncertainty here
-	I’m not sure why the accuracies for the proposed methods decrease with higher budgets? There’s an explanation in the appendix but I don’t really understand it, would love more clarity here
-	Why are the numbers in the \sum NF column floats? Is this an average over seeds?
-	In general, lots of experimental details are left out, there should be a section in the appendix  - I doubt I would be able to reproduce this result as is given the information in the paper currently
-	Sec 6.4: when you discuss the ablations, should point to the appendix where you do the experiment
-	I have no idea what the “reducing re-evaluations matters at scale” section means – I thought you were using VGG/Resnets for your method??
-	Overall: the paper is a little sloppy but very interesting problem proposal and useful experimental evidence

limitations_and_societal_impact: Good discussion of the limitations of a number of their assumptions – Section 4 is very nice. I think this work could be useful across industries, in particular high stakes situations and human trust in AI systems
review_title: null

Meta-review: This work introduces a Bayesian approach to the prediction update problem. In it they consider an interesting use case in which they want to label a large unlabelled dataset while taking advantage of new pertained classifiers as they become available. Overall, reviewers were positive about the paper, but had several good suggestions when it comes to improving the presentation. I encourage the authors to carefully incorporate reviewer feedback at the revision stage.
Decision: Accept (Poster)