Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: - This paper tackles an important problem: Suppose we have trained a model on a dataset, but we wish to adapt it quickly. For example, we might add new training points, delete training points, or wish to change the model and regularization. We could re-train a model, but that's slow - can we get a similar result but quicker and using less memory?
- This paper proposes K-prior: regularizing the model towards the original model in both weight space and predictions.
- They show that training on the K-prior objective on the entire dataset recovers the original model - which would not be true if we only have weight space regularization, or only have prediction regularization.
- In practice, to save memory, they only form the prior on a few points, and they describe heuristics to choose these points, with some mathematical justification
- They show that this methods work well in practice, compared to just replaying the points 

main_review: After rebuttal: added comments below

---------------------------------------------------------------------------------------
Strengths
- The paper is well written. It seems well executed, technically sound, and reports results honestly and thoroughly.
- The idea of K-priors is nice - it's pretty interesting that if you keep both the weight space regularization and prediction space regularization, then you can recover the original model. The math is simple, but I tend to view that as a strength
- Experimental results on UCI adult, MNIST, USPS, and CIFAR-10, show promise. It's interesting that the K-prior method does better than replaying the loss on the same set of points they use for the K-prior regularization. They also include ablations on using only the weight regularization.

Questions and suggestions for improvement:
- It would be more compelling if they include baselines from other papers. They say that prior methods focus on specific tasks, like adding points, or removing points, and don't do all the adaptations they do. However, it would still be nice to see how K-priors compare with methods specialized for adding new points. I'm not too familiar with the area or relevant baselines though.
- The paper says that the main difference between "Replay and K-priors is that the former uses the true label while K-priors use model-predictions." - does replay also add weight regularization besides replaying the loss?
- Just to confirm, what was the tuning protocol to choose tau, the weighting between the weight and prediction regularization?
- I think it would be good to discuss in more detail why K-priors do better than just replaying the loss. I believe replaying the loss would satisfy similar guarantees as K-priors? For example, in equation 10 and Theorem 1, if we replaced K(w) by sum_{i \in D} l_i(w) + R(w). So what's the benefit of K-priors?
- It seems like one benefit of K-priors could be that we get more (bits of) information from the model prediction than from the label, similar to in distillation, since the model prediction is a probability vector but the label is just one of C values, where C is the number of classes. The paper could be stronger if it explored these ideas and compared K-priors with replays.
- Their method of selecting points, for linear models, seems to roughly correspond to choosing points close to the margin. Is that correct?
- The experiments could be stronger if they did some grid search of hyperparameters for each method. For example, maybe weight priors need a stronger weight because they don't have the prediction space regularization? So ideally if they do the same grid search of hyperparameters for each method, and then compare them, it would be more convincing.

nit: In line 301, a neural network with 1 hidden layer is not deep, so remove deep :)

Overall, I think it's a nice idea, that's well motivated, and fairly well executed, so would be good to have in the conference!

limitations_and_societal_impact: Seems fine to me!
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a method called K-priors for adapting a model to newly coming data,  new regularization, or new architecture.  It suggests such adaption can be done by minimizing the Bregman divergence between features generated by previous model and current model,  as well as minimizing the Bregman divergence between parameters of previous model and current model. 
main_review: The novelty is incremental, as the form of Eq.7 is just a combination of knowledge distillation loss and a weight regularization, both of which are common in continual learning literature.  
And the claims are not supported by the definition of the method.  According to Eq.9, only when M=D the gradient can recover the gradient of previous loss, which is not surprising at all. However, authors claim that data out of the previous distribution can be used to approximate the previous gradient (Line 133), how can this be guaranteed?
Also, the suggested way to select memorized samples is cumbersome and there are existing approaches proposed for sample selection in continual learning as well, e.g. [1],[2].  The authors didn't discuss or compare with any such related work.  And how far this approximation can be to the previous gradient is not analyzied either. So, how to achieve the claim of 'recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data'?  For a specified accuracy, how large the memory should be?

[1]. Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., and Page-Caccia, L. Online Continual Learning with Maximal Interfered Retrieval. In Advances in Neural Information Processing Systems, pp. 11849–11860, 2019a.
[2]. Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. Gradient-based Sample Selection for Online Continual Learning. In Advances in Neural Information Processing Systems, pp. 11816–11825, 2019b.
limitations_and_societal_impact: Limitations are stated above. No societal impact in my understanding.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new regularizer to adapt the prior knowledge through weight prior and the memory that is smaller than the entire dataset but identical to be learned correctly. Previous methods to do this have focused on the "Add/Remove Data" task, but the authors extended the adaptation tasks to adapt regularizer change or architecture change through the proposed regularizer, K-priors. It also could be applied to a variety of models from a linear model to deep learning. They described the connections between K-priors and previous works for adding/removing data for SVMs, knowledge distillation and continual learning for deep learning, and Bayesian learning. They mainly validated that K-priors outperform the model without K-priors but using the same memory selected through K-priors for logistic regression and classification tasks. 
main_review: The authors tried to contribute the unified regularizer that is based on the previous knowledge (when there is no previous knowledge, it would be used like$l_2-\text{norm}$ weight regularizer). The previous knowledge exists as samples or trained parameters. They tried to use the two type knowledge with their prior. This paper has several strengths, one of them is that they tried to design the general prior that is not limited in deep learning, which is rare in those days (I don't think research on deep learning is not good, but diversity on research will make our community healthy more). Another is that they described the connections with previous works like knowledge distillation. However, the validations are on small size of experimental setting, and I am not sure about the stability of this prior. Another concern is about forgetting issue when updating the model multiple times. The memory is reset in every time, then the identical samples that are stored in memory will be removed and the model can forget that. However, it could be overcame through diving the entire memory to a sub-memory for each update I think. I agree too that the experiment for CIFAR-10 knowledge distillation (the dip in the middle in Fig 3 (d)) must be updated or analyzed more.

Questions
- In line 64-65, how $k$-fold Cross-validation is related with adaptation task?

limitations_and_societal_impact: As I mentioned in the previous section, stability could be a limitation of this research, but I am not sure.

When applying it to industrial or real applications, it sustains some samples as a memory, the right to be forgotten for the information on the internet may be infringed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes K-priors, giving a family of approaches aiming to quickly update trained networks to tasks changed in various ways (with added data, removed data, a changed regularizer, or a changed architecture). The main features of K-priors are: they approximate (or in some cases compute exactly) gradients of the original objective using the outputs of a trained model instead of the data labels; and they can approximate the original objective well with only a small fraction of the original data.
main_review: # Novelty
The difference between the K-priors objective in Eq. 10 and full retraining is fairly small - as stated in line 130, it is that model outputs (with old weights) $h(f_{w^*}^i)$ are used in place of data labels $y_i$. This is similar to Knowledge Distillation [22] (cited in paper), as noted in the paper. The "memorable past examples" approach of [37] (cited in paper) is combined with this to reduce memory requirements - as far as I know, such a combination is novel.

# Strengths
- The paper is well-motivated, and I believe that reducing the cost and environmental impact of retraining models is an important line of work.
- Equation 9 is an interesting result, showing that their approach exactly recovers the original gradients for generalized linear models.
- The experimental results show significant improvements with using K-priors over 'Replay' when only a small amount of past data is kept.

# Weaknesses
- (EDIT: The authors have presented results satisfying me that updating weights using K-priors can indeed be significantly faster than retraining.) A major aspect of the motivation for this work is to reduce the time/energy consumption for retraining, but this is not demonstrated anywhere in the experiments (only a reduction in the number of examples that need to be stored in memory is demonstrated). If this is not to be included then, at the very least, the abstract should not claim that the empirical results "confirm that the adaptation can be cheap and accurate". To me, "cheap" implies that the training time/energy consumption is reduced.
- It is difficult to interpet the performance on the "Remove old data" task by looking at the plots of validation accuracy - surely very good validation accuracy could be achieved by simply keeping the old weights and not removing any data.
- Despite the response to 3a on the checklist, there does not seem to be any code included with the submission.

# Writing
- The writing is mostly of a good standard, but I think the novel contributions should be more clearly stated.
- When performing adaptation, could the authors confirm whether weights are randomly initialised or initialised from the previously learned values?
- The introduction of Eq. 4 is confusing - is $y$ a scalar, a one-hot encoding, or other vector? Similarly for $f$, is it a scalar, vector, etc. (i.e. what is $\mathcal{F}$)?
- (EDIT: I am no longer concerned about this point, and now understand that the objective suggested for each task are unified by being something like "the simplest objectives which recover/approximate the gradients of the desired new objective without using data labels".) It is a little unclear what unifies the different objectives presented as K-priors - e.g. Equation 10 for adding data, in which $\mathcal{K}(w)$ has the same gradients as the old objective, and Equation 12 for changing the regulariser, in which $\mathcal{K}(w)$ is modified based  on the new regulariser, and then used on its own as the new objective. Then in equation 19 for changing the model class, $\mathcal{K}(\theta)$ is used as the new objective, and no longer contains a weight-space divergence. It would be helpful if the authors could be clearer about that unifies these, and distinguishes them from prior work.

EDIT SUMMARY: See bullet point 1 under "Weaknesses" and bullet point 4 under "Writing".
limitations_and_societal_impact: The authors claim, I think reasonably, that this work has little obvious potential for negative societal impact.

EDIT: Removed sentence regarding concerns about reduction in training time.
review_title: null

Meta-review: The paper presents a family of approaches for adapting pre-trained models to a variety of changes to the model architecture, training data or other aspects of the training setup. Given the cost of training models from scratch and the generality of the presented approach, this is a highly relevant piece of work with potential for impact.

After the discussion, three reviews recommend acceptance and one review recommends rejection. The negative review points to certain issues with the current version, but doesn't convince me that these issues rise to the level of rejection. After considering both positive and negative points, I lean towards recommending acceptance.

**Comments to the authors**: After reading the paper, I was left with the impression that clarity could be improved, and that there are places where details are missing. I see that some of these missing details and some additional experiments were supplied during the discussion with the reviewers. I would like to see these, together with any other reviewer feedback, incorporated into the final version of the paper.
Decision: Accept (Poster)