Knowledge-Adaptation Priors
Authors: Mohammad Emtiyaz Khan
Siddharth Swaroop
Venue: NeurIPS 2021 Poster (2021)
Keywords: Adaptation
Bayesian methods
Priors
Online learning

Abstract: Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.

Knowledge-Adaptation Priors
Mohammad Emtiyaz Khan"
Siddharth Swaroop"
RIKEN Center for Al Project
University of Cambridge Cambridge, UK
Tokyo, Japan emtiyaz. k iken. jp
ss21630cam ac uk
Abstract
Humans and animals have natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require complete retraining from scratch. We present Knowledge-adaatation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for wide- variety of tasks and models. This made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but ceamingly-unrelatee adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.
Introduction
Bechine-Learning (ML) at production often requires constant model updating which can have The production pipeline is continuously evolving. where new data are regularly pooled and labeled and old data become irrelevant. Regular tuning of hyperparameters is required to handle drifts [19], and sometimes even the model class/architecture may need to change. Due to this, the model is frequently retrained, retested, and redeployed, which can be extremely costly, especially when the data and model sizes are large. The cost can be reduced if, instead of repeated retraining, the system can quickly adapt to incremental changes. Humans and animals can naturally use their prior knowledge to handle wide-variety of changes in their surroundings, but such quick, wide, and accurate adaptation has been difficult to achieve in ML.
huge financial and environmental costs 19 44
In theory, this should be possible within a Bayesian framework where the posterior is used the prior for the future, but exact Bayes is computationally challenging and the design of generic Bayesian 39 In ML, simpler mechanisms are more popular, for example, in Support Vector Machines (SVMs) for adding/removing data 122 58 60], and in deep learning for model compression 24 Wrightspriors are used in online learning B3 and more recently for
priors has its own challenges [55
62]. but they are not suited for many other tasks, such as model compression. some settings, they also perform worse, for example, continual learning when compared to memory-based strategies [34] 46 45 9 All these previous works apply to narrow, specific settings, and designing generic adaptation-mechhe mechanisms remains an open challenge.
continual learning 30
40
47
33
We present Knowledge-adaptation priors (K-priors) for the design of generic iiaptaaioon-mchaniist The general principle of adaptation is to combine the weight and function-space divergences to faithfully reconstruct the gradient of the past. K-priors can handle wide variety of adaptation tasks (Fig. I] left) and work for range of models, such as generalized linear models, deep networks, and their Bayesian extensions. The principle unifies and generalizes many seemingly-unrelate existing works, for example, weight-priors [13), knowledge distillation (4].. SVMs 12 35
Authors contributed equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Pest Model
Past Memory
Past memory Used in K- K-priors
K-prion
Validaion aoc 196)
100
Add new data
Rlemoe old data
K-priors
pest modelt
40
Replay
Adaptation tasks
prir
Change Reguiarizer
Change architecture
Add
Remove old data
Change Regulanizrr
Change architecture
Validtion acs (%)
100
New
Past data
60
Updated Model
10
100
Memory past data) Memory sin (%
Figure I: Left: K-priors handle wide-varicty of adaptation tasks by using the past model and small memory of the past data. Middle: For adaptation, the model needs tweaking at only a handful of past examples (shown with dark purple markers). The adapted model (dashed red line) is very close to the retraining on the full batch (solid black line). Right: Results on binary classification on "USPS' digits with neural networks show that K-p K-priors (red square) obtain solutions close to the batch training (gray line) but by using only a fraction (2-5%) of the past data. also performs better than 'Replay" (blue circles) where the same memory is used for replay. See Sec. 5 for more details.
online Gaussian Processes [7]. and continual learning [36 45 9]. It leads to natural adaptation- mechanisms where models' predictions need to be readjusted only a handful of past experiences (Tg.. I middle). This is quick and easy to train with first-order optimization methods and, by choosing a sufficienlly large memory, we obtain results similar to retraining foriiinng------ee from-scratch (Fig. right). Code available at httttt:://ggiiie http://ggtttbbbt// com/team-approne http://aaaapoobaaeee//a/eaaeeeea bayes/
2
Adaptation in Machine Learning
2.1 Knowledge-ddaptation tasks
Our goal is to quickly and accurately adapt an already trained model to incremental changes in its training framework. Throughout refer to the trained model the base model. We denote its outputs by fw. (x) for inputs IR and assume that its parameters w. W C R²" are obtained by solving the following problem on the data D,
W, = arg min E(w), where ł(w)
Σ (((w) ([w++ R(w).
weW
iED
Here, (w) denotes the loss function on the i'th data example, and R(w) is a regularizer.
A good adaptation method should be able handle many types of changes. The simplest and most common change is to add/remove data examples, as shown below at the left where the data example j & D is added to get w and at the right where example D is removed to get w-,
W arg min
Σ ((w) RRw),
arg min WEW
Σ ((()) + R(w). RED
(2)
WEW
iEDUJ
We refer to these problems as 'Add/Remove Data" tasks.
Other changes are the 'Change Regularizer' (shown the left) task where the regularizer is replaced by a new one (ww), and the 'Change Model Class/Architecture task (shown on the right) where the model class/architecture is changed leading change of the parameter space from W to θ
Wg = arg min
Σ IED
e, (w) ((w),
0. = arg min Dee
Σ SED
(,(0) + R())..
(3)
WEW
Here, we assume the loss (,( has the same form (w()) ) but using the new model fo(x) for prediction, with A E θ as the new parameter. can be of a different dimension, and the new regularizer can be chosen accordingly. The change in the model class could be simple change in

---

features, for example it is common in linear models to add/remove features, or the change could be similar model compression or knowledge distillation (24), which may not have a regularizer.
Another type of change is to add privileged information, originally proposed by Vapnik and Izmailov [60]. The goal is include different types of data to improve the performance of the model. This is combined with knowledge distillation by Lopez-Paz et al. [37 and has recently been applied to 50 51]
domain adaptation in deep learning
There could be several other kinds of changes in the training framework, for example, those involving a combination of the changes discussed above. Our goal is to
change in the loss function, develop a method that can handle such wide-varied variety of changes, or "adaptation tasks' as we will call them throughout. Such adaptation can be useful to reduce the cost of model updating, for example in a continuously evolving MI. pipeline. Consider k-fold cross-validation, where the model is retrained from scratch for every data-fold and hyperparameter setting. Such retraining can be made cheaper and faster by reusing the model trained in the previous folds and adapting them for new folds and hyperparameters. Model reuse can also be useful during active-learning for dataset curation, where decision to include a new example can be made by using a quick Add Data adaptation. In summary, model adaptation can reduce the cost by avoiding the need to constantly retrain.
2.2 Challenges of knowledge adaptation
Knowledge adaptation in ML has proven to be challenging. Currently there are no methods that can handle many types of adaptation tasks. Most existing works apply narrowly to specific models and mainly focuses on adaptation to Add/Remove Data only. This includes many early proposals for
SVMs [12 59 25 20 49 35
31 58) recent ones for machine-unleaming [1121 41 22
53 g and
methods that use weight and functional priors, for example, for online learning 13, continual deep
learning 30 40 47 33
46
57 45 9] and Cluussaa-PProeee (GP) models [17 52] [6]..
The methodologies of these methods are entirely different from each other, and they do not appear to have any common principles behind their adaptation mechanisms. Our goal is fill this gap and propose a single method that can handle a wide-variety of tasks for a range of models.
We also note that there is prior work on the Change Regularizer task. Adaptation has been used to boost hy bypraaaraeeter tuning [6]] but only Add/Remove Data adaptation is used. Warm-starts have been employed as well [18), but it is often not sufficient and can even hurt performance (5].
2.3 Problem setting and notation
Throughout, we will use supervised problem where the loss is specified by an exponential-famm family,
((y,h())) = - log p(yl = =- -yy, A()),
(4)
where y E y denotes the scalar observation output, f F is the canonical natural parameter, A(f) is the log-partition function, and h(f) = E(y) = A(f) is the expectation parameter. A typical example the cross-entropy loss for binary outcomes y E [0, where A()) = log(1 + e and h(s) = (ʃ) is the Sigmoid function. It is straightforward to extend our method to vector observation and model outputs. An extension to other types of learning frameworks is discussed in the next section (see the discussion around Eg. 14).
Throughout the paper, we will use a shorthand for the model outputs, where we denote f fw (x.). We will repeatedly make use of the following expression for the derivative of the loss,
Vily,, (Cly,,,(Et)) h(f()) V [M(FW) vi].
(5)
3 Knowledge-AAaaptatin Priors (K-priors)
We present powwledge-ddaptationn priors (K-priors) to quickly and accurately adapt a model's knowledge to wide variety of changes in its training framework. K-priors, denoted below by K(W;; w.,,w,;) refer to a class of priors that use both weight and function-space regularizers,
K(w:; w., M) D, (f( (w) (ww)) + rD, (w|lw>),
(6)
where f(w) is a vector of fw (u:), defined at inputs M = (i,, U2,..., uM). The divergence Dy (+Il-) measures the discrepancies in the function space F, while (-Il.) measures the same in the
3

---

weight space W. Throughout, we will use Bregman divergences (p1, p2) - (p1)) - (p22)
Ve((2)) (pi P2), specified using a strictly-convex Bregman function ()(-).
K-priors are defined using the base model wa, the memory set M, and trade-off parameter 0. We keep 7 = 1 unless otherwise specified. It might also use other parameters required to define the divergence functions. We will sometimes omit the dependency parameters and refer K((w).
Our general principle of adaptation to use K((w) to faithfully reconstruct the gradients of the past training objective. This is possible due to the combination of weight and function-space divergences. Below, we illustrate this point for supervised learning for Generalized Linear Models (GLMs).
3.1 K-priors for GLMs
GLMs include models such as logistic and Poisson regression, and have a linear model fü Φ. w. with feature vectors Φ; = x(x.). The base model is obtained as follows,
W. arg min
Σ
Elyi, h(f))) R(w).
WEW
IED
In what follows, for simplicity, we use an L2 regularizer R(w) = _ ||? with d > 0.
We will now discuss K-prior that exactly recovers the gradients of this objective. For this, we choose Dw (-l+)) to be the Bregman divergence with R(w) as the Bregman function,
T (wl (.w) BR (w] w.) _ w
We set memory MM- = X. where X' is the set of all inputs from D. We regularize each example using separate divergences whose Bregman function is equal to the log-partition A()) (defined in Eq. 4
D (f(w)][f(w.)
Σ (H((F.) + constant.
Ex
Smaller memories are discussed later in this section. Setting 1, we get the following K-prior,
K(w; w w.
(M(F + 36 w-ll? VEX
which has a similar form to Eq. 7 but the outputs y/s are now replaced by the predictions h(f( and the base model w. serves the mean of Gaussian weight prior.
can now show that the gradient of the above K-prior is equal to that the objective used in Eq.
VK(w: w., x) = Σ Φ [M(f h( + - 6(w-w.),
iE
Σ IED
Φ [h((ft) yil + +ww Σ Φ [h( IED
-n - 8w.,
(10)
=Vi(w).
where the first line is obtained by using Eq. 5 and noting that V = d,, and the second line is obtained by adding and subtracting outputs y/ in the first term. The second term there is equal to because w, is a minimizer and therefore V/(w.). = In this case, the K-prior with M = X' exactly recovers the gradient of the past training objective.
Why are we able to recover exact gradients? This is because the structure of the K-prior closely follows the structure of Eq. 7 the gradient of each term in In[q.. 7is recovered by a corresponding divergence in the K-prior. The gradient recovery due to the property that the gradient of Bregman divergence is the difference between the dual parameters Vih(p):
Bp, P2) = Ve(pi) - Vv((2).
This leads to Eq. For the function-space divergence term, M(( h(f(.) are the differences in the (dual) expectation parameters. For the weight-space divergence term, we note that the dual space equal to the original parameter space W for the Ta2 regularizer, leading to w w.. Lastly, we find that terms cancel out by using the optimality of w., giving the exact gradients.

---

3.2 K-priors with limited memory
In practice, setting M X' might be as slow as full retraining, but for incremental changes, we may not need all of them (see Fice for example). Then, which inputs should we include? The answer lies in the reconstruction error e, shown below for M C C.
e(w; V((w) (r(W))VX((wwW)))
Σ 0, ( SERRM
(1I)
The error depends the "leftover" Φb for i E M and their discrepancies h h((()) h( ). A simple idea could be to include the inputs where predictions disagree the most, but this is not feasible because the candidates w are not known beforehand. The following approximation is more practical,
M(w G. (X)) -ollwwwww -w., where G.
Σ iEXIM
0. (o.
(12)
This is obtained by using the Taylor approximation h(ft) - R WI (VV.) T..T) (w w.)
in Eq. 11 (h'(f") is the derivative). The approximation is conveniently expressed in terms of the Generalized Gauss-Newton (GGN) matrix (88, denoted by G.).). The approximation suggests that M should be chosen to keep the leftover GGN matrix G. (X) orthogonal to w w.. Since w changes during training, reasonable approximation is to choose examples that keep the top-eigenvalues of the GGN matrix. This can be done by forming a low-rank approximation by using sketching methods, such as the leverage score [16] I 15 IO]. A cheaper alternative is to choose the examples with highest 1/ df(,,, ). The quantity is cheap to compute general, for example, for deep networks it is obtained with just forward pass. Such a set has been referred to as the 'memorable past' by Pan et al. [45), who found it to work well for classification. Due to its simplicity, we will use this method in our experiments, and leave the application of sketching methods future work.
K-priors with limited memory can achieve low reconstruction error. This is due to an important feature of K K-priors: they do not restrict inputs u, to lie within the training set X' The inputs can be arbitrary locations in the input space. This still works because ground-truth label is not needed for u, and only model predictions [w (u:) are used in K-priors. As long as the chosen represent X well, we can achieve low gradient-reconstrice
error, and sometimes even perfect reconstruction. Theoretical results regarding this point are discussed in App. A] where we present the optimal K K-prior which can theoretically achieve perfect reconstruction error by using singular vectors of (x()). ((x2)..... ((xx)]. When only top-M singular vectors are chosen, the error grows according to the leftover singular values. The optimal K-prior is difficult to realize in practice, but the result shows that theoretically possible to achieve low error with limited memory.
3.3 Adaptation using K-priors
We now discuss how the K-prior of Eq. 8 can be used for the Add/Remove Data tasks. Other adaptation tasks and corresponding K-priors are discussed in Ampp.. B
Because K-priors can reconstruct the gradient of E(w), we can use them to adapt instead of retraining from scratch. For example, to add/remove data from the GLM solution in Eq. 7 we can use the following K- K-prior regularized objectives,
arg min
(w(w) + K(ww; w.. M).
W_ arg min -le(w) + K(w: weW
(13)
WEW
Using Eq. 10 it is easy to show that this recovers the exact solution when all the past data is used. App. B details analogous results for the Change Regularizer and Change Model Class tasks.
Theorem 1. For M X, we have w+ = W and w_ w
For limited memory, we expect the solutions to be close when memory is large enough. This is because the error in the gradient is given by Eq. II The error can be reduced by choosing better M and/or by increasing its size to ultimately get perfect recovery.
We stress that Eq. 13 is fundamentally different from replay methods that optimize an objective similar to Eq. 2 but use a small memory of past examples [48). Unlike such methods, we use the predictions IT ). which we can think of as soft labels, with potentially more information than the true one-hot encoded labels yt. Given a fixed memory budget, we expect K-prior regularization to perform better than such replay methods, and observe this empirically in Sec. 5
5

---

3.4 K-priors for Generic Learning Problems
The main principle behind the design of K-priors construct such that the gradients can faithfully be reconstructed. As discussed earlier, this is often possible by exploiting the structure of the learning problem. For example, to replace an old objective such as Eq. 1 with loss (f) and regularizer Rwo)(w), with a new objective with loss 1 (f) and regularizer Reew (w), the divergences should be chosen such that they have the following gradients,
VDw(ww /w.) = VR (w) - (R)
VDD (f)(f(())ll(ww = Vf(w) T dm
(14)
where dm is an M-length vector with the discrepancy V Veren (fr) Tei as the i'th entry. The matrix B is added to counter the mismatch between D and M. Similar constructions can be used for other learning objectives. For non-differentiable functions, a Bayesian version can be used with the Kullback-Leibler (KL) divergence (we discuss example in the next section). We can use exponential family distributions which implies a Bregman divergence through KL [6] Since the gradient of such divergences is equal to the difference in the dual-pan parameetes,,,, the general principle is to use divergences with an appropriate dual space to swap the old information with new information.
4 K-priors: Extensions and Connections
The general principle of adaptation used in K-priors connects many existing works in specific settings. We will now discuss these connections to show that K-priors provide a unifying and generalizing principle for these seemingly unrelated methods in fields such online learning, deep learning, SVMs, Bayesian Learning, Gaussian Processes, and continual learning.
4.1 Weight-Priors
30 54 be seen as specialized cases of K- K-priors, where restrictive approximations are used. For example, the following quadratic regularizer,
Quadratic Gaussian weight-priors 13
Ryual (w; w.) = (w w.) (x) + 8I] (w www)),
which is often used in used in online and continual learning 13 30), can be seen first-order
approximation the K-prior in n/EE. 8) This follows by approximating the K-prior gradient [Eq. by using the Taylor approximation used Eq. 12 to get
VKC(w; w. X)
Σ Φb NEN
[ w. w.) (w - w.)] + 5(w w.) = V'Ruuad (W; W wx).
K-priors can be more accurate than weight priors but may require larger storage for the memory points. However, we expect the memory requirements to grow according to the rank of the feature matrix (see App. A) which may still be manageable. If not, we can apply sketching methods.
4.2
K-priors for Deep Learning and Connections to Knowledge Distillation
We now discuss the application to deep learning. It is clear that the functional term in K- K-priors is similar Knowledge distillation (KD) (24), which is popular approach for model compression in classification problems using a softmax function (the following expression is from Lopez-Paz et al.
37
also App.
(KD(w) = A Ee( X (M(FF..TT. IED
(15)
IED
The base model predictions are often scaled with a temperature parameter T > 0, and X [o, 1]. KD can be seen as a special case of K-priors without the weight-space term (7 0). K-priors extend KD in three ways, by (i) adding the weight-space term, (ii) allowing general link functions or divergence functions, and (iii) using potentially small number of examples in M instead of the whole dataset. With these extensions, K-priors can handle adaptation tasks other than compression. Due to their similarity, it is also possible borrow tricks used in KD to improve the performance of K-priors.
KD often yields solutions that are better than retraining from scratch. Theoretically the reasons behind this are not understood well, but we can view KD mechanism to reconstruct the past

---

gradients, similarly to K-priors. As we now show, this gives a possible explanation behind KD's success. Unlike GLMs, K-priors for deep learning do not recover the exact gradient of the past training objective, and there is an additional left-over term (a derivation is in App. C..
VK(w)=
V [M(TWW mi] dw
Σ Vl öw.
(16)
=V((w)
Additional term since VIL
where rw. : (([ ) yi is the residual of the base model. It turns out that the gradient of the KD objective in Eq. 13 has this exact same form when 0, T (derivation in App. Cj.
VlxD(w)= Σ V. [M(FW) yal - (1 A) Σ
V Vfnn w.
IDD
The additional term adds large gradients to push away from the high residual examples (the examples the teacher did not fit well). This similar to Sinilaritt-Contrer for SVMs from Vapnik and Izmailov [60], where sllack"-vrriables are used in dual formulation to improve the student, who could now be solving a simpler separable classification problem. The residuals r.. above play a similar role as the slack variables, but they do not require dual formulation. Instead, they arise due to the K-prior regularization primal formulation. In this sense, K-priors can be seen as an mosylto-impleeen scheme for Similarity Control, that could potentially be useful for student-teacher learning.
Lopez-Paz al. (37 use this idea further to generalize distillation and interpret residuals from the teachers as corrections for the student (see Eq. 6 in their paper). In general, it is desirable to trust the knowledge of the base model and use to improve the adapted model. These previous ideas are now unified in K- K-priors: we can provide the information about the decision boundary to the student in more accessible form than the original data (with true labels) could.
4.3 Adding/removing data for SVMs
K-prior regularized training yields equivalent solutions to the adaptation strategies used in SVM to add/remove data examples. K-priors be shown to be equivalent the primal formulation of such strategies 355. The key trick to show the equivalence is to use the representer theorem which we will now illustrate for the 'Add Data" task Bq. 13 Let 4 be the (N + 1) P feature matrix obtained on the dataset j. then by the representer theorem we know that there exists a B R N+1 such that W + I Taking the gradient of Eq. 13 and multiplying by 4+, we can write the optimality condition as,
0 0= + v(G((w+) K[(+)]]
Σ SEDUj
kit + 5K
(17)
where K4 = + and its i'th column is denoted by k. This is exactly the gradient of the primal
objective in the function-sey function-space defined over the full batch D U j; see Equation 3.6 in Chapelle [14]
The primal strategy is equivalent to the more common dual formulations [12 59 25 20 49 31 58) The function-space formulations could be computationally expensive, but speed-ups can be obtained by using support vectors. This is similar to the idea of using limited memory in K-priors in Sec.
4.4 K-priors for Bayesian Learning and Connections to GPs
K K-priors can be seamlessly used for adaptation within a Bayesian learning framework. Consider Gaussian approximation q. (w) trained on a variational counterpart of Eq. with prior p(w) oc exp (R(()), and its adapted version where we add data, as shown below (Dxul. [| is the KI. divergence).
q- (w) = arg min
Σ F., ([)) + Dx 4] (+(w) = arg min gEQ
Σ [.]]](w)] + Dxcle]1)). iEDLJ
qEQ
IED
Assuming the same setup as Sec. we can recover g+ (w) by using 9X (w) x exp [-K(w)] where we use the K-prior defined in Eq. 8 (note that normalization constant of gi is not required),
q w) arg min [yl](w)) + Dxr. qcl.
(18)
qEQ
7

---

This follows using Eq. 10 Details are in App. D In fact, when this Bayesian extension is written in the function-spe -pacce similarly to to|Eq. 17 it is related to the online updates used in GPs [7]. When Qx: built with limited memory, as described in [Sec. 3 the application is similar to sparse variational GPs, but now data examples are used as inducing inputs. These connections are discussed in more detail in App. D] Our K-prior formulations operates in the weight-space and can be easily trained with first-order methods, however an equivalent formulation in the function space can also be employed, as is clear from these connections. The above extensions can be extended to handle arbitrary exponential- family approximations by appropriately defining K-priors using KL divergences. We omit these details since this topic is more suitable for Bayesian version of this paper.
4.5 Memory-Based Methods for Deep Continual Learning
K-priors is closely related to recent functional regularization approaches proposed for deep continual learning [4 46 54 57 99. The recent FROMP approach of Pan al. [45] is closest to ours where the form of the functional divergence used is similar to our suggestion in Eg. 14 Specifically, comparing with Eq. 14 their functional divergence correspond the vector d,, with the i' i'th entry as [u[[ - h w. (u:)) for u; E .,, and the matrix B is (can be seen Nystrom approximation),
B= A(w)
(G..)) Vf(w.. A(W.)]
where A(w) is a diagonal matrix with 1 (fw (u:)) as the i'th diagonal entry, and G(w.) = Vf(w.) Afw.)Vf(w.) + 6I is the GGN matrix. They also propose use 'memorable past' examples obtained by sorting h' (fw.)) ). which is consistent with our theory (see (seeFq.. 11). Based on our work, we can interpret the approach of Pan et [45 as mechanism to reconstruct the gradient of the past, which gives very good performance in practice.
Another related approach is the gradient episodic memory (GEM) (36). where the goal is to ensure that the EieM [f(y,, (w) E(y;; J )] < O. This is similar in spirit to the student-teacher transfer of Vapnik and Izmailov (60) where the loss of the student is regularized using the model output of the teacher (see Eq. 7 in Vapnik and Izmailov (60) for an example). Lopez-Paz and Ranzato (36) relax the optimization problem to write in terms of the gradients, which is similar to K-priors, except that K- K-priors use first-order optimization method, which is simpler than the dual approach used in Lopez-Paz and Ranzato 36
Most of these approaches do not employ a weight-space divergence, and sometimes even the function- space divergence is replaced by the Euclidean one [7] [9. Often, the input locations sampled randomly, or using a simple replay method [9] which could be suboptimal. Some approaches propose computationally--xppe -xxpensseveeeeeeiiveeeeee methods for choosing examples store in memory (2 3. and some can be seen as related to choosing points with high leverage (3] The approach in Titsias et al. (57) uses inducing inputs which is closely connected to the online GP update. The method we proposed does
not contradict with these, but gives a more direct way to choose the points where the gradient errors are taken into consideration.
5 Experimental Results
We compare the performance of K-priors to retraining with full-batch data ('Batch") and retraining with replay from small memory ('Replay'), and use 1. For fair comparisons, we use the same memory for Replay and K-priors obtained by choosing points with highest ((,,) (see Sec. 3). Memory chosen randomly often gives much worse results and we omit these results. Replay uses the true label while K K-priors use preecc-preictions. We compare these three methods on the four adaptation tasks: 'Add Data', 'Remove Data', 'Change Regularizer', and 'Change Architecture' For the 'Add Data" task, we also compare to Weight-Priors with GGN.
Our overall finding is that, for GLMs and deep learning on small problems, K-priors can achieve the same performance as Batch, but with small fraction of data (often 2-10% of the full data (Fig. right, and 2}. Replay does much worse for small memory size, which clearly shows the advantage of using the model predictions (instead of true labels) in K-priors. Weight priors generally perform well, but they can do badly when the adaptation involves a drastic change for examples in M (see Fig. 3). Finally, for large deep-learning problems, results are promising but more investigations are required with extensive hyperparameter tuning.
8

---

Add new data
Remove old data
Change regularizer
Change model class
Validation atc (%)
80
(a) Aduit,
60-
logiatic
regression
40
Batch
Replay
20
K-prior
50
100
50
100
2
10
20
50
100
10
20
50
100
100
Validatiion ace (%)
(b) USPS,
80
logistic
regression
60
40 Memory size (% of past data)
50 100
2
20 50 100
10 20 100 Memory size (% of past data)
2
10 20 100
Memory size (% of past data)
Memory size 1% of past data)
Figure 2: K-priors (red squares) match Batch (grey) while mostly using 2-5% of the data (for only 3 tasks larger fraction required). K-priors always outperforms Replay which uses the true labels. K- K-priors replace the labels by the model predictions (see the discussion after Theorem
Several additional experiments are in App. E tion and find that it performs similarly to a warm start at w.. In App. E.5] we find that K-priors with limited memory are take much less time to reach a specified accuracy than both batch and replay. The low cost is due to small memory size. Replay also uses small memory but performs poorly.
In App. E.4, we study the effect of randomly initializa-
Logistic Regression on the 'UCI Adult' dataset. This is binary classification problem consisting of 16,100 examples to predict income of individuals. We randomly sample 10% of the training data (1610 examples), and report mean and standard deviation over 10 such splits. For training, we use the L-BFGS optimizer for logistic regression with polynomial basis. Results are summarized in Fig. 2a)). For the 'Add Data' task, the base model uses 9% of the data and we add 1% new data. For 'Remove Data', we remove 100 data examples (6% of the training set) picked by sorting h h'( )). For the 'Change Regularizer' task, we change the L2 regularizer from = 50 to 5, and for 'Change Model Class", we reduce the polynomial degree from 2 to 1. K-priors perform very well on the first three tasks, remaining very close to Batch, even when the memory sizes are down to 2%. 'Changing Model Class' is slightly challenging, but K- K-priors still significantly out- out-perform Replay.
Logistic Regression on the *SSS odd even' dataset. The USPS dataset consists of 10 classes (one for each digit), and has 7,291 training images of size 16 x 16. We split the digits into two classes: odd and even digits. Results are in Fig. b). For the 'Add Data' task, we add all examples for the digit 9 to the rest of the dataset, and for "Remove Data' we remove the digit 8 from the whole dataset. By adding/removing an entire digit, we enforce an inhomogeneous data split, making the tasks more challenging. The 'Change Regularizer' and 'Change Model Class' tasks are the same as the Adult dataset. K-priors perform very well on the 'Add Data' and 'Change Regularizer' tasks, always achieving close to Batch performance. For 'Remove Data', which is challenging task due to inhomogeneity, K-priors still only need to store 5% of past data to maintain close to 90% accuracy, whereas Replay requires 10% of the past data.
Neural Networks on the USSS odd vs even' dataset. This is repeat of the previous experiment but with neural network (a 1-hidden-layer MLP with 100 units). Results are in (right). The 'Change Regularizer' task now changes the architecture from a 2 2-hidden-layer MLP (100 units per layer) to 1-hidden-layer MLP with 100 units. We see that even with neural networks, K-priors perform very well, similarly out-performing Replay and remaining close to the Batch solution at small memory sizes.
to 10, and the 'Change Architecture task compresses
Weight-priors vs K-priors. As discussed in the main text, weight-priors can be seen as an approxi- mation of K-priors where h'' (((fw) are replaced by 'stale" h' H((fú,), ), evaluated at the old w. In Fig. 3 a), we visualize these 'stale" h h((fi. and compare them to K-priors which obtains values close to the ones found by Batch. Essentially, for the points at the diagonal the match is perfect, and we see that it is the case for K-priors but not for the weight-priors. We use logistic regression on the USPS data (the 'Add Data' task). This inhomogeneous data split difficult for weight-priors, and we show in Fig. 3(b) that weight-priors do not perform well. For homogeneous data splits, weight- do

---

(a) overr all é E D
(b) USPS Add Data
(c) CIFAR-10 Add data
(d) CIFAR-10 Knowledge Distillation
80
Validation ace (%)
0.2
60
Baxch Baxch traiming
90
75
55
80
01
50
Weight-priorr Kprior
202
70
10
20
50
100
500
Memory size (% of past data)
Memory size (% of past data)
Memory size (% of past data)
0.1
0.2
Ttaining with
Batch
K-prior
Weight-prior -
Replay
K-prior, temperature
Figure 3: (a) When compared at the Batch solution for the 'Add Data' task USPS, weight priors give incorrect values of h'(ww) (shown with black dots, each dot corresponds to a data example). Points on the diagonal means perfect match which is the case for K-priors (show with red dots). (b) Due to this, weight-prior (green diamonds) perform worse than K-priors (red squares). (c) For the 'Add data' task on CIFAR-10, K-priors outperform replay (blue circles), but performance can still be improved by using a temperature parameter (dark-red triangles). (d) The same is true for knowledge distillation [24], and we see that we can reduce memory size while still performing better than the student model.
better than this, but this result goes to show that they do not have any mechanisms fix the mistakes made in the past. In K- K-priors. we can always change M to improve the performance. We provide more plots and results in App. E. including results for weight-pricg -riiors on all the 'Add data' tasks we considered in this paper.
MNIST and CIFAR-10, neural networks. Finally, we discuss results on larger problems in deep learning. We show many adaptation tasks in App. Efoor 1 10-way classification on MNIST [32] with MLPs and 10-way classification CIFAR-10 10 with CifarNet [62], trained with the Adam optimizer 29]. In Fig. 3/c) we show one representative result for the 'Add data' task with CIFAR-10, where we add random 10% of CIFAR-10 training data to the other 90% (mean and standard deviation over runs). Although vanilla K-priors outperform Replay, there is now a bigger gap between K-prior and Batch even with 50% past data stored. However, when we use temperature (similar to knowledge distillation in (15) but with the weight term included), K-priors improves.
A similar result is shown in Fig. d()) for knowledge distillation (8 0 but with temperature parameter) where we are distill from a CifarNet teacher to a LeNet5-style student (details in App. E). Here, K-priors with 100% data is equivalent to Knowledge Distillation, but when we reduce the memory size using our method, we still outperform Batch (which trained from scratch on all data). Overall, our initial effort here suggests that K-priors can do better than Replay, and have potential to give better results with more hyperparameter tuning.
Discussion
In this paper, we proposed a class of new priors, called K-prior. We show general principles of obtaining accurate adaptation with K-priors which are based on accurate gradient reconstructions. The prior applies to a wide-variety of adaptation tasks for a range of models, and helps us to connect many existing, seemingly-unrelan
adaptation strategies in MI. Based on our adaptation principles, we derived practical methods to enable adaptation by tweaking models" predictions at few past examples. This is analogous to adaptation in humans and animals where past experiences is used for new situations. In practice, the amount of required past memory seems sufficiently low.
The financial and environmental costs of retraining are a huge concern for ML practitioners, which can be reduced with quick adaptations. The current pipelines and designs are specialized for an offline, static setting. Our approach here pushes towards a simpler design which will support more dynamic setting. The approach can eventually lead to new systems that learn quickly and flexibly, and also act sensibly across a wide range of tasks. This opens a path towards systems that learn incrementally in a continual fashion, with the potential to fundamentally change the way ML is used in scientific and industrial applications. We hope that this work will help others to more towards this goal in the future. We ourselves will continue to push this work in that direction.
10

---

Acknowledgements
We would like to thank the members of the Approximate-Bayesiaa---- Inference team at RIKEN- AIP. Special thanks to Dr. Thomas Möllenhoff (RIKEN-AIP), Dr. Gian Maria Marconi (RIKEN- AIP), Peter Nickl (RIKEN-AIP), and also to Prof. Richard E. Turner (University of Cambridge). Mohammad Emtiyaz Khan is partially supported by KAKENHI Grant-in-Aid for Scientific Research
(B), Research Project Number 20H04247. Siddharth Swaroop is partially supported by a Microsoft
Research EMEA PhD Award.
Author Contributions Statement
List of Authors: Mohammad Emtiyaz Khan (M.E.K.), Siddharth Swaroop (S.S.).
Both the authors were involved in the idea conception. S.S. derived version of theorem with some help from M.E.K. This was then modified and generalized by M.E.K. K. for generic adaptation tasks. The general principle of adaptation described in the paper are due to M.E.K., who also derived connections to SVMs and GPs, and extensions to Bayesian settings (with regular feedback from S.S.). Both authors worked together on the connections to Knowledge Distillation and Deep Continual Learning. S.S. performed all the experiments (with regular feedback from M.E.K.). M.E.K. wrote the main sections with the help of S.S., and S.S. wrote the section about the experiments. Both authors proof-read and reviewed the paper.
References
[1] Ahmed Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statis- tical guarantees. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[2] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[4] Shuang Ao, Xiang Li, and Charles Ling. Fast generalized distillation for semi-supervised domain adaptation. In Proceedings the AAAI Conference on Artificcal Intelligence, volume 31, 2017.
[5] Jordan Ash and Ryan P Adams. On warm-startin neural network training. In Advances in Neural Information Processing Systems, volume 33, pages 3884-3894, 2020.
[6] Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. Clustering with Bregman divergences. Journal of machine learning research, 6(Oct): 7705-1799, 2005.
[7] Ari Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. In International Conference on Learning Representations, 2019.
[8] Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In International Conference on Machine Learning, 2021.
[9] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. In Advances in Neural Information Processing Systems, volume 33, pages 15920-15999 2020.
[10] Daniele Calandriello, Luigi Carratino, Alessandro Lazaric, Michal Valko, and Lorenzo Rosasco. Gaussian process optimization with adaptive sketching: Scalable and no regret. Conference on Learning Theory, pages 533-557. PMLR, 2019.
[11] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, page 463-480, May 2015. doi: 10.1 10.1009.........55
[12] Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In Advances in Neural Information Processing Systems, volume 13, MIT Press, 2001.

---

[13] Nicolo Desa-Bianchi and Gabor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[14] Olivier Chapelle. Training a support vector machine in the primal. Neural Computation, 19(5): 1155-1178, 1178, May 2007. ISSN 0899-76677. doi: 10.1162/neco 200.. 1155.
[15] Michael B Cohen, Cameron Musco, and Christopher Musco. Ridge leverage scores for low-rank approximation. arXiv preprint arXiv: 1.10262,,, 6, 2015.
[16] Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):
15-18, 1977.
[17] Lehel Csató and Manfred Opper. Sparse on-line Gaussian processes. Neural computation, 14 (3):641-688, 2002.
[18] Dennis DeCoste and Kiri Wagstaff. Alpha seeding for support vector machines. Proceedings of the sixth ACM SIGKDD international conference Knowledge discovery and data mining- KDD '00, page 345-349. ACM Press, 2000. ISBN 978-1-58113-2555 doi: 10.1 145344593 347165.
[19] Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. Continual learning practice. arXiv preprint prrXiv/903..0 052200 2019.
[20] Hua Duan, Hua Li, Guoping He, and Qingtian Zeng. Decremental learning algorithms for nonlinear langrangian and least squares support vector machines. In Proceedings of the First International Symposium on Optimization and Systems Biology (OSB 07), pages 358-366. Citeseer, 2007.
[21] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304 9312, 2020.
[22] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3832 3842. PMLR, 13-18 Jul 2020.
[23] Ralf Herbrich, Neil Lawrence, and Matthias Seeger. Fast sparse gaussian process methods: The informative vector machine. In Advances in Neural Information Processing Systems, volume 15, MIT Press, 2003.
[24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 5555000335)) 2015.
[25] Masayuki Karasuyama and Ichiro Takeuchi. Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Networks, 17)):104481055) Jul 2010. ISSN 1045-9227, 1941-0093, doi: 10. 11111110TNN211102201221. 20448099.
[26] Mohammad Emtiyaz Khan. Decoupled variational Gaussian inference. In Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[27] Mohammad Emtiyaz Khan and Haavard Rue. wwweeennnnnnnnnnnneeeeeeeeeeeeee
from Bayesian principles. pdf
2020. btttps:/eemtiyyzz.ee gi thub 10 it/iii///apae hawp/s/ppesseeee
[28] Mohammad Emtiyaz Khan, Aleksandr Aravkin, Michael Friedlander, and Matthias Seeger. Fast dual variational inference for non-conjugate latent Gaussian models. In Proceedings of the 30th International Conference on Machine Learning, volume 28, pages 951-959, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[29] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[30] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinste Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-552 2017.
[31] Pavel Laskov, Christian Gehl, Stefan Krüger, Klaus-Robert Müller, Kristin P Bennett, and Emilio Parrado- Incremental support vector learning: Analysis, implementation and applications. Journal of machine learning research, 7(9), 2006.
12

---

[32] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:
7 //yann -eeuun com/ exdb/nnist/
[33] Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[34] Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2944 2017.
[35] Zhizheng Liang and YouFu Li. Incremental support vector machine learning in the primal and applications. Neurocomputiing 72(10):229-22558 Jun 2009. ISSN 0925-2312. doi:
10. www.:neuccccm......m
[36] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learn- ing. In Proceedings of the st International Conference on Neural Information Processing Systems, NIPS' 17, page 6470-6479, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978 9781510860964.
[37] David Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv: 1.03643, 2016.
[38] James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):-76, 2020.
[39] Eric Nalisnick, Jonathan Gordon, and Jose Miguel Hermandez-Lobato Predictive complexity priors. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 694-702. PMLR, 13-15 Apr 2021.
[40] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
[41] Quoe Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. In Advances in Neural Information Processing Systems, volume 33, pages 16025- 16036. Curran Associates, Inc., 2020.
[42] Frank Nielsen. On a variational definition for the jensen-shannon shannon symmetrization of distances based on the information radius. Entropy, 23(4))464, 2021.
[43] M. Opper and C. Archambeau. The variational Gaussian approximation revisited. Neural Computation, 21(3):786-795 2009.
[44] Andrei Paleyes, Raoul- Urma, and Neil D. Lawrence. Challenges in deploying machine learning: survey of case studies. arXiv:2011.0990 09926 [cs], Jan 2021. arXiv: 2011.09926.
[45] Pingho Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Tumner, and Mohammad Emtiyaz. Khan. Continual deep learning by functional regularisation of memorable past. In Advances in Neural Information Processing Systems, volume 33, pages 4453-4464. Curran Associates, Inc., 2020.
[46] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[47] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[48] Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7 (2):123-146, 1995.
[49] Enrique Romero, Ignacio Barrio, and Lluís Belanche. Incremental and decremental learning for linear support vector machines. In International Conference Artificial Neural Networks, pages 209-218. Springer, 2007.
[50] Sebastian Ruder, Parsa Ghaffari, and John Breslin. Knowledge adaptation: Teaching to adapt. arXiv preprint 17Xiv:1702.02052 2017.
[51] Nikolaos Sarafianos, Michalis Vrigkas, and loannis A Kakadiaris. Adaptive SVM+: Learning with privileged information for domain adaptation. In Proceedings of the IEEE International Conference Computer Vision Workshops, pages 2637-2644, 2017.
13

---

[52] Simo Särkkä, Arn Solin, and Jouni Hartikainen. Spatiotemporal learning via infinite- dimensional Bayesian filtering and smoothing: A look Gaussian process regression through Kalman filtering. IEFF Signal Processing Magazine, 30(4):51-61, 2013.
[53] Sebastian Schelter. amnesia-towards machine learning models that can forget user data very fast. In 1st International Workshop Applied Al for Database Systems and Applications (AIDB19), 2019.
[54] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Trabska-Barwinska Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4528-4537
PMLR, 2018.
[55] Daniel Simpson, Havard Rue, Andrea Riebler, Thiago G. Martins, and Sigrunn H. Sørbye. Penalising model component complexity: A principled, practical approach to constructing priors. Statistical Science, 32(1):1-28, 2017. ISSN 08834237, 21688745.
[56]
Arno Solin, James Hensman, and Richard E Turner. Infinite-horizon Gaussian processes. In Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[57] Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Machine Learning. 2020.
[58] Cheng-Hae -ha Tsai, thich-Yen Lin, and Chih-Jen Lin. Incremental and decremental training for linear classification. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, page 343-352. ACM, Aug 2014.
[59] Amund Tveit, Magnus Lie Hetland, and Håavard Engum. Incremental and decremental proximal support vector classification using decay coefficients. In International Conference on Data Warehousing and Knowledge Discovery, pages 422-429. Springer, 2003.
[60] Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: similarity control and knowledge transfer. Journal of Machine Learning Research, 16(1):2033-2049, 2015.
[61] Zeyi Wen, Bin Li, Ramamohanarao Kotagiri, Jian Chen, Yawen Chen, and Rui Zhang. Improv- ing efficiency SVM k-fold cros-valldattinn by alpha seeding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
[62] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
14


Reviews:
[{"review_id": "z-jD7RU3LR", "reviewer": "Reviewer_2Zuk", "summary": "- This paper tackles an important problem: Suppose we have trained a model on a dataset, but we wish to adapt it quickly. For example, we might add new training points, delete training points, or wish to change the model and regularization. We could re-train a model, but that's slow - can we get a similar result but quicker and using less memory?\n- This paper proposes K-prior: regularizing the model towards the original model in both weight space and predictions.\n- They show that training on the K-prior objective on the entire dataset recovers the original model - which would not be true if we only have weight space regularization, or only have prediction regularization.\n- In practice, to save memory, they only form the prior on a few points, and they describe heuristics to choose these points, with some mathematical justification\n- They show that this methods work well in practice, compared to just replaying the points \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "After rebuttal: added comments below\n\n---------------------------------------------------------------------------------------\nStrengths\n- The paper is well written. It seems well executed, technically sound, and reports results honestly and thoroughly.\n- The idea of K-priors is nice - it's pretty interesting that if you keep both the weight space regularization and prediction space regularization, then you can recover the original model. The math is simple, but I tend to view that as a strength\n- Experimental results on UCI adult, MNIST, USPS, and CIFAR-10, show promise. It's interesting that the K-prior method does better than replaying the loss on the same set of points they use for the K-prior regularization. They also include ablations on using only the weight regularization.\n\nQuestions and suggestions for improvement:\n- It would be more compelling if they include baselines from other papers. They say that prior methods focus on specific tasks, like adding points, or removing points, and don't do all the adaptations they do. However, it would still be nice to see how K-priors compare with methods specialized for adding new points. I'm not too familiar with the area or relevant baselines though.\n- The paper says that the main difference between \"Replay and K-priors is that the former uses the true label while K-priors use model-predictions.\" - does replay also add weight regularization besides replaying the loss?\n- Just to confirm, what was the tuning protocol to choose tau, the weighting between the weight and prediction regularization?\n- I think it would be good to discuss in more detail why K-priors do better than just replaying the loss. I believe replaying the loss would satisfy similar guarantees as K-priors? For example, in equation 10 and Theorem 1, if we replaced K(w) by sum_{i \\in D} l_i(w) + R(w). So what's the benefit of K-priors?\n- It seems like one benefit of K-priors could be that we get more (bits of) information from the model prediction than from the label, similar to in distillation, since the model prediction is a probability vector but the label is just one of C values, where C is the number of classes. The paper could be stronger if it explored these ideas and compared K-priors with replays.\n- Their method of selecting points, for linear models, seems to roughly correspond to choosing points close to the margin. Is that correct?\n- The experiments could be stronger if they did some grid search of hyperparameters for each method. For example, maybe weight priors need a stronger weight because they don't have the prediction space regularization? So ideally if they do the same grid search of hyperparameters for each method, and then compare them, it would be more convincing.\n\nnit: In line 301, a neural network with 1 hidden layer is not deep, so remove deep :)\n\nOverall, I think it's a nice idea, that's well motivated, and fairly well executed, so would be good to have in the conference!\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Seems fine to me!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "iA_wF-4QpAa", "reviewer": "Reviewer_AAdt", "summary": "The paper proposes K-priors, giving a family of approaches aiming to quickly update trained networks to tasks changed in various ways (with added data, removed data, a changed regularizer, or a changed architecture). The main features of K-priors are: they approximate (or in some cases compute exactly) gradients of the original objective using the outputs of a trained model instead of the data labels; and they can approximate the original objective well with only a small fraction of the original data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Novelty\nThe difference between the K-priors objective in Eq. 10 and full retraining is fairly small - as stated in line 130, it is that model outputs (with old weights) $h(f_{w^*}^i)$ are used in place of data labels $y_i$. This is similar to Knowledge Distillation [22] (cited in paper), as noted in the paper. The \"memorable past examples\" approach of [37] (cited in paper) is combined with this to reduce memory requirements - as far as I know, such a combination is novel.\n\n# Strengths\n- The paper is well-motivated, and I believe that reducing the cost and environmental impact of retraining models is an important line of work.\n- Equation 9 is an interesting result, showing that their approach exactly recovers the original gradients for generalized linear models.\n- The experimental results show significant improvements with using K-priors over 'Replay' when only a small amount of past data is kept.\n\n# Weaknesses\n- (EDIT: The authors have presented results satisfying me that updating weights using K-priors can indeed be significantly faster than retraining.) A major aspect of the motivation for this work is to reduce the time/energy consumption for retraining, but this is not demonstrated anywhere in the experiments (only a reduction in the number of examples that need to be stored in memory is demonstrated). If this is not to be included then, at the very least, the abstract should not claim that the empirical results \"confirm that the adaptation can be cheap and accurate\". To me, \"cheap\" implies that the training time/energy consumption is reduced.\n- It is difficult to interpet the performance on the \"Remove old data\" task by looking at the plots of validation accuracy - surely very good validation accuracy could be achieved by simply keeping the old weights and not removing any data.\n- Despite the response to 3a on the checklist, there does not seem to be any code included with the submission.\n\n# Writing\n- The writing is mostly of a good standard, but I think the novel contributions should be more clearly stated.\n- When performing adaptation, could the authors confirm whether weights are randomly initialised or initialised from the previously learned values?\n- The introduction of Eq. 4 is confusing - is $y$ a scalar, a one-hot encoding, or other vector? Similarly for $f$, is it a scalar, vector, etc. (i.e. what is $\\mathcal{F}$)?\n- (EDIT: I am no longer concerned about this point, and now understand that the objective suggested for each task are unified by being something like \"the simplest objectives which recover/approximate the gradients of the desired new objective without using data labels\".) It is a little unclear what unifies the different objectives presented as K-priors - e.g. Equation 10 for adding data, in which $\\mathcal{K}(w)$ has the same gradients as the old objective, and Equation 12 for changing the regulariser, in which $\\mathcal{K}(w)$ is modified based  on the new regulariser, and then used on its own as the new objective. Then in equation 19 for changing the model class, $\\mathcal{K}(\\theta)$ is used as the new objective, and no longer contains a weight-space divergence. It would be helpful if the authors could be clearer about that unifies these, and distinguishes them from prior work.\n\nEDIT SUMMARY: See bullet point 1 under \"Weaknesses\" and bullet point 4 under \"Writing\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim, I think reasonably, that this work has little obvious potential for negative societal impact.\n\nEDIT: Removed sentence regarding concerns about reduction in training time.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "cIhEFgKO5ck", "reviewer": "Reviewer_5dEe", "summary": "This paper proposes a method called K-priors for adapting a model to newly coming data,  new regularization, or new architecture.  It suggests such adaption can be done by minimizing the Bregman divergence between features generated by previous model and current model,  as well as minimizing the Bregman divergence between parameters of previous model and current model. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty is incremental, as the form of Eq.7 is just a combination of knowledge distillation loss and a weight regularization, both of which are common in continual learning literature.  \nAnd the claims are not supported by the definition of the method.  According to Eq.9, only when M=D the gradient can recover the gradient of previous loss, which is not surprising at all. However, authors claim that data out of the previous distribution can be used to approximate the previous gradient (Line 133), how can this be guaranteed?\nAlso, the suggested way to select memorized samples is cumbersome and there are existing approaches proposed for sample selection in continual learning as well, e.g. [1],[2].  The authors didn't discuss or compare with any such related work.  And how far this approximation can be to the previous gradient is not analyzied either. So, how to achieve the claim of 'recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data'?  For a specified accuracy, how large the memory should be?\n\n[1]. Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., and Page-Caccia, L. Online Continual Learning with Maximal Interfered Retrieval. In Advances in Neural Information Processing Systems, pp. 11849–11860, 2019a.\n[2]. Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. Gradient-based Sample Selection for Online Continual Learning. In Advances in Neural Information Processing Systems, pp. 11816–11825, 2019b.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are stated above. No societal impact in my understanding. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "_OSD5Xmfxx7", "reviewer": "Reviewer_wqo5", "summary": "This paper proposes a new regularizer to adapt the prior knowledge through weight prior and the memory that is smaller than the entire dataset but identical to be learned correctly. Previous methods to do this have focused on the \"Add/Remove Data\" task, but the authors extended the adaptation tasks to adapt regularizer change or architecture change through the proposed regularizer, K-priors. It also could be applied to a variety of models from a linear model to deep learning. They described the connections between K-priors and previous works for adding/removing data for SVMs, knowledge distillation and continual learning for deep learning, and Bayesian learning. They mainly validated that K-priors outperform the model without K-priors but using the same memory selected through K-priors for logistic regression and classification tasks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors tried to contribute the unified regularizer that is based on the previous knowledge (when there is no previous knowledge, it would be used like$l_2-\\text{norm}$ weight regularizer). The previous knowledge exists as samples or trained parameters. They tried to use the two type knowledge with their prior. This paper has several strengths, one of them is that they tried to design the general prior that is not limited in deep learning, which is rare in those days (I don't think research on deep learning is not good, but diversity on research will make our community healthy more). Another is that they described the connections with previous works like knowledge distillation. However, the validations are on small size of experimental setting, and I am not sure about the stability of this prior. Another concern is about forgetting issue when updating the model multiple times. The memory is reset in every time, then the identical samples that are stored in memory will be removed and the model can forget that. However, it could be overcame through diving the entire memory to a sub-memory for each update I think. I agree too that the experiment for CIFAR-10 knowledge distillation (the dip in the middle in Fig 3 (d)) must be updated or analyzed more.\n\nQuestions\n- In line 64-65, how $k$-fold Cross-validation is related with adaptation task?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As I mentioned in the previous section, stability could be a limitation of this research, but I am not sure.\n\nWhen applying it to industrial or real applications, it sustains some samples as a memory, the right to be forgotten for the information on the internet may be infringed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 Hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a family of approaches for adapting pre-trained models to a variety of changes to the model architecture, training data or other aspects of the training setup. Given the cost of training models from scratch and the generality of the presented approach, this is a highly relevant piece of work with potential for impact.

After the discussion, three reviews recommend acceptance and one review recommends rejection. The negative review points to certain issues with the current version, but doesn't convince me that these issues rise to the level of rejection. After considering both positive and negative points, I lean towards recommending acceptance.

**Comments to the authors**: After reading the paper, I was left with the impression that clarity could be improved, and that there are places where details are missing. I see that some of these missing details and some additional experiments were supplied during the discussion with the reviewers. I would like to see these, together with any other reviewer feedback, incorporated into the final version of the paper.
Decision: Accept (Poster)