Knowledge-Adaptation Priors

1 Introduction . Machine-Learning ( ML ) at production often requires constant model updating which can have huge financial and environmental costs [ 19 , 44 ] . The production pipeline is continuously evolving , where new data are regularly pooled and labeled and old data become irrelevant . Regular tuning of hyperparameters is required to handle drifts [ 19 ] , and sometimes even the model class/architecture may need to change . Due to this , the model is frequently retrained , retested , and redeployed , which can be extremely costly , especially when the data and model sizes are large . The cost can be reduced if , instead of repeated retraining , the system can quickly adapt to incremental changes . Humans and animals can naturally use their prior knowledge to handle a wide-variety of changes in their surroundings , but such quick , wide , and accurate adaptation has been difficult to achieve in ML . In theory , this should be possible within a Bayesian framework where the posterior is used as the prior for the future , but exact Bayes is computationally challenging and the design of generic Bayesian priors has its own challenges [ 55 , 39 ] . In ML , simpler mechanisms are more popular , for example , in Support Vector Machines ( SVMs ) for adding/removing data [ 12 , 58 , 60 ] , and in deep learning for model compression [ 24 ] . Weight-priors are used in online learning [ 13 ] , and more recently for continual learning [ 30 , 40 , 47 , 33 , 62 ] , but they are not suited for many other tasks , such as model compression . In some settings , they also perform worse , for example , in continual learning when compared to memory-based strategies [ 34 , 46 , 45 , 9 ] . All these previous works apply to narrow , specific settings , and designing generic adaptation-mechanisms remains an open challenge . We present Knowledge-adaptation priors ( K-priors ) for the design of generic adaptation-mechanisms . The general principle of adaptation is to combine the weight and function-space divergences to faithfully reconstruct the gradient of the past . K-priors can handle a wide variety of adaptation tasks ( Fig . 1 , left ) and work for a range of models , such as generalized linear models , deep networks , and their Bayesian extensions . The principle unifies and generalizes many seemingly-unrelated existing works , for example , weight-priors [ 13 ] , knowledge distillation [ 24 ] , SVMs [ 12 , 35 , 58 ] , * Authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Change Regularizer K-priors Past Model Past Memory Add new data Remove old data Change Regularizer Change architecture Updated Model Adaptation tasks Past data New data Pa st mo de l Past memory Used in K-priors Add new data Remove old dataBatch K-prior V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) Memory size ( % of past data ) ( a ) Adult , logistic regression ( b ) USPS , logistic regression ( c ) USPS , neural network Add new data Remove old data Change regularizer Change model class Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) Memory size ( % of past data ) ( a ) Adult , logistic regression ( b ) USPS , logistic regression ( c ) USPS , neural network Add new data Remove old data Change regularizer Change model class Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) e ory size ( of st t ) ( a ) Adult , logistic regression ( b ) , logistic regression ( c ) USPS , neur l network Add n w d ta Remove old data Change regularizer Change model class e ory size ( of past t ) Memory size ( % of past data ) Memory size ( % of past data ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) Memory size ( % of past data ) ( a ) Adult , logistic regression ( b ) USPS , logistic regression ( c ) USPS , neural network ta Remove old data Change regularizer Change model class Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) Change architecture ( b ) USPS Add Data ( a ) over all ( c ) CIFAR-10 Add data V al id at io n ac c ( % ) Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) ( d ) CIFAR-10 Knowledge Distillation wb Figure 3 : ( a ) When compared at the Batch solution for the ‘ Add Data ’ task on USPS , weight priors give incorrect values of h0 ( f iw ) ( shown with black dots , each dot correspond to a data examples ) . Points on the diagonal means a perfect match which is the case for K-priors ( show with red dots ) . ( b ) Due to this , weight-priors ( green diamonds ) perform worse than K-priors ( red squares ) . ( c ) For the ’ Add data ’ task on CIFAR-10 , K-priors outperform replay ( blue circles ) , but performance can still be improved by using a temperature parameter ( dark-red triangles ) . ( d ) The same is true for knowledge distillation [ 21 ] , and we see that we can reduce memory size while still performing better than the student model . The dip in he middle , we believe , is due to suboptimal hyperparameter tuning . [ 29 ] with MLPs and 10-way classification on CIFAR-10 with CifarNet [ 57 ] , trained with the Adam319 optimizer [ 26 ] . In Figure 3 ( c ) we show one representative result for the ‘ Add data ’ task with CIFAR-320 10 , where we add a random 10 % of CIFAR-10 training data to the other 90 % ( mean and standard321 A similar result is shown in Figure 3 ( d ) for knowledge distillation ( = 0 but with a temperature325 parameter ) where we are distill from a CifarNet teacher to a LeNet5-style student ( details in Ap-326 pendix D ) . Here , K-priors with 100 % data is equivalent to Knowledge Distillation , but when we327 reduce the memory size using our method , we still outperform Batch ( which is trained from scratch328 on all data ) . There is a dip in performance at memory sizes of 10 % past data , which we believe is due329 to the use of a suboptimal ⌧ or . More empirical effort is require to tune various hyperparameters to330 get a consistent behavior at all memory sizes . Overall , our initial effort here suggests that K-priors331 can do better than Replay , and have potential to give better results with more hyperparameter tuning.332 6 Discussion333 In this paper , we proposed a class of new priors , called K-prior . We show general principles of334 obtaining accurate adaptation with K-priors which are based on accurate gradient reconstructions.335 These principles have many properties that a good prior is expected to have [ 50 ] . The prior applies336 to a wide-variety of adaptation tasks for a range of models , and helps us to connect many existing,337 seemingly-unrelated adaptation strategies in ML . Based on our adaptation principles , we derived338 practical methods to enable adaptation by tweaking models ’ predictions at a few past examples . This339 is analogous to adaptation in humans and animals where past experiences is used for new situations.340 In practice , the amount of required past memory seems sufficiently low for many tasks . Overall,341 K-priors provide an intuitive yet practical mechanisms for generic adaptation in ML.342 The financial and environmental costs of retraining are a huge concern for ML practitioners , which343 can be reduced with quick adaptations . During this work , we realized how little work has been done344 on this topic . The current pipelines and designs are specialized for an offline , static setting . Our345 approach here pushes towards a simpler design which will support a more dynamic setting . The346 approach can eventually lead to new systems that learn quickly and flexibly , and also act sensibly347 across a wide range of tasks . This opens a path towards systems that learn incrementally in a continual348 fashion , with the potential to fundamentally change the way ML is used in scientific and industrial349 applications . We hope that this work will help others to do more work towards this goal in the future.350 We ourselves will continue to push this work in that direction.351 9 ( b ) USPS Add Data ( a ) over all ( c ) CIFAR-10 Add data V al id at io n ac c ( % ) Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) ( d ) CIFAR-10 Knowledge Distillation wb Figure 3 : ( a ) When compared at the Batch solution for the ‘ Add Data ’ task on USPS , weight priors give incorrect values of h0 ( f iw ) ( shown with black dots , each dot correspond to a data examples ) . Points on the diagonal means a perfect match which is the case for K-priors ( show with red dots ) . ( b ) Due to this , weight-priors ( green diamonds ) perform worse than K-priors ( red squares ) . ( c ) For the ’ Add data ’ task on CIFAR-10 , K-priors utperform replay ( blue circles ) , but performance can still be improved by usi g a temperatur parameter ( dark-red riangles ) . ( d ) The same is true for knowledge distillation [ 21 ] , and we see that we can reduce memory size whil s ill performing better than the student model . The dip in the middle , we believe , is d e to suboptimal hyp parameter tuning . [ 29 ] with MLPs and 10-way classification on CIFAR-10 with CifarNet [ 57 ] , trained with the Adam319 optimizer [ 26 ] . In Figure 3 ( c ) we show one representative result for the ‘ Add data ’ task with CIFAR-320 10 , where we add a random 10 % of CIFAR-10 training data to the other 90 % ( mean and standard321 deviation over 3 runs ) . Although vanilla K-priors outperform Replay , there is now a bigger gap322 between K-prior and Batch even with 50 % past data stored . However , when we use a temperature323 ( similar to knowledge distillation in ( 14 ) but with the weight term included ) , K-priors improves.324 A similar result is shown in Figure 3 ( d ) for knowledge distillation ( = 0 but with a temperature325 parameter ) where we are distill from a CifarNet teacher to a LeNet5-style student ( details in Ap-326 pendix D ) . Here , K-priors with 100 % data is equivalent to Knowledge Distillation , but when we327 reduce the memory size using our method , we still outperform Batch ( which is trained from scratch328 on all data ) . There is a dip in performance at memory sizes of 10 % past data , which we believe is due329 to the use of a suboptimal ⌧ or . More empirical effort is require to tune various hyperparameters to330 get a consistent behavior at all memory sizes . Overall , our initial effort here suggests that K-priors331 can do better than Replay , and have potential to give better results with more hyperparameter tuning.332 6 Discussion333 In this paper , we proposed a class of new priors , called K-prior . We show general principles of334 obtaining accurate adaptation with K-priors which are based on accurate gradient reconstructions.335 These principles have many properties that a good prior is expected to have [ 50 ] . The prior applies336 to a wide-variety of adaptation tasks for a range of models , and helps us to connect many existing,337 seemingly-unrelated adaptation strategies in ML . Based on our adaptation principles , we derived338 practical methods to enable adaptation by tweaking models ’ predictions at a few past examples . This339 is analogous to adaptation in humans and animals where past experiences is used for new situations.340 In practice , the amount of required past memory seems sufficiently low for many tasks . Overall,341 K-priors provide an intuitive yet practical mechanisms for generic adaptation in ML.342 The financial and environmental costs of retraining are a huge concern for ML practitioners , which343 can be reduced with quick adaptations . During this work , we realized how little work has been done344 on this topic . The current pipelines and designs are specialized for an offline , static setting . Our345 approach here pushes towards a simpler design which will support a more dynamic setting . The346 approach can eventually lead to new systems that learn quickly and flexibly , and also act sensibly347 across a wide range of tasks . This opens a path towards systems that learn incrementally in a continual348 fashion , with the potential to fundamentally change the way ML is used in scientific and industrial349 applications . We hope that this work will help others to do more work towards this goal in the future.350 We ourselves will continue to push this work in that direction.351 9 online Gaussian Processes [ 17 ] , and continual learning [ 36 , 45 , 9 ] . It leads to natural adaptationmechanisms where models ’ predictions need to be readjusted only at a handful of past experiences ( Fig . 1 , middle ) . This is quick and easy to train with first-order optimization methods and , by choosing a sufficiently large memory , we obtain results similar to etraining-fro -scratch ( Fig . 1 , ight ) . Code is available at https : //github.com/team-approx-b yes/kpriors .

Summary: - This paper tackles an important problem: Suppose we have trained a model on a dataset, but we wish to adapt it quickly. For example, we might add new training points, delete training points, or wish to change the model and regularization. We could re-train a model, but that's slow - can we get a similar result but quicker and using less memory? - This paper proposes K-prior: regularizing the model towards the original model in both weight space and predictions. - They show that training on the K-prior objective on the entire dataset recovers the original model - which would not be true if we only have weight space regularization, or only have prediction regularization. - In practice, to save memory, they only form the prior on a few points, and they describe heuristics to choose these points, with some mathematical justification - They show that this methods work well in practice, compared to just replaying the points