Knowledge-Adaptation Priors

1 Introduction . Machine-Learning ( ML ) at production often requires constant model updating which can have huge financial and environmental costs [ 19 , 44 ] . The production pipeline is continuously evolving , where new data are regularly pooled and labeled and old data become irrelevant . Regular tuning of hyperparameters is required to handle drifts [ 19 ] , and sometimes even the model class/architecture may need to change . Due to this , the model is frequently retrained , retested , and redeployed , which can be extremely costly , especially when the data and model sizes are large . The cost can be reduced if , instead of repeated retraining , the system can quickly adapt to incremental changes . Humans and animals can naturally use their prior knowledge to handle a wide-variety of changes in their surroundings , but such quick , wide , and accurate adaptation has been difficult to achieve in ML . In theory , this should be possible within a Bayesian framework where the posterior is used as the prior for the future , but exact Bayes is computationally challenging and the design of generic Bayesian priors has its own challenges [ 55 , 39 ] . In ML , simpler mechanisms are more popular , for example , in Support Vector Machines ( SVMs ) for adding/removing data [ 12 , 58 , 60 ] , and in deep learning for model compression [ 24 ] . Weight-priors are used in online learning [ 13 ] , and more recently for continual learning [ 30 , 40 , 47 , 33 , 62 ] , but they are not suited for many other tasks , such as model compression . In some settings , they also perform worse , for example , in continual learning when compared to memory-based strategies [ 34 , 46 , 45 , 9 ] . All these previous works apply to narrow , specific settings , and designing generic adaptation-mechanisms remains an open challenge . We present Knowledge-adaptation priors ( K-priors ) for the design of generic adaptation-mechanisms . The general principle of adaptation is to combine the weight and function-space divergences to faithfully reconstruct the gradient of the past . K-priors can handle a wide variety of adaptation tasks ( Fig . 1 , left ) and work for a range of models , such as generalized linear models , deep networks , and their Bayesian extensions . The principle unifies and generalizes many seemingly-unrelated existing works , for example , weight-priors [ 13 ] , knowledge distillation [ 24 ] , SVMs [ 12 , 35 , 58 ] , * Authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Change Regularizer K-priors Past Model Past Memory Add new data Remove old data Change Regularizer Change architecture Updated Model Adaptation tasks Past data New data Pa st mo de l Past memory Used in K-priors Add new data Remove old dataBatch K-prior V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) Memory size ( % of past data ) ( a ) Adult , logistic regression ( b ) USPS , logistic regression ( c ) USPS , neural network Add new data Remove old data Change regularizer Change model class Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) Memory size ( % of past data ) ( a ) Adult , logistic regression ( b ) USPS , logistic regression ( c ) USPS , neural network Add new data Remove old data Change regularizer Change model class Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) e ory size ( of st t ) ( a ) Adult , logistic regression ( b ) , logistic regression ( c ) USPS , neur l network Add n w d ta Remove old data Change regularizer Change model class e ory size ( of past t ) Memory size ( % of past data ) Memory size ( % of past data ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) V al id at io n ac c ( % ) Memory size ( % of past data ) ( a ) Adult , logistic regression ( b ) USPS , logistic regression ( c ) USPS , neural network ta Remove old data Change regularizer Change model class Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) Change architecture ( b ) USPS Add Data ( a ) over all ( c ) CIFAR-10 Add data V al id at io n ac c ( % ) Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) ( d ) CIFAR-10 Knowledge Distillation wb Figure 3 : ( a ) When compared at the Batch solution for the ‘ Add Data ’ task on USPS , weight priors give incorrect values of h0 ( f iw ) ( shown with black dots , each dot correspond to a data examples ) . Points on the diagonal means a perfect match which is the case for K-priors ( show with red dots ) . ( b ) Due to this , weight-priors ( green diamonds ) perform worse than K-priors ( red squares ) . ( c ) For the ’ Add data ’ task on CIFAR-10 , K-priors outperform replay ( blue circles ) , but performance can still be improved by using a temperature parameter ( dark-red triangles ) . ( d ) The same is true for knowledge distillation [ 21 ] , and we see that we can reduce memory size while still performing better than the student model . The dip in he middle , we believe , is due to suboptimal hyperparameter tuning . [ 29 ] with MLPs and 10-way classification on CIFAR-10 with CifarNet [ 57 ] , trained with the Adam319 optimizer [ 26 ] . In Figure 3 ( c ) we show one representative result for the ‘ Add data ’ task with CIFAR-320 10 , where we add a random 10 % of CIFAR-10 training data to the other 90 % ( mean and standard321 A similar result is shown in Figure 3 ( d ) for knowledge distillation ( = 0 but with a temperature325 parameter ) where we are distill from a CifarNet teacher to a LeNet5-style student ( details in Ap-326 pendix D ) . Here , K-priors with 100 % data is equivalent to Knowledge Distillation , but when we327 reduce the memory size using our method , we still outperform Batch ( which is trained from scratch328 on all data ) . There is a dip in performance at memory sizes of 10 % past data , which we believe is due329 to the use of a suboptimal ⌧ or . More empirical effort is require to tune various hyperparameters to330 get a consistent behavior at all memory sizes . Overall , our initial effort here suggests that K-priors331 can do better than Replay , and have potential to give better results with more hyperparameter tuning.332 6 Discussion333 In this paper , we proposed a class of new priors , called K-prior . We show general principles of334 obtaining accurate adaptation with K-priors which are based on accurate gradient reconstructions.335 These principles have many properties that a good prior is expected to have [ 50 ] . The prior applies336 to a wide-variety of adaptation tasks for a range of models , and helps us to connect many existing,337 seemingly-unrelated adaptation strategies in ML . Based on our adaptation principles , we derived338 practical methods to enable adaptation by tweaking models ’ predictions at a few past examples . This339 is analogous to adaptation in humans and animals where past experiences is used for new situations.340 In practice , the amount of required past memory seems sufficiently low for many tasks . Overall,341 K-priors provide an intuitive yet practical mechanisms for generic adaptation in ML.342 The financial and environmental costs of retraining are a huge concern for ML practitioners , which343 can be reduced with quick adaptations . During this work , we realized how little work has been done344 on this topic . The current pipelines and designs are specialized for an offline , static setting . Our345 approach here pushes towards a simpler design which will support a more dynamic setting . The346 approach can eventually lead to new systems that learn quickly and flexibly , and also act sensibly347 across a wide range of tasks . This opens a path towards systems that learn incrementally in a continual348 fashion , with the potential to fundamentally change the way ML is used in scientific and industrial349 applications . We hope that this work will help others to do more work towards this goal in the future.350 We ourselves will continue to push this work in that direction.351 9 ( b ) USPS Add Data ( a ) over all ( c ) CIFAR-10 Add data V al id at io n ac c ( % ) Memory size ( % of past data ) Memory size ( % of past data ) Memory size ( % of past data ) ( d ) CIFAR-10 Knowledge Distillation wb Figure 3 : ( a ) When compared at the Batch solution for the ‘ Add Data ’ task on USPS , weight priors give incorrect values of h0 ( f iw ) ( shown with black dots , each dot correspond to a data examples ) . Points on the diagonal means a perfect match which is the case for K-priors ( show with red dots ) . ( b ) Due to this , weight-priors ( green diamonds ) perform worse than K-priors ( red squares ) . ( c ) For the ’ Add data ’ task on CIFAR-10 , K-priors utperform replay ( blue circles ) , but performance can still be improved by usi g a temperatur parameter ( dark-red riangles ) . ( d ) The same is true for knowledge distillation [ 21 ] , and we see that we can reduce memory size whil s ill performing better than the student model . The dip in the middle , we believe , is d e to suboptimal hyp parameter tuning . [ 29 ] with MLPs and 10-way classification on CIFAR-10 with CifarNet [ 57 ] , trained with the Adam319 optimizer [ 26 ] . In Figure 3 ( c ) we show one representative result for the ‘ Add data ’ task with CIFAR-320 10 , where we add a random 10 % of CIFAR-10 training data to the other 90 % ( mean and standard321 deviation over 3 runs ) . Although vanilla K-priors outperform Replay , there is now a bigger gap322 between K-prior and Batch even with 50 % past data stored . However , when we use a temperature323 ( similar to knowledge distillation in ( 14 ) but with the weight term included ) , K-priors improves.324 A similar result is shown in Figure 3 ( d ) for knowledge distillation ( = 0 but with a temperature325 parameter ) where we are distill from a CifarNet teacher to a LeNet5-style student ( details in Ap-326 pendix D ) . Here , K-priors with 100 % data is equivalent to Knowledge Distillation , but when we327 reduce the memory size using our method , we still outperform Batch ( which is trained from scratch328 on all data ) . There is a dip in performance at memory sizes of 10 % past data , which we believe is due329 to the use of a suboptimal ⌧ or . More empirical effort is require to tune various hyperparameters to330 get a consistent behavior at all memory sizes . Overall , our initial effort here suggests that K-priors331 can do better than Replay , and have potential to give better results with more hyperparameter tuning.332 6 Discussion333 In this paper , we proposed a class of new priors , called K-prior . We show general principles of334 obtaining accurate adaptation with K-priors which are based on accurate gradient reconstructions.335 These principles have many properties that a good prior is expected to have [ 50 ] . The prior applies336 to a wide-variety of adaptation tasks for a range of models , and helps us to connect many existing,337 seemingly-unrelated adaptation strategies in ML . Based on our adaptation principles , we derived338 practical methods to enable adaptation by tweaking models ’ predictions at a few past examples . This339 is analogous to adaptation in humans and animals where past experiences is used for new situations.340 In practice , the amount of required past memory seems sufficiently low for many tasks . Overall,341 K-priors provide an intuitive yet practical mechanisms for generic adaptation in ML.342 The financial and environmental costs of retraining are a huge concern for ML practitioners , which343 can be reduced with quick adaptations . During this work , we realized how little work has been done344 on this topic . The current pipelines and designs are specialized for an offline , static setting . Our345 approach here pushes towards a simpler design which will support a more dynamic setting . The346 approach can eventually lead to new systems that learn quickly and flexibly , and also act sensibly347 across a wide range of tasks . This opens a path towards systems that learn incrementally in a continual348 fashion , with the potential to fundamentally change the way ML is used in scientific and industrial349 applications . We hope that this work will help others to do more work towards this goal in the future.350 We ourselves will continue to push this work in that direction.351 9 online Gaussian Processes [ 17 ] , and continual learning [ 36 , 45 , 9 ] . It leads to natural adaptationmechanisms where models ’ predictions need to be readjusted only at a handful of past experiences ( Fig . 1 , middle ) . This is quick and easy to train with first-order optimization methods and , by choosing a sufficiently large memory , we obtain results similar to etraining-fro -scratch ( Fig . 1 , ight ) . Code is available at https : //github.com/team-approx-b yes/kpriors .

Summary: This paper proposes a new regularizer to adapt the prior knowledge through weight prior and the memory that is smaller than the entire dataset but identical to be learned correctly. Previous methods to do this have focused on the "Add/Remove Data" task, but the authors extended the adaptation tasks to adapt regularizer change or architecture change through the proposed regularizer, K-priors. It also could be applied to a variety of models from a linear model to deep learning. They described the connections between K-priors and previous works for adding/removing data for SVMs, knowledge distillation and continual learning for deep learning, and Bayesian learning. They mainly validated that K-priors outperform the model without K-priors but using the same memory selected through K-priors for logistic regression and classification tasks.