K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning

1 INTRODUCTION . Deep neural networks have revolutionized many areas of machine intelligence and are now used for many vision tasks that even few years ago were considered nearly impenetrable ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; Liu et al. , 2016 ) . Advances in neural networks and hardware is resulting in much of the computation being shifted to consumer devices , delivering faster response , and better security and privacy guarantees ( Konečný et al. , 2016 ; Howard et al. , 2017 ) . As the space of deep learning applications expands and starts to personalize , there is a growing need for the ability to quickly build and customize models . While model sizes have dropped dramatically from > 50M parameters of the pioneering work of AlexNet ( Krizhevsky et al. , 2012 ) and VGG ( Simonyan & Zisserman , 2014 ) to < 5M of the recent Mobilenet ( Sandler et al. , 2018 ; Howard et al. , 2017 ) and ShuffleNet ( Zhang et al. , 2017 ; Ma et al. , 2018 ) , the accuracy of models has been improving . However , delivering , maintaining and updating hundreds of models on the embedded device is still a significant expense in terms of bandwidth , energy and storage costs . While there still might be space for improvement in designing smaller models , in this paper we explore a different angle : we would like to be able to build models that require only a few parameters to be trained in order to be re-purposed to a different task , with minimal loss in accuracy compared to a model trained from scratch . While there is ample existing work on compressing models and learning as few weights as possible ( Rosenfeld & Tsotsos , 2018 ; Sandler et al. , 2018 ; Howard et al. , 2017 ) to solve a single task , to the best of our awareness , there is no prior work that tries to minimize the number of model parameters when solving many tasks together . Our contribution is a novel learning paradigm in which each task carries its own model patch – a small set of parameters – that , along with a shared set of parameters constitutes the model for that task ( for a visual description of the idea , see Figure 1 , left side ) . We put this idea to use in two scenarios : a ) in transfer learning , by fine-tuning only the model patch for new tasks , and b ) in multi-task learning , where each task performs gradient updates to both its own model patch , and the ∗Work done while at Google . shared parameters . In our experiments ( Section 5 ) , the largest patch that we used is smaller than 10 % of the size of the entire model . We now describe our contribution in detail . Transfer learning We demonstrate that , by fine-tuning less than 35K parameters in MobilenetV2 ( Sandler et al. , 2018 ) and InceptionV3 ( Szegedy et al. , 2016 ) , our method leads to significant accuracy improvements over fine-tuning only the last layer ( 102K-1.2M parameters , depending on the number of classes ) on multiple transfer learning tasks . When combined with fine-tuning the last layer , we train less than 10 % of the model ’ s parameters in total.We also show the effectiveness of our method over last-layer-based fine-tuning on transfer learning between completely different problems , namely COCO-trained SSD model ( Liu et al. , 2016 ) to classification over ImageNet ( Deng et al. , 2009 ) . Multi-task learning We explore a multi-task learning paradigm wherein multiple models that share most of the parameters are trained simultaneously ( see Figure 1 , right side ) . Each model has a task-specific model patch . Training is done in a distributed manner ; each task is assigned a subset of available workers that send independent gradient updates to both shared and task-specific parameters using standard optimization algorithms . Our results show that simultaneously training two such MobilenetV2 ( Sandler et al. , 2018 ) models on ImageNet ( Deng et al. , 2009 ) and Places365 ( Zhou et al. , 2017 ) reach accuracies comparable to , and sometimes higher than individually trained models . Domain adaptation We apply our multi-task learning paradigm to domain adaptation . For ImageNet ( Deng et al. , 2009 ) , we show that we can simultaneously train MobilenetV2 ( Sandler et al. , 2018 ) models operating at 5 different resolution scales , 224 , 192 , 160 , 128 and 96 , while sharing more than 98 % of the parameters and resulting in the same or higher accuracy as individually trained models . This has direct practical benefit in power-constrained operation , where an application can switch to a lower resolution to save on latency/power , without needing to ship separate models and having to make that trade-off decision at the application design time . The cascade algorithm from Streeter ( 2018 ) can further be used to reduce the average running time by about 15 % without loss in accuracy . The rest of the paper is organized as follows : we describe our method in Section 2 and discuss related work in Section 3 . In Section 4 , we present simple mathematical intuition that contrasts the expressiveness of logit-only fine-tuning and that of our method . Finally , in Section 5 , we present detailed experimental results . 2 METHOD . The central concept in our method is that of a model patch . It is essentially a small set of perchannel transformations that are dispersed throughout the network resulting in only a tiny increase in the number of model parameters . Suppose a deep networkM is a sequence of layers represented by their parameters ( weights , biases ) , W1 , . . . , Wn . We ignore non-trainable layers ( e.g. , some kinds of activations ) in this formulation . A model patch P is a set of parameters W′i1 , . . . , W ′ ik , 1 ≤ i1 , . . . , ik ≤ n that , when applied to M , adds layers at positions i1 , . . . , in . Thus , a patched model M′ = W1 , . . . , Wi1 , W′i1 , . . . , Win , W ′ in , . . . , Wn In this paper , we introduce two kinds of patches . We will see below that they can be folded with the other layers in the network , eliminating the need to perform any explicit addition of layers . In Section 5 , we shed some light on why the particular choice of these patches is important . Scale-and-bias patch This patch applies per-channel scale and bias to every layer in the network . In practice this transformations can often be absorbed into normalization layer such as Batch Normalization ( Ioffe & Szegedy , 2015 ) . Let X be an activation tensor . Then , the batch-normalized version of X BN ( X ) = γ X− µ ( X ) σ ( X ) + β where µ ( X ) , σ ( X ) are mean and standard deviation computed per minibatch , and γ , β are learned via backpropagation . These statistics are computed as mini-batch average , while during inference they are computed using global averages . The scale-and-bias patch corresponds to all the γ , β , µ , σ in the network . Using BN as the model patch also satisfies the criterion that the patch size should be small . For instance , the BN parameters in both MobilenetV2 ( Sandler et al. , 2018 ) and InceptionV3 network performing classification on ImageNet amounts to less than 40K parameters , of about 1 % for MobilenetV2 that has 3.5 million Parameters , and less than 0.2 % for Inception V3 that has 25 million parameters . While we utilize batch normalization in this paper , we note that this is merely an implementation detail and we can use explicit biases and scales with similar results . Depthwise-convolution patch The purpose of this patch is to re-learn spatial convolution filters in a network . Depth-wise separable convolutions were introduced in deep neural networks as way to reduce number of parameters without losing much accuracy ( Howard et al. , 2017 ; Chollet , 2017 ) . They were further developed in Sandler et al . ( 2018 ) by adding linear bottlenecks and expansions . In depthwise separable convolutions , a standard convolution is decomposed into two layers : a depthwise convolution layer , that applies one convolutional filter per input channel , and a pointwise layer that computes the final convolutional features by linearly combining the depthwise convolutional layers ’ output across channels . We find that the set of depthwise convolution layers can be repurposed as a model patch . They are also lightweight - for instance , they account for less than 3 % of MobilenetV2 ’ s parameters when training on ImageNet . Next , we describe how model patches can be used in transfer and multi-task learning . Transfer learning In transfer learning , the task is to adapt a pretrained model to a new task . Since the output space of the new task is different , it necessitates re-learning the last layer . Following our approach , we apply a model patch and train the patched parameters , optionally also the last layer . The rest of the parameters are left unchanged . In Section 5 , we discuss the inclusion/exclusion of the last layer . When the last layer is not trained , it is fixed to its random initial value . Multitask learning We aim to simultaneously , but independently , train multiple neural networks that share most weights . Unlike in transfer learning , where a large fraction of the weights are kept frozen , here we learn all the weights . However , each task carries its own model patch , and trains a patched model . By training all the parameters , this setting offers more adaptability to tasks while not compromising on the total number of parameters . To implement multi-task learning , we use the distributed TensorFlow paradigm1 : a central parameter server receives gradient updates from each of the workers and updates the weights . Each worker reads the input , computes the loss and sends gradients to the parameter server . We allow subsets of workers to train different tasks ; workers thus may have different computational graphs , and taskspecific input pipelines and loss functions . A visual depiction of this setting is shown in Figure 1 . 1https : //www.tensorflow.org/guide/extend/architecture 3 RELATED WORK . One family of approaches ( Yosinski et al. , 2014 ; Donahue et al. , 2014 ) widely used by practitioners for domain adaptation and transfer learning is based on fine-tuning only the last layer ( or sometimes several last layers ) of a neural network to solve a new task . Fine-tuning the last layer is equivalent to training a linear classifier on top of existing features . This is typically done by running SGD while keeping the rest of the network fixed , however other methods such as SVM has been explored as well ( Kim et al. , 2013 ) . It has been repeatedly shown that this approach often works best for similar tasks ( for example , see Donahue et al . ( 2014 ) ) . Another frequently used approach is to use full fine-tuning ( Cui et al. , 2018 ) where a pretrained model is simply used as a warm start for the training process . While this often leads to significantly improved accuracy over last-layer fine-tuning , downsides are that 1 ) it requires one to create and store a full model for each new task , and 2 ) it may lead to overfitting when there is limited data . In this work , we are primarily interested in approaches that allow one to produce highly accurate models while reusing a large fraction of the weights of the original model , which also addresses the overfitting issue . While the core idea of our method is based on learning small model patches , we see significant boost in performance when we fine-tune the patch along with last layer ( Section 5 ) . This result is somewhat in contrast with Hoffer et al . ( 2018 ) , where the authors show that the linear classifier ( last layer ) does not matter when training full networks . Mapping out the conditions of when a linear classifier can be replaced with a random embedding is an important open question . Li et al . ( 2016 ) show that re-computing batch normalization statistics for different domains helps to improve accuracy . In Rosenfeld & Tsotsos ( 2018 ) it was suggested that learning batch normalization layers in an otherwise randomly initialized network is sufficient to build non-trivial models . Recomputing batch normalization statistics is also frequently used for model quantization where it prevents the model activation space from drifting ( Krishnamoorthi , 2018 ) . In the present work , we significantly broaden and unify the scope of the idea and scale up the approach by performing transfer and multi-task learning across completely different tasks , providing a powerful tool for many practical applications . Our work has interesting connections to meta-learning ( Nichol & Schulman , 2018 ; Finn et al. , 2017 ; Chen et al. , 2018 ) . For instance , when training data is not small , one can allow each task to carry a small model patch in the Reptile algorithm of Nichol & Schulman ( 2018 ) in order to increase expressivity at low cost .

Summary: the paper introduces a new way of fine-tuning neural networks. Instead of re-training the whole model or fine-tuning the last few layers, the authors propose to fine-tune a small set of model patches that affect the network at different layers. The results show that this way of fine-tuning is superior to above mentioned typical ways either in accuracy or in the number of tuned parameters in three different settings: transfer learning, multi-task learning and domain adaptation.