K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning

1 INTRODUCTION . Deep neural networks have revolutionized many areas of machine intelligence and are now used for many vision tasks that even few years ago were considered nearly impenetrable ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; Liu et al. , 2016 ) . Advances in neural networks and hardware is resulting in much of the computation being shifted to consumer devices , delivering faster response , and better security and privacy guarantees ( Konečný et al. , 2016 ; Howard et al. , 2017 ) . As the space of deep learning applications expands and starts to personalize , there is a growing need for the ability to quickly build and customize models . While model sizes have dropped dramatically from > 50M parameters of the pioneering work of AlexNet ( Krizhevsky et al. , 2012 ) and VGG ( Simonyan & Zisserman , 2014 ) to < 5M of the recent Mobilenet ( Sandler et al. , 2018 ; Howard et al. , 2017 ) and ShuffleNet ( Zhang et al. , 2017 ; Ma et al. , 2018 ) , the accuracy of models has been improving . However , delivering , maintaining and updating hundreds of models on the embedded device is still a significant expense in terms of bandwidth , energy and storage costs . While there still might be space for improvement in designing smaller models , in this paper we explore a different angle : we would like to be able to build models that require only a few parameters to be trained in order to be re-purposed to a different task , with minimal loss in accuracy compared to a model trained from scratch . While there is ample existing work on compressing models and learning as few weights as possible ( Rosenfeld & Tsotsos , 2018 ; Sandler et al. , 2018 ; Howard et al. , 2017 ) to solve a single task , to the best of our awareness , there is no prior work that tries to minimize the number of model parameters when solving many tasks together . Our contribution is a novel learning paradigm in which each task carries its own model patch – a small set of parameters – that , along with a shared set of parameters constitutes the model for that task ( for a visual description of the idea , see Figure 1 , left side ) . We put this idea to use in two scenarios : a ) in transfer learning , by fine-tuning only the model patch for new tasks , and b ) in multi-task learning , where each task performs gradient updates to both its own model patch , and the ∗Work done while at Google . shared parameters . In our experiments ( Section 5 ) , the largest patch that we used is smaller than 10 % of the size of the entire model . We now describe our contribution in detail . Transfer learning We demonstrate that , by fine-tuning less than 35K parameters in MobilenetV2 ( Sandler et al. , 2018 ) and InceptionV3 ( Szegedy et al. , 2016 ) , our method leads to significant accuracy improvements over fine-tuning only the last layer ( 102K-1.2M parameters , depending on the number of classes ) on multiple transfer learning tasks . When combined with fine-tuning the last layer , we train less than 10 % of the model ’ s parameters in total.We also show the effectiveness of our method over last-layer-based fine-tuning on transfer learning between completely different problems , namely COCO-trained SSD model ( Liu et al. , 2016 ) to classification over ImageNet ( Deng et al. , 2009 ) . Multi-task learning We explore a multi-task learning paradigm wherein multiple models that share most of the parameters are trained simultaneously ( see Figure 1 , right side ) . Each model has a task-specific model patch . Training is done in a distributed manner ; each task is assigned a subset of available workers that send independent gradient updates to both shared and task-specific parameters using standard optimization algorithms . Our results show that simultaneously training two such MobilenetV2 ( Sandler et al. , 2018 ) models on ImageNet ( Deng et al. , 2009 ) and Places365 ( Zhou et al. , 2017 ) reach accuracies comparable to , and sometimes higher than individually trained models . Domain adaptation We apply our multi-task learning paradigm to domain adaptation . For ImageNet ( Deng et al. , 2009 ) , we show that we can simultaneously train MobilenetV2 ( Sandler et al. , 2018 ) models operating at 5 different resolution scales , 224 , 192 , 160 , 128 and 96 , while sharing more than 98 % of the parameters and resulting in the same or higher accuracy as individually trained models . This has direct practical benefit in power-constrained operation , where an application can switch to a lower resolution to save on latency/power , without needing to ship separate models and having to make that trade-off decision at the application design time . The cascade algorithm from Streeter ( 2018 ) can further be used to reduce the average running time by about 15 % without loss in accuracy . The rest of the paper is organized as follows : we describe our method in Section 2 and discuss related work in Section 3 . In Section 4 , we present simple mathematical intuition that contrasts the expressiveness of logit-only fine-tuning and that of our method . Finally , in Section 5 , we present detailed experimental results . 2 METHOD . The central concept in our method is that of a model patch . It is essentially a small set of perchannel transformations that are dispersed throughout the network resulting in only a tiny increase in the number of model parameters . Suppose a deep networkM is a sequence of layers represented by their parameters ( weights , biases ) , W1 , . . . , Wn . We ignore non-trainable layers ( e.g. , some kinds of activations ) in this formulation . A model patch P is a set of parameters W′i1 , . . . , W ′ ik , 1 ≤ i1 , . . . , ik ≤ n that , when applied to M , adds layers at positions i1 , . . . , in . Thus , a patched model M′ = W1 , . . . , Wi1 , W′i1 , . . . , Win , W ′ in , . . . , Wn In this paper , we introduce two kinds of patches . We will see below that they can be folded with the other layers in the network , eliminating the need to perform any explicit addition of layers . In Section 5 , we shed some light on why the particular choice of these patches is important . Scale-and-bias patch This patch applies per-channel scale and bias to every layer in the network . In practice this transformations can often be absorbed into normalization layer such as Batch Normalization ( Ioffe & Szegedy , 2015 ) . Let X be an activation tensor . Then , the batch-normalized version of X BN ( X ) = γ X− µ ( X ) σ ( X ) + β where µ ( X ) , σ ( X ) are mean and standard deviation computed per minibatch , and γ , β are learned via backpropagation . These statistics are computed as mini-batch average , while during inference they are computed using global averages . The scale-and-bias patch corresponds to all the γ , β , µ , σ in the network . Using BN as the model patch also satisfies the criterion that the patch size should be small . For instance , the BN parameters in both MobilenetV2 ( Sandler et al. , 2018 ) and InceptionV3 network performing classification on ImageNet amounts to less than 40K parameters , of about 1 % for MobilenetV2 that has 3.5 million Parameters , and less than 0.2 % for Inception V3 that has 25 million parameters . While we utilize batch normalization in this paper , we note that this is merely an implementation detail and we can use explicit biases and scales with similar results . Depthwise-convolution patch The purpose of this patch is to re-learn spatial convolution filters in a network . Depth-wise separable convolutions were introduced in deep neural networks as way to reduce number of parameters without losing much accuracy ( Howard et al. , 2017 ; Chollet , 2017 ) . They were further developed in Sandler et al . ( 2018 ) by adding linear bottlenecks and expansions . In depthwise separable convolutions , a standard convolution is decomposed into two layers : a depthwise convolution layer , that applies one convolutional filter per input channel , and a pointwise layer that computes the final convolutional features by linearly combining the depthwise convolutional layers ’ output across channels . We find that the set of depthwise convolution layers can be repurposed as a model patch . They are also lightweight - for instance , they account for less than 3 % of MobilenetV2 ’ s parameters when training on ImageNet . Next , we describe how model patches can be used in transfer and multi-task learning . Transfer learning In transfer learning , the task is to adapt a pretrained model to a new task . Since the output space of the new task is different , it necessitates re-learning the last layer . Following our approach , we apply a model patch and train the patched parameters , optionally also the last layer . The rest of the parameters are left unchanged . In Section 5 , we discuss the inclusion/exclusion of the last layer . When the last layer is not trained , it is fixed to its random initial value . Multitask learning We aim to simultaneously , but independently , train multiple neural networks that share most weights . Unlike in transfer learning , where a large fraction of the weights are kept frozen , here we learn all the weights . However , each task carries its own model patch , and trains a patched model . By training all the parameters , this setting offers more adaptability to tasks while not compromising on the total number of parameters . To implement multi-task learning , we use the distributed TensorFlow paradigm1 : a central parameter server receives gradient updates from each of the workers and updates the weights . Each worker reads the input , computes the loss and sends gradients to the parameter server . We allow subsets of workers to train different tasks ; workers thus may have different computational graphs , and taskspecific input pipelines and loss functions . A visual depiction of this setting is shown in Figure 1 . 1https : //www.tensorflow.org/guide/extend/architecture 3 RELATED WORK . One family of approaches ( Yosinski et al. , 2014 ; Donahue et al. , 2014 ) widely used by practitioners for domain adaptation and transfer learning is based on fine-tuning only the last layer ( or sometimes several last layers ) of a neural network to solve a new task . Fine-tuning the last layer is equivalent to training a linear classifier on top of existing features . This is typically done by running SGD while keeping the rest of the network fixed , however other methods such as SVM has been explored as well ( Kim et al. , 2013 ) . It has been repeatedly shown that this approach often works best for similar tasks ( for example , see Donahue et al . ( 2014 ) ) . Another frequently used approach is to use full fine-tuning ( Cui et al. , 2018 ) where a pretrained model is simply used as a warm start for the training process . While this often leads to significantly improved accuracy over last-layer fine-tuning , downsides are that 1 ) it requires one to create and store a full model for each new task , and 2 ) it may lead to overfitting when there is limited data . In this work , we are primarily interested in approaches that allow one to produce highly accurate models while reusing a large fraction of the weights of the original model , which also addresses the overfitting issue . While the core idea of our method is based on learning small model patches , we see significant boost in performance when we fine-tune the patch along with last layer ( Section 5 ) . This result is somewhat in contrast with Hoffer et al . ( 2018 ) , where the authors show that the linear classifier ( last layer ) does not matter when training full networks . Mapping out the conditions of when a linear classifier can be replaced with a random embedding is an important open question . Li et al . ( 2016 ) show that re-computing batch normalization statistics for different domains helps to improve accuracy . In Rosenfeld & Tsotsos ( 2018 ) it was suggested that learning batch normalization layers in an otherwise randomly initialized network is sufficient to build non-trivial models . Recomputing batch normalization statistics is also frequently used for model quantization where it prevents the model activation space from drifting ( Krishnamoorthi , 2018 ) . In the present work , we significantly broaden and unify the scope of the idea and scale up the approach by performing transfer and multi-task learning across completely different tasks , providing a powerful tool for many practical applications . Our work has interesting connections to meta-learning ( Nichol & Schulman , 2018 ; Finn et al. , 2017 ; Chen et al. , 2018 ) . For instance , when training data is not small , one can allow each task to carry a small model patch in the Reptile algorithm of Nichol & Schulman ( 2018 ) in order to increase expressivity at low cost .

Summary: This paper explored the means of tuning the neural network models using less parameters. The authors evaluated the case where only the batch normalisation related parameters are fine tuned, along with the last layer, would generate competitive classification results, while using very few parameters comparing with fine tuning the whole network model. However, several questions are raised concerning the experiment design and analysis: