Year: 2021

Abstract:
Knowledge transfer between heterogeneous source and target networks and tasks has received a lot of attention in recent times as large amounts of quality labelled data can be difficult to obtain in many applications. Existing approaches typically constrain the target deep neural network (DNN) feature representations to be close to the source DNNs feature representations, which can be limiting. We, in this paper, propose a novel adversarial multi-armed bandit approach which automatically learns to route source representations to appropriate target representations following which they are combined in meaningful ways to produce accurate target models. We see upwards of 5% accuracy improvements compared with the stateof-the-art knowledge transfer methods on four benchmark (target) image datasets CUB200, Stanford Dogs, MIT67 and Stanford40 where the source dataset is ImageNet. We qualitatively analyze the goodness of our transfer scheme by showing individual examples of the important features our target network focuses on in different layers compared with the (closest) competitors. We also observe that our improvement over other methods is higher for smaller target datasets making it an effective tool for small data applications that may benefit from transfer learning.

heading: 1 INTRODUCTION
text: Deep learning models have become increasingly good at learning from large amounts of labeled data. However, it is often difficult and expensive to collect sufficient a amount of labeled data for training a deep neural network (DNN). In such scenarios, transfer learning (Pan & Yang, 2009) has emerged as one of the promising learning paradigms that have demonstrated impressive gains in several domains such as vision, natural language, speech, etc., and tasks such as image classification (Sun et al., 2017; Mahajan et al., 2018), object detection (Girshick, 2015; Ren et al., 2015), segmentation (Long et al., 2015; He et al., 2017), question answering (Min et al., 2017; Chung et al., 2017), and machine translation (Zoph et al., 2016; Wang et al., 2018). Transfer learning utilizes the knowledge from information-rich source tasks to learn a specific (often information-poor) target task.
There are several ways to transfer knowledge from source task to target task (Pan & Yang, 2009), but the most widely used approach is fine-tuning (Sharif Razavian et al., 2014) where the target DNN being trained is initialized with the weights/representations of a source (often large) DNN (e.g. ResNet (He et al., 2016)) that has been pre-trained on a large dataset (e.g. ImageNet (Deng et al., 2009)). In spite of its popularity, fine-tuning may not be ideal when the source and target tasks/networks are heterogeneous i.e. differing feature spaces or distributions (Ryu et al., 2020; Tsai et al., 2020). Additionally, the pretrained source network can get overwritten/forgotten which prevents its usage for multiple target tasks simultaneously.
Among the myriad of other transfer techniques, the most popular approach involves matching the features of the output (or gradient of the output) of the target model to that of the source model (Jang et al., 2019; Li et al., 2018; Zagoruyko & Komodakis, 2016). In addition to the output features, a few methods attempt to match the features of intermediate states between the source and target models. Here, in this paper, we focus on the latter by guiding the target model with the intermediate source knowledge representations. Besides differing on what features to match, these methods also vary on how to match features, typically based on a regularization approach.
While common approaches allow knowledge transfer between heterogeneous tasks/networks, it is also important to recognize that constraining the target DNN representations to be close to certain
source DNN representations may be sub-optimal. For example, a source model, trained to classify cats vs dogs may be accessed at different levels to provide internal representations of tiger or wolf images to guide the target task in classifying tigers vs wolves. Since the source model is trained with a large number of parameters and labeled examples of cats and dogs, it will have learned several patterns that distinguish cat images from dog images. It is postulated that concepts or representations such as the shape of the tail, eyes, mouth, whiskers, fur, etc. are useful to differentiate them (Neyshabur et al., 2020), and it is further possible to reuse these learned patterns to generalize to new (related) tasks by accessing representations at the appropriate level. This example raises three important questions related to knowledge transfer between the source-target models: 1) What knowledge to transfer? 2) Where to transfer? 3) How to transfer the source knowledge?
While the what and where have been considered in prior literature (Jang et al., 2019), our work takes a novel and principled approach to the questions of what, where and how to transfer knowledge in the transfer learning paradigm. Specifically, and perhaps most importantly, we address the question of how to transfer knowledge, going beyond the standard matching techniques, and take the perspective that it might be best to let the target network decide what source knowledge is useful rather than overwriting one’s knowledge to match the source representations. Figure 1 illustrates our approach to knowledge transfer where the question of what and where is addressed by an adversarial multiarmed bandit (routing function) and the how is addressed by an aggregation operation detailed later. In building towards these goals, we make the following contributions:
• We propose a transfer learning method that takes a novel and principled approach to automatically decide which source layers (if any) to receive knowledge from. To achieve this, we propose an adversarial multi-armed bandit (AMAB) to learn the parameters of our routing function.
• We propose to meaningfully combine feature representations received from the source network with the target network-generated feature representations. Among various aggregation operations that are considered, AMAB also plays a role in selecting the best one. This is in contrast with existing methods that force the target representation to be similar to source representation.
• Benefits of the proposed method are demonstrated on multiple datasets. Significant improvements are observed over seven existing benchmark transfer learning methods, particularly when the target dataset is small. For example, in our experiment on ImageNet-based transfer learning on the target Stanford 40 Actions dataset, our auto-transfer learning method achieved more than 15% improvement in accuracy over the best competitor.

heading: 2 RELATED WORK
text: Transfer learning from a pretrained source model is a well-known approach to handle target tasks with a limited label setup. A key aspect of our work is that we seek to transfer knowledge between heterogeneous DNNs and tasks. Recent work focused on feature and network weight matching to address this problem where the target network is constrained to be near the source network weights and/or feature maps. Network matching based onL2−SP regularization penalizes the `2 distance of the pretrained source network weights and weights of the target networks to restrict the search space of the target model and thereby hinder the generalization (Xuhong et al., 2018). Recent work (Li et al., 2018) has shown that it is better to regularize feature maps of the outer layers than the network weights and reweighting the important feature via attention. Furthermore, attention-based feature distillation and selection (AFDS) matches the features of the output of the convolutional layers between the source-target models and prunes the unimportant features for computational efficiency. Similar matching can also be applied to match the Jacobians (change in output with respect to input rather than matching the output) between source and target networks (Srinivas & Fleuret, 2018). Previous works (Dhurandhar et al., 2018; 2020) also suggested that rather than matching the output of a complex model, it could also be used to weight training examples of a smaller model.
Learning without forgetting (LwF) (Li & Hoiem, 2017) leverages the concept of distillation (Hinton et al., 2015) and takes it further by introducing the concept of stacking additional layers to the source network, retraining the new layers on the target task, and thus adapting to different source and target tasks. SpotTune (Guo et al., 2019) introduced an adaptive fine-tuning mechanism, where a policy network decides which parts of a network to freeze vs fine-tune. FitNet (Romero et al., 2014) introduced an alternative to fine-tuning, where the internal feature representations of teacher networks were used as a guide to training the student network by using `2 matching loss between the two feature maps. Attention Transfer (AT) (Zagoruyko & Komodakis, 2016) used a similar approach to FitNet, except the matching loss was based on attention maps. The most relevant comparison to our work is that of Learning to Transfer (L2T-ww) (Jang et al., 2019), which matches source and target feature maps but uses a meta-learning based approach to learn weights for useful pairs of source-target layers for feature transfer. Unlike L2T-ww, our method uses a very different principled approach to combine the feature maps in a meaningful way (instead of feature matching) and let the target network decide what source knowledge is useful rather than overwriting one’s knowledge to match the source representations. Finally, Ji et al. (2021) uses knowledge distillation based approach to transfer knowledge between source and target networks.

heading: 3 AUTO-TRANSFER METHOD
text: In this section, we describe our main algorithm for Auto-Transfer learning and explain in detail the adversarial bandit approach that dynamically chooses the best way to combine source and target representations in an online manner when the training of the target proceeds.
What is the best way to train a target network such that it leverages pre-trained source representations speeding up training on the target task in terms of sample and time efficiency? We propose a routing framework to answer this: At every target layer, we propose to route one of the source representations from different layers and combine it with a trainable operation (e.g. a weighted addition) such that the composite function can be trained together. We propose to use a bandit algorithm to make the routing/combination choices in an online manner, i.e. which source layer’s representation to route to a given target layer and how to combine, while the training of the target network proceeds. The bandit algorithm intervenes once every epoch of training to make choices using rewards from evaluation of the combined network on a hold out set, while the latest choice made by the bandit is used by the training algorithm to update the target network parameters on the target task. We empirically show the benefit of this approach with other baselines on standard benchmarks. We now describe this framework of source-target representation transfer along with the online algorithm.

heading: 3.1 ROUTING REPRESENTATIONS
text: For a given image x, let {f1S(x), f2S(x), · · · , fNS (x)} and {f1T (x), f2T (x), · · · , fMT (x)} be the intermediate feature representations for image x from the source and the target networks, respectively. Let us assume the networks have trainable parameters WS ∈ Rds and WT ∈ Rdt where ds and
dt are the total number of trainable parameters of the networks. Clearly, the representations are a function of the trainable parameters of the respective networks. We assume that the source network is pre-trained. These representations could be the output of the convolutional or residual blocks of the source and target networks.
Our Key Technique: For the i-th target representation f iT , our proposed method a) maps i to one of the N intermediate source representations, f jS , or NULL (zero valued) representation; b) uses Tj , a trainable transformation of the representation f j S , to get f̃ j S , i.e. f̃ j S(x) = Tj(f j S(x)); and c) combines transformed source f̃ jS and the target representations f i T using another trainable operation⊕
chosen from a set of operationsM. LetW ⊕ i,j be the set of trainable parameters associated with the operator chosen. We describe the various possible operations below. The target network uses the combined representation in place of the original i-th target representation:
f̃ iT (x) = Tj(f j S(x)) ⊕ f iT (x) (1)
In the above equation, the trainable parameters of the operator depend on the i and j (that dependence is hidden for convenience in notation).
The set of choices are discrete, that is, P = {[N ]∪NULL}×M where [N ] denotes set ofN source representations. Each choice has a set of trainable parameters Tj ,W ⊕ i,j in addition to the trainable parametersWT of the target network.

heading: 3.2 LEARNING THE CHOICE THROUGH ADVERSARIAL BANDITS
text: To pick the source-target mapping and the operator choice, we propose an adversarial bandit-based online routing function (Auer et al., 2002) that picks one of the choices (with its own trainable parameters) containing information on what, where and how to transfer to the target representation i. Briefly, adversarial bandits choose actions at from a discrete choice of actions at time t, and the environment presents an adversarial reward rt(at) for that choice. The bandit algorithm minimizes the regret with respect to the best action a∗ in hindsight. In our non-stationary problem setting, the knowledge transfer from the source model changes the best action (and the reward function) at every round as the target network adapts to this additional knowledge. This is the key reason to use adversarial bandits for making choices as it is agnostic to an action dependent adversary.
Bandit Update: We provide our main update Algorithm 1 for a given target representation i from layer (`). At each round t, the update algorithm maintains a probability vector πt over a set of all possible actions from routing choice space P . The algorithm chooses a routing choice at = (jt → `, ⊕t
) randomly drawn according to the probability vector πt (in Line 7). Here jt is the selected source representation to be transfered to the target layer l and combined with target representation i using the operator ⊕t. Reward function: The reward rt for the selected routing choice is then computed by evaluating gain in the loss due to the chosen source-target combination as follows: the prediction gain is the difference between the target network’s losses on a hold out set Dv with and without the routing choice at i.e., L(fMT (x)) − L(f̃MT (x)) for a given image x from the hold out data. This is shown in the Algorithm 3 EVALUATE. The reward function is used in Lines 4 and 5 to update the probability vector πp,t almost identical to the update in the classical EXP3.P algorithm of (Auer et al., 2002). Note that if the current version of the trainable parameters is not available, then a random initialization is used. In our experiments, this reward value is mapped to the [−1, 1] range to feed as a reward to the bandit update algorithm.
Environment Update: Given the choice j → i and the operator ⊕
, the target network is trained for one epoch over all samples in the training data DT for the target task. Algorithm 2 TRAINTARGET updates the target network weightsWT and other trainable parameters (W ⊕ i,j , Tj) of the routing choice at for each epoch on the entire target training dataset. Our main goal is to train the best target network that can effectively combine the best source representation chosen. Here, L is the loss function which operates on the final representation layer of the target network. αt = 1/t and β is the exploration parameter. We set β = 0.4 and γ = 10−3.
Algorithm 1 AMAB - Update Algorithm for Target Layer ` 1: Inputs: Learning rate αt, Exploration parameter β, Number of Epochs E. Routing choice set P Initialize: w0,p, r̃0,p ← 0.
2: for t ∈ [1 : E] do 3: for p ∈ P do 4: wt,p ← log [ (1− αt) exp { wt−1,p + γr̃t−1,p } + αtK−1 ∑ j 6=p exp { wt−1,j + γr̃t−1,j }] 5:
πt,p ← (1− β) ewt,p∑K j=1 e wt,j + β K (2)
6: end for 7: Choose action at ∼ πt. Let at = (jt → `, ⊕t ).
8: Obtain current version of trainable parameters: ( WT , Tjt ,W ⊕t i,j ) . Use the standard random
initialization if not initialized. 9: rt,at ← EVALUATE(at, ( WT , Tjt ,W ⊕t i,j ) )
10: ( WT , Tjt ,W ⊕t i,j ) ← TRAIN-TARGET(at, ( WT , Tjt ,W ⊕t i,j ) )
11: r̃t,p ←
{ rt,p πt,p
if p = at, 0 otherwise
12: end for
Algorithm 2 TRAIN-TARGET - Train Target Network 1: Inputs: Target training dataset DT , Target loss L(·). Routing choice: (j → i, ⊕ ). Seed weight
parameters: WT [0], Tj [0],W ⊕ i,j [0].
2: Randomly shuffle DT . 3: for k ∈ [1 : |DT |] do 4: x← DT [k]. 5: ( WT [k], Tj [k],W ⊕ i,j [k] ) ← ( WT [k − 1], Tj [k − 1],W ⊕ i,j [k − 1]
) −ηk∇(WT ,Tj ,W⊕i,j)L(f̃MT (x))
6: end for 7: Output: Last iterate of ( WT , Tj ,W ⊕ i,j ) Algorithm 3 EVALUATE - Evaluate Target Network
1: Inputs: Routing Choice: (j → i, ⊕ ). Weight parameters: WT , Tj ,W ⊕ i,j . Target Loss L().
Target task hold out set Dv . 2: Output: 1|Dv| ∑ x∈Dv L(fMT (x))− L(f̃MT (x)).

heading: 3.3 ROUTING CHOICES
text: The routing choice (j → i, ⊕
i,j) can be seen as deciding where, what and how to transfer/combine the source representations with the target network.
Where to transfer? The routing function j → i decides which one of the N intermediate source features is useful for a given target feature f iT . In addition to these combinations, we allow the routing function to ignore the transfer using the NULL option. This allows the target network to potentially discard the source knowledge if it’s unrelated to the target task.
What to transfer? Once a pair of source-task (j → i) combination is selected, the routing function decides what relevant information from the source feature f jS should be transferred to the target
network using the transformation Tj . We use a Convolution-BatchNorm block to transfer useful features to the target network f̃ jS = BN(Conv(f j S)). Here, Tj = BN(Conv(·)). The convolution layer can select for relevant channels from the source representation and the batch normalization (Ioffe & Szegedy, 2015) addresses the covariant-shift between the source and the target representations, we believe that this combination is sufficient to ”match” the two representations. This step also ensures that the source feature has a similar shape to that of the target feature.
How to transfer (i.e. combine the representations)? Given a pair of source and target feature representations (j → i), the routing function chooses one of the following operations (i.e. ⊕ ) to combine them. We describe the class of operationsM, i.e. the various ways (1) is implemented.
1. Identity (Iden) operation allows the target network just to use the target representation f iT after looking at the processed source representation f̃ jS from the previous Conv-BN step. 2. Simple Addition (sAdd) adds the source and target features: f̃ iT = f̃ j S + f i T . 3. Weighted Addition (wAdd) modifies sAdd with weights for the source and target features. These weights constitute W ⊕ i,j . i.e. the trainable parameters of this operation choice: f̃ i T = wS,i,j ∗
f̃ jS + wT,i,j ∗ f iT . 4. Linear Combination (LinComb) uses the linear block (without bias term) along with the average
pooling to weight the features: f iT = LinS,i,j(f̃ j S) ∗ f̃ j S + LinT,i,j(f i T ) ∗ f iT where Lin·,i,j is a
linear transformation with its own trainable parameters. 5. Feature Matching (FM) follows the earlier work and forces the target feature to be similar to the
source feature. This operation adds a regularization term wi,j‖f̃ jS − f iT ‖ to the target objective L when we train. 6. Factorized Reduce (FactRed) use two convolution modules to reduce the number of channels c in the source and target features to c/2 and concat them together: f iT = concat(Conv
c/2 S,i,j(f̃ j S),Conv c/2 T,i,j(f i T )).
An action a from the search space is given by [(j → i), ⊕
i,j ]. The total number of choice combinations is O((N + 1)M). Typically N and M are very small numbers, for instance, when Resnet is used as a source and target networks, we have N = 4,M = 5. For large action search spaces, action pruning (Even-Dar et al., 2006) and greedy approaches (Bayati et al., 2020) can be used to efficiently learn the best combinations as demonstrated in our experiment section.

heading: 4 EXPERIMENTS
text: In this section, we present experimental results to validate our Auto-Transfer methods. We first show the improvements in model accuracy that can be achieved over various baselines on six different datasets and two network/task setups. We then demonstrate superiority in limited sample size and limited training time usecases. Finally, we use visual explanations to offer insight as to why performance is improved using our transfer method. Experimental results on a toy example can be found in the supplement.

heading: 4.1 EXPERIMENTAL SETUP
text: Our transfer learning method is compared against existing baselines on two network/task setups. In the first setup, we transfer between similar architectures of different complexities; we use a 34-layer ResNet (He et al., 2016) as the source network pre-trained on ImageNet and an 18-layer ResNet as the target network. In the second setup, we transfer between two very different architectures; we use an 32-layer ResNet as the source network pretrained on TinyImageNet and a 9-layer VGG (Simonyan & Zisserman, 2014) as the target network. For ImageNet based transfer, we apply our method to four target tasks: Caltech-UCSD Bird 200 (Wah et al., 2011), MIT Indoor Scene Recognition (Quattoni & Torralba, 2009), Stanford 40 Actions (Yao et al., 2011) and Stanford Dogs (Khosla et al., 2011). For TinyImageNet based transfer, we apply our method on two target tasks: CIFAR100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011)
We investigate different configurations of transfer between source and target networks. In the full configuration, an adverserial multi-armed bandit (AMAB) based on Exponential-weight algorithm
for Exploration and Exploitation (EXP3) selects (source, target) layer pairs as well as one of one of five aggregation operations to apply to each pair (operations are independently selected for each pair). In the route configuration, the AMAB selects (source, target) layer pairs but the aggregation operation is fixed to be weighted addition. In the fixed configuration, transfer is done between manually selected pairs of source and target layers with matching spatial size of intermediate representations. Transfer can go between any layers, but the key is that the pairs are manually selected. In each case, during training, the source network is passive and only shares the intermediate feature representation of input images hooked after each residual block. After pairs are decided, the target network does aggregation of each pair of source-target representation in feed-forward fashion. The weight parameters of aggregation are trained to act as a proxy to how much source representation is useful for the target network/task. For aggregating features of different spatial sizes, we simply use a bilinear interpolation.

heading: 4.2 EXPERIMENTS ON TRANSFER BETWEEN SIMILAR AND DIFFERENT ARCHITECTURES
text: In the first setup, we evaluate all three Auto-Transfer configurations, full, fixed, and route, on various visual classification tasks, where transfer is from a Resenet-34 model to a Resnet-18 model. Our findings are compared with an independently trained Resnet-18 model (Scratch), another Resnet-18 model tuned for ImageNet and finetuned to respective tasks (Finetune), and the following existing baselines: Learning without forgetting (LwF) (Li & Hoiem, 2017), Attention Transfer (AT) (Zagoruyko & Komodakis, 2016), Feature Matching (FM) (Romero et al., 2014), Learning What and Where to Transfer (L2T-ww) (Jang et al., 2019) and Show, Attend and Distill (SAaD) (Ji et al., 2021). Results are shown in Table 6. Each experiment is repeated 3 times.
First, note that the Auto-Transfer Fixed configuration already improves performance on (almost) all tasks as compared to existing benchmarks. The fixed approach lets the target model decide how much source information is relevant when aggregating the representations. This result supports our approach to feature combination and demonstrates that it is more effective than feature matching. This even applies to the benchmark methods that go beyond and learn where to transfer to. Next, note that the Auto-Transfer Route configuration further improves the performance over the one-toone configuration across all tasks. For example, on the Stanford40 dataset, Auto-Transfer Route improves accuracy over the second best baseline by more than 15%. Instead of manually choosing source and target layer pairs, we automatically learn the best pairs through our AMAB setup. This result suggests that learning the best pairs through our AMAB setup to pick source-target pairs is a useful strategy over manual selection as done in the one-to-one configuration.
Next, note that Auto-Transfer Full, which allows all aggregation operations, does well but does not outperform Auto-Transfer Route. Indeed, the Auto-Transfer Full results showed that selected operations were all leaning to weighted addition, but other operations were still used as well. We conjecture that weighted addition is best for aggregation, but the additional operations allowed in Auto-Transfer Full introduce noise and make it harder to learn the best transfer procedure. Additionally, we conducted experiments by fixing aggregation to each of 5 operations and running Auto-Transfer Route and found that weighted addition gave best performance Table 8.
In order to demonstrate that our transfer method does not rely on the source and target networks being similar architectures, we proceed to transfer knowledge from a Resnet-32 model to a VGG-9 model. Indeed, Table 6 in the appendix demonstrates that Auto-Transfer significantly improves over other baselines for CIFAR100 and STL-10 datasets

heading: 4.3 EXPERIMENTS ON LIMITED AMOUNTS OF TRAINING SAMPLES
text: Transfer learning emerged as an effective method due to performance improvements on tasks with limited labelled training data. To evaluate our Auto-Transfer method in such data constrained scenario, we train our Auto-Transfer Route method on all datasets by limiting the number of training samples. We vary the samples per class from 10% to 100% at 10% intervals. At 100%, Stanford40 has∼100 images per class. We compare the performance of our model against Scratch and L2T-ww for Stanford40 and report results in Figure 2 (top). Auto-Transfer Route significantly improves the performance over existing baselines. For example, at 60% training set (∼60 images per class), our method achieves 77.90% whereas Scratch and L2T-ww achieve 29% and 46%, respectively. To put this in perspective, Auto-Transfer Route requires only 10% images per class to achieve better accu-
racy than achieved by L2T-ww with 100% of the images. We see similar performance with other three datasets: CUB200, MIT67, Stanford Dogs (Figure 9).

heading: 4.4 IMPROVEMENTS IN TRAINING AND INFERENCE TIMES
text: In order to assess training metrics and stability of learning, we visualize the test accuracy over training steps in Figure 2 (bottom) for the Stanford40 dataset. The results show that our method learns significantly quicker relative to the second closest baseline. For example, at epoch 25, our method achieves 74.55% accuracy whereas L2T-ww and Scratch achieve 25.55% and 21.69%, respectively. In terms of training time, Auto-Transfer Route took ∼300 minutes to train 200 epochs on the Stanford40 dataset, whereas L2T-ww and Scratch models took 610 and 170 minutes, respectively. Taken together, our method significantly improves performance over the second baseline with less than half the runtime. We report additional experiments with training curves plotted against training time in appendix (Figure 7) and inference times plotted against test accuracy (Figure 8).

heading: 4.5 VISUAL EXPLANATIONS
text: In order to qualitatively analyze what bandit AutoTransfer Route is learning, Grad-CAM (Selvaraju et al., 2017) based visual explanations are presented in Figure 3 (additional explanations are in Figures 11, 12, and 13 in the appendix). Grad-CAM highlights pixels that played an important role in correctly labelling the input image. For each target task, we present a (random) example image that is correctly labelled by bandit Auto-Transfer but incorrectly classified by Scratch, along with layer-wise Grad-CAM images that illustrate what each layer of the target model focuses on. For each image, we report the
incorrect label, correct label and class probability for correct (pc) and incorrect (pi) labels.
Overall, we observe that our method pays attention to relevant visual features in making correct decisions. For example, in the first image from MIT67 dataset, the Scratch model incorrectly labelled it as a gameroom while the correct class is bedroom (pi = 0.67, pc = 0.007). The Grad-CAM explanations show that layers 1-3 of the Scratch model pay attention to the green floor which is atypical to a bedroom and common in gamerooms (e.g. pool tables are typically green). The last layer focuses on the surface below the window that looks like a monitor/tv that is typically found in gamerooms. On the other hand, our model correctly identifies the class as bedroom (pc = 0.57) by paying attention to the bed and surrounding area at each layer.
To visualize an example from a harder task, consider the indigo bunting image from the CUBS dataset. The Scratch model classifies the image as a bluejay (pi = 0.85, pc = 0.09), but our model correctly predicts it as a bunting (pc = 0.99). Indigo buntings and blue jays are strikingly similar, but blue jays have white faces and buntings have blue faces. We clearly see this attribute picked up by the bandit Auto-Transfer model in layers 2 and 3. We hypothesize that the source model, trained on millions of images, provides useful fine-grained information useful for classifying similar classes.

heading: 5 CONCLUSION
text: In this paper, we have put forth a novel perspective where we leverage and adapt an adversarial multi-armed bandit approach to transfer knowledge across heterogeneous tasks and architectures. Rather than constraining target representations to be close to the source, we dynamically route source representations to appropriate target representations also combining them in novel and meaningful ways. Our best combination strategy of weighted addition leads to significant improvement over state-of-the-art approaches on four benchmark datasets. We also observe that we produce accurate target models faster in terms of (training) sample size and number of epochs. Further visualization based qualitative analysis reveals that our method produces robust target models that focus on salient features of the input more so than its competitors, justifying our superior performance.

heading: ETHICS STATEMENT
text: We believe that one major ethical implication of our approach, which also applies to other transfer learning approaches, is the bias introduced by the source model to the target task when deployed. We currently depend on the pretrained source models with having done ethical implications and having circumscribed their limitations and risks to society. Another concern associated with the proposed approach is the choices made by the adversarial bandits based on the reward payoff which could be manipulated for intentional advantages. One possibility to mitigate such concerns is to analyze the behavior of the target model on individual inputs. To this end, we have visualized (in the appendix) regions in the input that the target model focuses on for both correct and incorrect predictions made by our model in comparison with the best performing baseline to facilitate such inquiries.

heading: REPRODUCIBILITY DISCUSSION
text: All experimental details (including the resources used for training and evaluating the models) are provided in Section 4 and the appendix A.3. All datasets are public. Code is provided in the supplementary material with instructions to run the models.

heading: A APPENDIX

heading: A.1 TOY EXAMPLE
text: In this section, we simulate our experiment on a toy example. We compare our Auto-Transfer with the other baselines: L2T-ww and Scratch. In this simulation, we consider Auto-Transfer with a fixed (one-to-one) setup for simplicity in our experiment analysis.
We consider predicting a sine wave function (y = sin(x)) as our source task and a sinc function (y = sin(x)x ) as our target task. Clearly, the features from the pretrained source model will help the target task in predicting the sinc function. Both the input data point x and the output value y are one-dimensional vectors (din = dout = 1). We use a shallow linear network consists of 4 linear blocks: f1 = Lin(din,h1)(x), f2 = Lin(h1,h2)(f1), f3 = Lin(h1,h2)(f2), out = Lin(h3,dout)(f3) for a datapoint x. For source network, we set the hidden size to 64 (i.e., h1 = h2 = h3 = 64) and 16 for the target network. We sampled 30, 000 data points to generate training set (x,y) and 10, 000 test-set data points for the source network and (i.e., x is sampled from a Gaussian distribution and y = sin(x)). Similarly, we generated 1000 training examples and 800 test set examples for the target network. Both the source and the target networks are trained for E = 50 epochs.
Figure 4 (left) shows the source model prediction for the test data. Given the shallow linear network with 64 hidden dimensions and 30, 000 training example, the source model perfectly predicts the sin(x) function. Figure 4 (right) shows the predictions from the scratch target model, source model, L2T-ww and Auto-Transfer for the target test data. We report the Auto-Transfer with fixed choice of [(0,0),(1,1),(2,2), wtAdd] for this experiment. We can see that the Auto-Transfer accurately predicts the target task even when there is a limited amount of labeled examples. Our results show the test set loss for the target data is relatively less compared to the other baselines (0.0030 MSE loss for Auto-Transfer vs 0.0033 and 0.125 MSE loss for the scratch and L2T-ww).

heading: A.2 REAL DATASETS
text: We evaluate the performance of Auto-Transfer on six benchmarks with different tasks: Stanford Actions 40 dataset for action recognition, CUBS Birds 200 dataset for object recognition, Stanford Dogs 120 for fine-grained object recognition, MIT Indoors 67 for scene classification, CIFAR 100 and STL-10 image recognition datasets.
Stanford Actions 40. Stanford Actions 40 dataset contains images of humans performing 40 actions. There are about 180-300 images per class. We do not use bounding box and other annotation information for training. There are a total of 9,532 images, making it the smallest dataset in our benchmark experiments.
Caltech-UCSD Birds-200-2011. CUB-200-2011 is a bird classification datset with 200 bird species. Each species is associated with a wikipedia article and organized by scientific classifi-
cation. Each image is annotated with bounding box, part location, and attribute labels. We use only classification labels during training. There are a total of 11,788 images.
Stanford Dogs 120. The Stanford Dogs dataset contains images of 120 breeds of dogs from around the world. There are exactly 100 examples per category in the training set. It is used for the task of fine-grained image categorization. We do not use the bounding box annotations. There are a total of 20,580 images.
MIT Indoors 67. MIT Indoors 67 is a scene classification task containing 67 indoor scene categories, each of which consists of at most 80 images for training and 20 for testing. Indoor scene recognition is challenging because spatial properties, background information and object characters are expected to be extracted. There are 15,620 images in total.
CIFAR 100. CIFAR 100 is a image recognition task containing 100 different classes with 600 images in each class. There are 500 training images and 100 testing images per class. It is a subset of tiny images datastet.
STL 10. STL 10 is a image recognition task containing 10 classes. It is inspired by the CIFAR-10 dataset but with some modifications. In particular, each class has fewer labeled training examples than in CIFAR-10.

heading: A.3 EXPERIMENT DETAILS
text: For our experimental analysis in the main paper, we set the number of epochs for training to E = 200. The learning rate for SGD is set to 0.1 with momentum 0.9 and weight decay 0.001. The learning rate for the ADAM is set to 0.001 with and weight decay of 0.001. We use Cosine Annealing learning rate scheduler for both optimizers. The batch size for training is set to 64. Our target networks were randomly initialized before training.
The target models were trained in parallel on two machines with the specifications shown in Table 2.

heading: A.4 TRAINING AND TESTING PERFORMANCE

heading: A.5 ADDITIONAL EXPERIMENTS ON LIMITED AMOUNTS OF DATA
text: To evaluate our Auto-Transfer method in data constrained scenario further, we train our AutoTransfer (route) method on the CUB200, Stanford Dogs and MIT67 datasets by limiting the number of training samples (Figure 9). We vary the samples per class from 10% to 100% at 10% intervals.

heading: A.6 ABLATION STUDIES

heading: TRAINING THE NETWORK USING BANDIT SELECTED PAIRS
text: To evaluate the importance of training the target network with adversarial multi-armed bandit, we retrained our target network with a fixed source-layer configuration selected at 200th epoch of previous best bandit based experiments. For Eg. in our best bandit based experiment for CUB200, the source,target pairs were {(2,1), (3,2), (2,3), (2,4)}. As seen in Table 7, we find that this experiment decreased performance in comparison to bandit based one in all target tasks. This confirms the need for bandit based decision maker, that learns combination weights and pairs over training steps.

heading: TRAINING THE NETWORK USING DIFFERENT AGGREGATION OPERATORS
text: To evaluate how different aggregation operators influence Auto-Transfer, we train Auto-Transfer Route by fixing aggregation to 5 different operations. Identity (iden), Simple Addition (sAdd), Weighted Addition (wtAdd), Linear Combination (LinComb) and Factored Reduction (FactRed). Results for Stanford40 dataset is found in Table 8. We find that weighted addition performs the best.
A.7 VISUALIZING INTERMEDIATE REPRESENTATIONS

heading: A.8 ADDITIONAL EXPLANATIONS USING GRAD-CAM
text: We here offer more examples of visual explanations of what is being transferred using Auto-Transfer Route. The first example in Figure 11 is an image of cooking from the Stanford40 dataset. The Scratch model incorrectly classifies the image as cutting (pi = 0.88, pc = 0.01) by paying attention to only the cooking surface that looks like a table and person sitting down (typical for someone cutting vegetables). On the other hand, our model correctly labels the image (pc = 0.99) by paying attention to the wok and cooking utensils such as water pot, etc. We hypothesize that this surrounding information is provided by the source model which is useful in making the correct decision.
The second example in Figure 11 is from the Stanford Dogs dataset (Figure 11). The scratch model fails to pay attention to relevant class information (dog) and labels a chihuahua as german sheperd (pi = 0.23, pc = 0.0002) by focusing on the flower, while our method picks the correct label (pc = 0.99). Bandid Auto-Transfer gets knowledge about the flower early on and then disregards this knowledge before attending to relevant class information. Further examples of visual explanations comparing to L2T-ww (Figure 12) and counter-examples where our method identifies the wrong label (Figure 13) follow below. For these counter-examples we find that the task is typically hard. For eg. playing violin vs playing guitar. And, the class probability of incorrect label is closer to that of correct label, suggesting that our method was not confident in predicting wrong class.

Review summaries:
This work addresses the challenge of automatic knowledge transfer between different networks. The authors propose to use an adversarial  multi-armed bandit to decide where, what and how to combine outputs from different layers of the two networks, improving on the recent line of work that achieves transfer by enforcing closeness between the representations obtained by the two networks.  For each layer, the proposed method introduces some additional parameters in the form of intermediate representations and parametrized ways to combine such intermediate representations with the source network's output.  The multi-armed bandit is trained to choose which intermediate representation and which aggregating function to use. The reward is determined based on a hold out set. The authors compare the proposed approach with some previous work, showing improvements, and perform a qualitative analysis of the results.

The paper proposes an algorithm to transfer a "source" pre-trained deep model into a new one for a given target task. The idea is to wire in a certain way the intermediate representations of the source model into those of the "target" one. The algorithm considers a (small) discrete space of potential such ways to connect both models (location, source processing, and type of merging), and applies a bandit algorithm to sequentially spend training budget under one of the configurations. Experiments are provided that show big gains with respect to standard approaches (like finetuning the source model).

To transfer knowledge between heterogeneous source and target networks and tasks, this paper proposes a novel adversarial multi-armed bandit approach (AMAB) which automatically learns to route source representations to appropriate target representations. It combines feature representations received from the source network with the target network-generated feature representations via various aggregation operations. The work is interesting and makes sense to some extent.