Title: LEARNING TO TRANSFER LEARN

Abstract: We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset. L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses. The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set. We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets. In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin.

1. INTRODUCTION:

Deep neural networks excel at understanding images (He et al., 2016; Simonyan and Zisserman, 2015; Zagoruyko and Komodakis, 2016) , text (Conneau et al., 2017; Devlin et al., 2018; Lai et al., 2015) and audio (van den Oord et al., 2016; Amodei et al., 2016; Chiu et al., 2018) . The performance of deep neural networks improves significantly with more training data (Hestness et al., 2017) . As the applications of deep neural networks diversify and span use cases with small training datasets, conventional training approaches are often insufficient to yield high performance. It becomes highly beneficial to utilize extra source datasets and "transfer" the relevant information to the target training dataset. Transfer learning, commonly in the form of obtaining a pre-trained model on a large-scale source dataset and then further training it on the target dataset (known as fine-tuning), has become the standard recipe for most real-world artificial intelligence applications. Compared to training from random initialization, fine-tuning yields considerable performance improvements and convergence speedup, as demonstrated for object recognition (Razavian et al., 2014) , semantic segmentation (Long et al., 2015) , language understanding (Devlin et al., 2018) , speech synthesis (Arik et al., 2018) , audio-visual recognition (Moon et al., 2014) and language translation (Zoph et al., 2016) .

Towards the motivation of pushing the performance of transfer learning, recent studies (Ngiam et al., 2018; Mahajan et al., 2018; Lee et al., 2019) have explored the direction of matching the source and target dataset distributions. Even simple methods to encourage domain similarity, such as prior class distribution matching in Domain Adaptive Transfer Learning (DATL) (Ngiam et al., 2018) , are shown to be effective -indeed, in some cases, the distribution similarity is shown to be more important than the scale of the source dataset. In this paper, our goal is to push this direction further by introducing a novel RL-based meta-learning framework. Our framework, learning to transfer learn (L2TL), adaptively infers the beneficial source samples directly from the performance on the target task. There are numerous cases that source samples could have features that are implicitly relevant to the target samples and would benefit the learning process, but they may belong to different classes. For example, consider the classification problem for bird images. The source dataset may not contain bird images, but may have airplane images with similar visual patterns that would aid the training of the bird classifier. L2TL framework is designed to automatically handle such cases with its policy learning, and can push the performance further in ways that manual source dataset selection or fixed domain similarity methods may not be able to. We demonstrate state-of-the-art transfer learning results given fixed models in wide range of scenarios:

• Source and target datasets from similar domains: We consistently outperform the fine-tuning baseline with a 0.6%-1.3% relative accuracy gain on five fine-grained datasets, and consistently outperform DATL with a 0.3%-1.5% relative accuracy gain.

• Low-shot target dataset regime: We consistently outperform the fine-tuning baseline on five fine-grained datasets, up to 6.5% relative accuracy gain for five samples per class. • Source and target datasets from dissimilar domains: We outperform the fine-tuning baseline, up to 1.7% relative accuracy gain on a texture dataset and 0.7 relative AUC gain on Chest X-Ray dataset.

In addition, L2TL yields ranking of the source data samples according to their contributions to the target task, that open horizons for new forms of interpretable insights for model developers.

2. RELATED WORK:

Adaptive transfer learning: There is a long history of transfer learning for neural networks, particularly in the form of fine-tuning (Yosinski et al., 2014) (Girshick et al., 2014) . Various directions were recently considered to improve standard fine-tuning. One direction is carefully choosing which portion of the network to adapt while optimizing the information extraction from the source dataset.

In (Guo et al., 2019) , a policy network is used to make routing decisions on whether to pass the input through the fine-tuned or the pre-trained layers. In (Li et al., 2018) , a regularization scheme is proposed to promote the similarity of the fine-tuned model with the pre-trained model as a favorable inductive bias. Another direction is carefully choosing which input samples are relevant to the target task, as in our paper. (Ge and Yu, 2017) uses filter bank responses to select nearest neighbor source samples and demonstrates improved performance. In (Cui et al., 2018) , domain similarity between source and target datasets is quantified using Earth Mover's Distance (EMD). Transfer learning is demonstrated to benefit from pre-training on a source domain that is similar in EMD. With a simple greedy subset creation selection criteria, promising results are shown for improving the target test set performance. Domain adaptive transfer learning (DATL) (Ngiam et al., 2018) employs probabilistic shaping, where the value is proportional to the ratios of estimated label prior probabilities. L2TL does not use an indirect similarity metric like proximity of filter bank responses, EMD or prior class probabilities. Instead, it aims to assign weights to optimize the target set metric directly.

Reweighing training examples: Reweighing of constituent training terms has been considered for various performance goals. (Ren et al., 2019) applies gradient descent-based meta-learning with the goal of providing noise robustness and class balance in learning. Focal loss (Lin et al., 2017) is another soft weighting scheme that emphasizes harder examples. In (Jiang et al., 2018) , a studentteacher training framework is utilized such that the teacher model provides a curriculum via a sample weighting scheme for the student model to focus on samples whose labels are likely to be correct. (Ghorbani and Zou, 2019) studies the value of examples by estimating the data Shapley value, and it shows that removing examples with low values would not harm performance. Reweighting of examples is also used in self-paced learning (Kumar et al., 2010) where the weights are optimized to learn easier examples first. Different from these, our paper focuses on optimizing the weights for each class, with the purpose of improving the transfer learning performance.

Meta-learning: Meta-learning broadly refers to learning to learn frameworks (Schmidhuber et al., 1997) whose goal is to improve the adaptation to a new task with the information extracted from other tasks. Meta learners are typically based on inspiration from known learning algorithms like gradient descent (Finn et al., 2017) or derived from black box neural networks (Santoro et al., 2016) .

In few-shot learning, the use of validation loss as a meta-objective has been explored (Ravi and Larochelle, 2017) . However, for optimization problems with non-differentiable objectives like neural architecture search, RL-based meta-learning is shown to be a promising approach (Zoph and Le, 2017; Pham et al., 2018) . RL-based optimization has successfully been applied in many search spaces, e.g. learning a data augmentation policy (Cubuk et al., 2018) . The specific form of meta-learning application in L2TL is novel -it employs guidance on the source dataset information extraction with the reward from the target validation dataset performance. Different from few-shot learning, we consider a more common real-world scenario where an easily accessible source dataset is integrated.

3. LEARNING FROM SOURCE AND TARGET DATASETS:

We consider a training objective function L(Ω, ζ S , ζ T , λ, α s , α t ) 1 that is jointly optimized for a source dataset D S and a target dataset D T in the general form: 

EQUATION

(•; Ω, ζ S ) = h S (•; ζ S ) • g(•; Ω) and f T (•; Ω, ζ T ) = h T (•; ζ T ) • g(•;

Ω), g can be a high capacity function with large number of trainable parameters that can be represented with a deep neural network, and h T and h S are low capacity functions with small number of parameters that can be represented with very shallow neural networks. 4 The learning goal of Eq. 1 is to generalize to unseen samples from a held-out target validation dataset, and maximize a target performance metric R on it: x ,y ∼D T R(f T (x ; Ω, ζT ), y )). R may or may not be differentiable with respect to x and y such as the top-1 accuracy or area under the curve (AUC) for classification. Ω, ζT are the pre-trained weights optimized in Eq. 1. Without transfer learning, i.e., when only the target dataset is considered, α s [i] = 0 and α t [i] = 1 for all i. In fine-tuning, the optimization is first considered for the source dataset for N S steps with uniform weighing of the samples λ(x, y) = 1, and then for the target dataset using the pre-trained weights Ω, ζT , i.e.:

EQUATION

Next, we describe our method towards optimal learning from source and target datasets. We propose learning to transfer learn (L2TL) framework (shown in Fig. 1 ) to learn the weight assignment adaptively, rather than using a fixed weight assignment function λ(x, y; Φ) to measure the relatedness between the source domain and the target domain. Learning of the adaptive weights in L2TL is guided by the performance metric R on a held-out target validation dataset. Thus, beyond targeting general relatedness, the framework directly targets relatedness for the specific goal of improvement in target evaluation performance.

4. LEARNING TO TRANSFER LEARN:

H N 6 P g R f / M 3 / Z g = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x o W L C v Y B T S i T 6 W 0 7 d D I J M x O h h P 6 G G x e K u P V n 3 P k 3 T t o s t P X A w O G c e 7 l n T p g I r o 3 r f j u l t f W N z a 3 y d m V n d 2 / / o H p 4 1 N Z x q h i 2 W C x i 1 Q 2 p R s E l t g w 3 A r u J Q h q F A j v h 5 D b 3 O 0 + o N I / l o 5 k m G E R 0 J P m Q M 2 q s 5 P s R N W N G R X Y / I / 1 q z a 2 7 c 5 B V 4 h W k B g W a / e q X P 4 h Z G q E 0 T F C t e 5 6 b m C C j y n A m c F b x U 4 0 J Z R M 6 w p 6 l k k a o g 2 y e e U b O r D I g w 1 j Z J w 2 Z q 7 8 3 M h p p P Y 1 C O 5 l n 1 M t e L v 7 n 9 V I z v A 4 y L p P U o G S L Q 8 N U E B O T v A A y 4 A q Z E V N L K F P c Z i V s T B V l x t Z U s S V 4 y 1 9 e J e 2 L u u f W v Y f L W u O m q K M M J 3 A K 5 + D B F T T g D p r Q A g Y J P M M r v D m p 8 + K 8 O x + L 0 Z J T 7 B z D H z i f P 9 r 3 k Y w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e p g f B I Y 9 B z P b q b H N 6 P g R f / M 3 / Z g = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x o W L C v Y B T S i T 6 W 0 7 d D I J M x O h h P 6 G G x e K u P V n 3 P k 3 T t o s t P X A w O G c e 7 l n T p g I r o 3 r f j u l t f W N z a 3 y d m V n d 2 / / o H p 4 1 N Z x q h i 2 W C x i 1 Q 2 p R s E l t g w 3 A r u J Q h q F A j v h 5 D b 3 O 0 + o N I / l o 5 k m G E R 0 J P m Q M 2 q s 5 P s R N W N G R X Y / I / 1 q z a 2 7 c 5 B V 4 h W k B g W a / e q X P 4 h Z G q E 0 T F C t e 5 6 b m C C j y n A m c F b x U 4 0 J Z R M 6 w p 6 l k k a o g 2 y e e U b O r D I g w 1 j Z J w 2 Z q 7 8 3 M h p p P Y 1 C O 5 l n 1 M t e L v 7 n 9 V I z v A 4 y L p P U o G S L Q 8 N U E B O T v A A y 4 A q Z E V N L K F P c Z i V s T B V l x t Z U s S V 4 y 1 9 e J e 2 L u u f W v Y f L W u O m q K M M J 3 A K 5 + D B F T T g D p r Q A g Y J P M M r v D m p 8 + K 8 O x + L 0 Z J T 7 B z D H z i f P 9 r 3 k Y w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e p g f B I Y 9 B z P b q b H N 6 P g R f / M 3 / Z g = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x o W L C v Y B T S i T 6 W 0 7 d D I J M x O h h P 6 G G x e K u P V n 3 P k 3 T t o s t P X A w O G c e 7 l n T p g I r o 3 r f j u l t f W N z a 3 y d m V n d 2 / / o H p 4 1 N Z x q h i 2 W C x i 1 Q 2 p R s E l t g w 3 A r u J Q h q F A j v h 5 D b 3 O 0 + o N I / l o 5 k m G E R 0 J P m Q M 2 q s 5 P s R N W N G R X Y / I / 1 q z a 2 7 c 5 B V 4 h W k B g W a / e q X P 4 h Z G q E 0 T F C t e 5 6 b m C C j y n A m c F b x U 4 0 J Z R M 6 w p 6 l k k a o g 2 y e e U b O r D I g w 1 j Z J w 2 Z q 7 8 3 M h p p P Y 1 C O 5 l n 1 M t e L v 7 n 9 V I z v A 4 y L p P U o G S L Q 8 N U E B O T v A A y 4 A q Z E V N L K F P c Z i V s T B V l x t Z U s S V 4 y 1 9 e J e 2 L u u f W v Y f L W u O m q K M M J 3 A K 5 + D B F T T g D p r Q A g Y J P M M r v D m p 8 + K 8 O x + L 0 Z J T 7 B z D H z i f P 9 r 3 k Y w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e p g f B I Y 9 B z P b q b H N 6 P g R f / M 3 / Z g = " > A A A B 8 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i Q i 6 L L o x o W L C v Y B T S i T 6 W 0 7 d D I J M x O h h P 6 G G x e K u P V n 3 P k 3 T t o s t P X A w O G c e 7 l n T p g I r o 3 r f j u l t f W N z a 3 y d m V n d 2 / / o H p 4 1 N Z x q h i 2 W C x i 1 Q 2 p R s E l t g w 3 A r u J Q h q F A j v h 5 D b 3 O 0 + o N I / l o 5 k m G E R 0 J P m Q M 2 q s 5 P s R N W N G R X Y / I / 1 q z a 2 7 c 5 B V 4 h W k B g W a / e q X P 4 h Z G q E 0 T F C t e 5 6 b m C C j y n A m c F b x U 4 0 J Z R M 6 w p 6 l k k a o g 2 y e e U b O r D I g w 1 j Z J w 2 Z q 7 8 3 M h p p P Y 1 C O 5 l n 1 M t e L v 7 n 9 V I z v A 4 y L p P U o G S L Q 8 N U E B O T v A A y 4 A q Z E V N L K F P c Z i V s T B V l x t Z U s S V 4 y 1 9 e J e 2 L u u f W v Y f L W u O m q K M M J 3 A K 5 + D B F T T g D p r Q A g Y J P M M r v D m p 8 + K 8 O x + L 0 Z J T 7 B z D H z i f P 9 r 3 k Y w = < / l a t e x i t >

Batch approximations may be optimal for different batch sizes for source and target dataset and thus may employ different batch normalization parametrization.

In f (•; W) representation, W denote the trainable parameters. Source datasets are typically much larger and contain more classes, hence hS may have higher number of parameters than hT .

Algorithm 1: L2TL -Learning to Transfer Learn N ← number of training iterations for i ← 1 to N do l s ← 0, l t ← 0 for j ← 1 to B S do Sample x j , y j from D S Calculate classification loss L S (x j , y j ; Ω, ζ S ) Calculate example weight λ(x j , y j ; Φ)

l s = l s + λ • L S l s = α i • l s for k ← 1 to B T do Sample x k , y k from D T Calculate classification loss L T (x k , y k ; Ω, ζ T ) l t = l t + L T Update Ω, ζ S , ζ T using stochastic gradient descent with loss l s + l t r ← 0 for k ← 1 to B P do Sample x k , y k from D T Calculate reward R(f T (x k ), y k ) r = r + R Update Φ

with reward r using policy gradient While optimizing for λ(x, y; Φ), one straightforward option for scaling coefficients would be alternating them between (1, 0) and (0, 1) -i.e. training the source dataset until convergence with optimized Φ and then training the target dataset until convergence with the pre-trained weights from the source dataset. Yet, the approach may potentially require many alternating update steps and the computational cost may become prohibitively high. Instead, we design the policy model in L2TL to output (α s [i], α t [i]) along with λ. 5 The policy optimization step is decoupled from the gradient-descent based optimization for Ω, ζ S and ζ T . Updates are reflected to the policy model via the information embodied in Ω and ζ T . Algorithm. 1 overviews the training updates steps.

In the first phase of a learning iteration, we apply gradient decent-based optimization to learn the encoder weights Ω, and the classifier layer weights ζ S , ζ T to minimize the loss function L:

EQUATION

In this learning phase, the policy model is fixed, and its actions are sampled to determine weights. The loss might be skewed when most of source dataset samples in a batch are unrelated, while some batches contain more related examples. To ease this problem, we sample a larger batch and dynamically select more related examples. At each iteration, we sample a training batch of size M S • B S , and use the top B S of them with the highest weights for training updates. This approach also yields computational benefits as the gradients would not be computed for most source dataset samples until convergence.

In the second phase of a learning iteration, the goal is to optimize policy weight Φ that maximizes the evaluation metric R D T on the target validation set with given encoder weights from the first phase:

EQUATION

)

We treat this step as an RL problem, such that the policy model outputs the actions for λ(x, y; Φ) and α towards optimization of a reward. In its general form λ(x, y; Φ) may yield a very high dimensionality for optimization of Φ. For simplicity, we consider sample-independent modeling of λ(x, y; Φ), similar to (Ngiam et al., 2018) , i.e., λ(x, y; Φ) = λ(y; Φ). For more efficient optimization, we discretize the possible values of λ(y; Φ) into pre-defined number of actions, in the range λ(y) ∈ [0, 1]. We define n actions, such that each action k ∈ [0, n -1] corresponds to a weight value k/(n -1).

For example, when n = 11, the weight values are [0, 0.1, 0.2, . . . , 1.0]. We also discretize the possible values for α, using n actions. Each action k corresponds to βk /(n -1), where β is a hyperparameter to constrain the value range of α. The search space has n × (c S ) n possibilities, where c S is the number of classes in the source dataset. When training the policy model, we use policy gradient to maximize the reward on the target dataset D T , using a batch size of B P . We use the evaluation mode for the convolutional encoder. At iteration t, we denote the advantage A t = R tb t , where b t is the baseline. Following (Pham et al., 2018) , we use the moving average baseline to reduce variance, i.e., b t = (1γ)b t + γR t , where γ is the decay rate. The policy gradient is computed using REINFORCE (Williams, 1992) and optimized using Adam (Kingma and Ba, 2014) .

5. EXPERIMENTS:

We demonstrate the performance of L2TL in various scenarios. As the source dataset, we use the ImageNet dataset (Russakovsky et al., 2015) 

5.1. SOURCE AND TARGET DATASETS FROM SIMILAR DOMAINS:

We initially consider the scenario of target datasets with classes that mostly exist in the source dataset.

In this scenario, DATL (Ngiam et al., 2018) was proven to be effective. We evaluate L2TL on five datasets focusing on different subsets. The detailed dataset splits are presented in Appendix A. The reward is measured on the validation set using top-1 accuracy.

Table 1 : Transfer learning performance with ImageNet source dataset, compared to MixDCNN (Wang et al., 2015) , EMD (Cui et al., 2018) , OPAM (Peng et al., 2017) and DATL (Ngiam et al., 2018 ) ( * denotes the results fine-tuning baseline from (Ngiam et al., 2018) ). The results of L2TL along with fine-tuning and DATL benchmarks are shown in Table 1 . As can be seen, L2TL outperforms fine-tuning across all the datasets with 0.6%-1.3% relative accuracy difference, which demonstrates the strength of L2TL in selecting related source examples across various domains. DATL performs worse than fine-tuning on Birdsnap and Aircraft, unlike L2TL. This underlines the importance of leveraging the visual similarity in the ways beyond the label match as in DATL. The results using ANON, shown in Table 2 , shows that L2TL preserves its benefits in learning relatedness for much larger scale datasets.

Method:

Table 3 : Top chosen classes from ImageNet source dataset that are related to the target datasets. For Birds, the top source class is "bea eater" which is one of the bird species in ImageNet. The second top "aepyceros melampus" is an antelope that has narrow mouth, which is similar to some birds with sharp spout. The "valley" also matches the background in some images. For Cars, we interestingly observe the high-weight class "barrel, cask", which indeed include wheels and car-looking body types in many images. "Terrapin" is a reptile that crawls on the ground with four legs, whose shape looks like vehicles in some way. For Food, the high-weight classes are the least relevant intuitively but still contain classes with visually-relevant patterns -e.g., "caldron, cauldron" may contain images with food inside, but most "seashore, coast" are less related to food. Analysis of high-weight source samples: To build insights on the learned weights, we sample 10k actions from the policy and rank the source labels according to their weights. Table 3 shows the classes with the highest weights and Fig. 2 visualize a few representative examples from them. These demonstrate that L2TL can judiciously extract the related classes from the source based on the pattern/shape of the objects, or background scenes.

Figure 2 : Representative examples from the source datasets with high weight from L2TL for different target datasets. In most cases, we observe them to be highly-related to the target dataset.

Effectiveness of RL::

We study the effectiveness of RL training comparing it to two baselines: (i) random search: where the policy model is not optimized and random actions are chosen as the policy output, and (ii) uniform weights: a constant importance weight is assigned to all training samples. Note that for these baselines, α is still optimized via policy gradient. The hyperparameters are cross-validated on the validation set. We show the best results of the baselines. As shown in Fig. 3 , L2TL outperforms both after sufficient number of iterations, demonstrating the importance of reweighting via policy gradient.

Low-shot regime: In the extreme regime of very small number of training examples, generalizing to unseen examples is most challenging as the model can be prone to overfitting. Fig. 4 shows performance in the small data regime for the same datasets. In most cases, we observe significant increase in performance when the number of examples per class is smaller. For five examples per class, the gap is as high as 6.5% (for Stanford Car). We observe that the gap between the L2TL and the fine-tune baseline often becomes smaller when more examples are used, but still remains as high as 1.5% In this section, we consider two target datasets with classes that do not exist in source dataset:

• Describable Textures Dataset (DTD) dataset (Cimpoi et al., 2014) : It contains textural images in the wild from 47 classes such as striped and matted. The dataset has 20 splits and we evaluate the testing results on the first split. Each training, validation, and testing split has 1,880 images. • Chest X-Ray Dataset CheXpert (Irvin et al., 2019) : This medical dataset has been recently introduced for chest radiograph interpretation. It consists of 224,316 chest radiographs of 65,240 patients labeled for 14 observations as positive, negative, or uncertain. Following (Irvin et al., 2019) , we report AUC on five classes and we regard "uncertain" examples as positive. 6 For L2TL, we use the mean AUC as the reward.

Table 4 shows the results for DTD. We observe that ImageNet fine-tuning greatly improves the classification results. L2TL further improves the fine-tuning baseline by 1.5%, demonstrating the significance of L2TL enabling the use of related source classes instead of using all classes. For the low-shot example case, we observe a more than 5% gain when only 10 examples are provided per class, validating the effectiveness of L2TL in more accurate transfer learning. Fig. 5 shows that L2TL is able to utilize visual similarities between the source and the target classes. Table 5 shows the results for CheXpert. L2TL performs better than the fine-tuning baseline by an AUC margin of .007. There are not many straightforward visual similarities to humans between ImageNet and CheXpert, but L2TL is still capable of discovering them to improve performance. The similarities occur in the form of texture pattern for most DTD classes. For example, "praire chicken" images from ImageNet typically contain patterns very relevant to "lined" from DTD. 

6. COMPUTATIONAL COST OF TRAINING:

Table 6 presents the computational cost for fine-tuning, DATL and L2TL. In DATL, given a new target dataset, a new model has to be trained on the resampled data until convergence. This step is time-consuming for large-scale source datasets. In L2TL, the transfer learning step is more expensive than fine-tuning, as it requires the computation on both source and target datasets. Yet, it only requires a single training pass on the source dataset, and yields much lower training time compared to DATL. 

7. CONCLUSIONS AND FUTURE WORK:

We propose a novel RL-based framework, L2TL, to improve transfer learning on a target dataset by careful extraction of information from a source dataset. L2TL considers joint optimization of models for source and target tasks, while using adaptive weights for scaling of constituent loss terms. We use the performance metric on the target validation set as the reward to train the policy model, which outputs the weights for each source class adaptively. We demonstrate state-of-the-art performance of L2TL for various datasets. The performance benefit of L2TL typically gets more significant as the target dataset size gets smaller. Our framework does not utilize an explicit similarity metric, but learns source class weights to directly optimize the target dataset performance. In cases where source and target datasets come from substantially different domains, L2TL still yields clear improvements. This improvement often comes from utilizing the source dataset classes that have relevant visual patterns despite belonging to a substantially different class.

Our general L2TL framework can be pushed beyond the approximations in this paper. A search space with a higher optimization granularity is expected to improve the results. Modeling the x dependence of the λ(x, y; Φ) function using a policy gradient can be a promising step towards that approach (particularly for the datasets with high intra-class variance, such a model can help to select particular samples within the class) albeit likely accompanied by increased computational complexity.

:

Without loss of generality, we can optimize a single weight αs[i] (setting αt[i] = 1) as the optimization is scale invariant.

Our reproduced results are matched with(Irvin et al., 2019) on mean AUC. However, there are variances as we can see that for some classes, we achieve slightly worse than(Irvin et al., 2019). This may because of the small number of validation examples (200) used.

Figures and Tables:

t e x i t s h a 1 _ b a s e 6 4 = " e p g f B I Y 9 B z P b q b

Figure 1: Overall diagram of L2TL. Dashed boxes correspond to trainable functions.

Figure 4: Number of examples per class vs. top-1 accuracy for L2TL and fine-tuning

Figure 5: Top source classes with the highest weight from ImageNet while transferring to DTD target. Representative images from each ImageNet class are shown along with related examples from DTD.The similarities occur in the form of texture pattern for most DTD classes. For example, "praire chicken" images from ImageNet typically contain patterns very relevant to "lined" from DTD.

1)1 Function arguments are not often shown in the paper for notational convenience.where (x, y) are the input and output pairs (x j , y j ∼ D S , x k , y k ∼ D T ), B S and B T are the source and target batch sizes 2 , α s [i] and α t [i] are the scaling coefficients at i th iteration, λ is the importance weighing function, f S (•; Ω, ζ S ) and f T (•; Ω, ζ T ) are encoding functions for the source and the target datasets with trainable parameters Ω, ζ S and ζ T 3 . To maximally benefit from the source dataset, a vast majority of the trainable parameters should be shared. If we consider the decompositions, f S

Table content: <table/>

containing 1.28M images from 1K classes, and also a much larger internal dataset, named as ANON dataset, containing ∼300M images from 18,291 classes to demonstrate scalability. Target datasets are chosen based on the scenarios. Hyperparameters of the encoder models are chosen from the published baselines and policy model parameters are cross-validated on a validation set. All hyperparameters are presented in Appendix B. For datasets that the testing accuracy is reported using the model trained on training and validation examples, L2TL is first trained on the training split using the reward from the validation set. Then, the learned control variables are then used to train the joint model on the training and validation examples. We do not use the test set during model training. For the fine-tuning experiments, we also use best set of hyperparameters evaluated on the validation set. The results are averaged over three runs.

Table content: <table/>

Results on the test set for Birds using images from ANON.

Table content: <table><tr><td/><td colspan="4">Target dataset test accuracy (%)</td><td/></tr><tr><td/><td>Birds</td><td>Pets</td><td colspan="3">Cars Aircraft Food</td></tr><tr><td>MixDCNN</td><td>74.8</td><td>-</td><td>-</td><td>82.5</td><td>-</td></tr><tr><td>EMD</td><td>-</td><td>-</td><td>91.3</td><td>85.5</td><td>88.7</td></tr><tr><td>OPAM</td><td>-</td><td>93.8</td><td>92.2</td><td>-</td><td>-</td></tr><tr><td>Fine-tuning  *</td><td>77.2</td><td>93.3</td><td>91.5</td><td>88.8</td><td>88.7</td></tr><tr><td>DATL  *</td><td>76.6</td><td>94.1</td><td>92.1</td><td>87.8</td><td>88.9</td></tr><tr><td>Fine-tuning</td><td>77.1</td><td>93.1</td><td>92.0</td><td>88.2</td><td>88.4</td></tr><tr><td>L2TL</td><td>78.1</td><td>94.4</td><td>92.6</td><td>89.1</td><td>89.2</td></tr><tr><td/><td>Method</td><td colspan="2">Accuracy (%)</td><td/><td/></tr><tr><td/><td colspan="2">Fine-tuning</td><td>74.9</td><td/><td/></tr><tr><td/><td>DATL</td><td/><td>81.7</td><td/><td/></tr><tr><td/><td>L2TL</td><td/><td>82.4</td><td/><td/></tr></table>

when 60 examples per class are used (for Birdsnap). The results show that L2TL can yield significant improvements in real-world tasks where the number of training examples are limited.

Table content: <table><tr><td>Top-1 accuracy</td><td colspan="2">0.600 0.625 0.650 0.675 0.700 0.725 0.750 0.775</td><td colspan="2">5000 10000 15000 20000 # of Iters Birdsnap Random Uniform L2TL</td><td colspan="2">Top-1 accuracy</td><td colspan="2">0.86 0.88 0.90 0.92 0.94</td><td colspan="2">Oxford-IIIT Pets 5000 10000 15000 20000 # of Iters Random Uniform L2TL</td><td colspan="2">Top-1 accuracy</td><td colspan="2">0.92 0.80 0.82 0.84 0.86 0.88 0.90</td><td colspan="2">Stanford Car 5000 10000 15000 20000 # of Iters Random Uniform L2TL</td><td>Top-1 accuracy</td><td colspan="2">0.800 0.775 0.600 0.625 0.650 0.675 0.700 0.725 0.750</td><td>FGVC Aircraft 5000 10000 15000 20000 # of Iters Random Uniform L2TL</td><td>Top-1 accuracy</td><td>0.90 0.88 0.76 0.78 0.80 0.82 0.84 0.86</td><td>Food-101 5000 10000 15000 20000 # of Iters Random Uniform L2TL</td></tr><tr><td colspan="20">Figure 3: Performance comparison between L2TL, random search and uniform weights. The curves</td></tr><tr><td colspan="20">are more oscillatory at the beginning, but become stable later during the training.</td></tr><tr><td>Top-1 accuracy</td><td>0.3 0.4 0.5 0.6 0.7</td><td colspan="2">20 # of examples per class 40 60 Birdsnap Finetune L2TL Performance gap</td><td colspan="2">0.01 0.00 0.01 0.02 0.03 0.04 0.05 Accuracy gap Top-1 accuracy</td><td colspan="2">0.86 0.88 0.90 0.92</td><td colspan="2">20 # of examples per class 40 60 Oxford-IIIT Pets Finetune L2TL Performance gap</td><td>0.01 0.00 0.01 0.02 0.03 0.04 0.05 Accuracy gap</td><td>Top-1 accuracy</td><td colspan="2">0.4 0.5 0.6 0.7 0.8</td><td colspan="2">5 # of examples per class 10 15 20 Stanford Car Finetune L2TL Performance gap</td><td>0.01 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Accuracy gap</td><td>Top-1 accuracy</td><td>0.3 0.4 0.5 0.6 0.7</td><td>5 # of examples per class 10 15 20 25 FGVC Aircraft Finetune L2TL Performance gap</td><td>0.01 0.00 0.01 0.02 0.03 0.04 0.05 Accuracy gap</td><td>Top-1 accuracy</td><td>0.4 0.5 0.6 0.7 0.8 0.9</td><td>0 # of examples per class 200 400 L2TL Finetune Food-101 Performance gap</td><td>0.010 0.005 0.000 0.005 0.010 0.015 Accuracy gap 0.020 0.025 0.030</td></tr></table>

Results on the test set for DTD on split 1.

Table content: <table><tr><td>Method</td><td>Acc</td></tr><tr><td>Random Init</td><td>57.4</td></tr><tr><td>Fine-tuning</td><td>70.3</td></tr><tr><td>L2TL</td><td>72.0</td></tr><tr><td colspan="2">10-shot, Fine-tuning 55.0</td></tr><tr><td>10-shot, L2TL</td><td>60.1</td></tr></table>

AUC comparisons (based on the same evaluation protocol as in(Irvin et al., 2019)) on the CheXpert dataset. * denotes the results reported in(Irvin et al., 2019).

Table content: <table><tr><td>Method</td><td colspan="6">Atelectasis Cardiomegaly Consolidation Edema Pleural Effusion Mean</td></tr><tr><td>Fine-tuning*</td><td>.858</td><td>.832</td><td>.899</td><td>.941</td><td>.934</td><td>.893</td></tr><tr><td>Fine-tuning</td><td>.852</td><td>.838</td><td>.900</td><td>.945</td><td>.928</td><td>.893</td></tr><tr><td>L2TL</td><td>.861</td><td>.844</td><td>.915</td><td>.948</td><td>.932</td><td>.900</td></tr></table>

Computational cost of training using Inception-V3 on Cloud TPU v2 when the source dataset is ImageNet. The last column of total time assumes availability of a pre-trained source model.

Table content: <table><tr><td>Method</td><td colspan="2">Number of iterations Pre-Transfer</td><td colspan="2">Time per iterations Pre-Transfer</td><td>From</td><td>Total time With pre-trained</td></tr><tr><td/><td>training</td><td>learning</td><td>training</td><td>learning</td><td>scratch</td><td>source model</td></tr><tr><td colspan="2">Fine-tuning 213,000</td><td>20,000</td><td>0.14s</td><td>0.21s</td><td>9.5h</td><td>1.2h</td></tr><tr><td>DATL</td><td>713,000</td><td>20,000</td><td>0.14s</td><td>0.21s</td><td>28.9h</td><td>20.6h</td></tr><tr><td>L2TL</td><td>213,000</td><td>20,000</td><td>0.14s</td><td>0.75s</td><td>12.5h</td><td>4.2h</td></tr></table>

Peer review (iclr2020):
Summary & Pros - This paper proposes a transfer learning approach based on fine-tuning and reinforcement learning. This approach adaptively computes the importance weights of classes in the source dataset, for maximizing the transfer effect. - The authors consider the importance weight as an action, and evaluation on the validation set as a reward. Using this formulation, they learn a policy network (i.e., importance weight) via policy gradient. This scheme does not require explicit similarity metrics between source and target samples. - The proposed method outperforms fine-tuning and DATL in several settings. Concerns #1: Verification of RL-based approach - Policy gradient algorithms should update policy networks to maximize the Q-value function. However, the proposed method considers maximizing only an instant reward at the current iteration in a greedy manner. Also, a number of episodes are typically required for learning while the proposed method considers only one episode. Thus it is hard to guarantee that policy gradient can find a good policy. - I think this method is more related to gradient approximation of objective (R) than RL because the advantage A_t can be considered as R(theta_t+1)-R(theta_t), so the gradient might be approximated as A_t*dP(lambda)/dtheta_t. - In Algorithm 1, computing lambda is deterministic. However, REINFORCE is based on stochastic policy. So I wonder how to apply REINFORCE into the proposed framework more precisely. - Detailed ablation studies on RL-based importance weights should be provided to demonstrate the effectiveness of RL, e.g., weight distribution while training, or using multi-step rewards instead of instant (or one-step) rewards. Concerns #2: Novelty of the proposed method - I think the novelty seems to be limited because the proposed method just trains class-wise importance weights using the validation set. - For training the weights, the proposed method uses REINFORCE without any modification to specialize this problem, thus I think this paper has a limited contribution. Concerns #3: Incremental improvements - The performance gains of L2TL over fine-tuning seem to be incremental even many samples in the source dataset are irrelevant to the target task (Table 1 & 4). I think ImageNet has many irrelevant samples to the fine-grained tasks, e.g., ImageNet has a small number of bird-related class. - For each target task, the proposed method requires all samples (>100GB) in the source dataset while fine-tuning just needs pre-trained models (< 100MB). Despite the high complexity, the performance gain is relatively small. - When fine-tuning, a regularization technique can also provide meaningful improvements on such fine-grained datasets, e.g., MaxEnt [1]. - Overall, to prove the effectiveness of the proposed method, I think it should provide more improvements. [1] Dubey, Abhimanyu, et al. *Maximum-Entropy Fine Grained Classification.* Advances in Neural Information Processing Systems. 2018.