Title: GRADMIX: MULTI-SOURCE TRANSFER ACROSS DO-MAINS AND TASKS

Abstract: The machine learning and computer vision community is witnessing an unprecedented rate of new tasks being proposed and addressed, thanks to the power of deep convolutional networks to find complex mappings from X to Y. The advent of each task often accompanies the release of a large-scale human-labeled dataset, for supervised training of the deep network. However, it is expensive and timeconsuming to manually label sufficient amount of training data. Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task. While previous works mostly focus on transfer learning from a single source, we study multi-source transfer across domains and tasks (MS-DTT), in a semi-supervised setting. We propose GradMix, a model-agnostic method applicable to any model trained with gradient-based learning rule. GradMix transfers knowledge via gradient descent, by weighting and mixing the gradients from all sources during training. Our method follows a meta-learning objective, by assigning layer-wise weights to the source gradients, such that the combined gradient follows the direction that can minimize the loss for a small set of samples from the target dataset. In addition, we propose to adaptively adjust the learning rate for each mini-batch based on its importance to the target task, and a pseudo-labeling method to leverage the unlabeled samples in the target domain. We perform experiments on two MS-DTT tasks: digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method against multiple baselines.

1. INTRODUCTION:

Deep convolutional networks (ConvNets) have significantly improved the state-of-the-art for visual recognition, by finding complex mappings from X to Y. Unfortunately, these impressive gains in performance come only when massive amounts of paired labeled data (x, y) s.t. x ∈ X , y ∈ Y are available for supervised training. For many application domains, it is often prohibitive to manually label sufficient training data, due to the significant amount of human efforts involved. Hence, there is strong incentive to develop algorithms that can reduce the burden of manual labeling, typically by leveraging off-the-shelf labeled datasets from other related domains and tasks.

There has been a large amount of efforts in the research community to address adapting deep models across domains (Ganin & Lempitsky, 2015; Long et al., 2016; Tzeng et al., 2017) , to transfer knowledge across tasks (Luo et al., 2017; He et al., 2017; Zamir et al., 2018) , and to learn efficiently in a few shot manner (Finn et al., 2017; Ravi & Larochelle, 2017; Ren et al., 2018a) . However, most works focus on a single-source and single-target scenario. Recently, some works (Xu et al., 2018; Mancini et al., 2018) propose deep approaches for multi-source domain adaptation, but they assume that the source and target domains have shared label space (task).

In many computer vision applications, there often exist multiple labeled datasets available from different domains and/or tasks related to the target application. Hence, it is important and practically valuable that we can transfer knowledge from as many source datasets as possible. In this work, we formalize this problem as multi-source domain and task transfer (MS-DTT). Given a set of labeled source dataset, S = {S 1 , S 2 , ..., S k }, we aim to transfer knowledge to a sparsely labeled target dataset T . Each source dataset S i could come from a different domain compared to T , or from a different task, or different in both domain and task. We focus on a semi-supervised setting, where only few samples in T have labels.

Most works achieve domain transfer by aligning the feature distribution of source domain and target domain (Long et al., 2015b; 2016; Ganin & Lempitsky, 2015; Tzeng et al., 2015; Mancini et al., 2018; Xu et al., 2018) . However, this method could be suboptimal for MS-DTT. The reason is that in MS-DTT, the distribution of source data p(x Si , y Si ) and target data p(x T , y T ) could be significantly different in both input space and label space, thus simply aligning their input space may generate indiscriminative features for the target classes. In addition, feature alignment introduces additional layers and loss terms, which require careful design to perform well.

In this work, we propose a generic and scalable method, namely GradMix, for semi-supervised MS-DTT. GradMix is a model-agnostic method, applicable to any model that uses gradient-based learning rule. Our method does not introduce extra layers or loss functions for feature alignment. Instead, we perform knowledge transfer via gradient descent, by weighting and mixing the gradients from all the source datasets during training. We follow a meta-learning paradigm and model the most basic assumption: the combined gradient should minimize the loss for a set of unbiased samples from the target dataset. We propose an online method to weight and mix the source gradients at each training iteration, such that the knowledge most useful for the target task is preserved through the gradient update. Our method can adaptively adjust the learning rate for each mini-batch based on its importance to the target task. In addition, we propose a pseudo-labeling method based on model ensemble to learn from the unlabeled data in target domain. We perform extensive experiments on two sets of MS-DTT task, including digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method compared to multiple baselines. Our code is available at https://www.url.com.

2. RELATED WORK:

Domain Adaptation. Domain adaptation seeks to learn from source domain a well-performing model on the target domain, by addressing the domain shift problem (Csurka, 2017) . Most existing works focus on aligning the feature distribution of the source domain and target domain. Several works attempt to learn domain-invariant features by minimizing Maximum Mean Discrepancy (Long et al., 2015b; 2016; Sun & Saenko, 2016) . Another class of method uses adversarial discriminative models, i.e. learn domain-agnostic representations by maximizing a domain confusion loss (Ganin & Lempitsky, 2015; Tzeng et al., 2015; Luo et al., 2017) . Recently, multi-source domain adaptation with deep model has been studied. Mancini et al. (2018 ) use DA-layers (Carlucci et al., 2017; Li et al., 2017b) to minimize the distribution discrepancy of network activations. Xu et al. (2018) propose multi-way adversarial domain discriminator that minimizes the domain discrepancies between the target and each of the sources. However, both methods (Mancini et al., 2018; Xu et al., 2018) assume that the source and target domains have a shared label space.

Transfer Learning. Transfer learning extends domain adaptation into more general cases, where the source and target domain could be different in both input space and label space (Pan & Yang, 2010; Weiss et al., 2016) . In computer vision, transfer learning has been widely studied to overcome the deficit of labeled data by adapting models trained for other tasks. With the advance of deep supervised learning, ConvNets trained on large datasets such as ImageNet (Russakovsky et al., 2015) have achieved state-of-the-art performance when transfered to other tasks (e.g. object detection (He et al., 2017) , semantic segmentation (Long et al., 2015a) , image captioning (Donahue et al., 2015) , etc.) by simple fine-tuning. In this work, we focus on the setting where source and target domains have the same input space and different label spaces.

Meta-Learning. Meta-learning aims to utilize knowledge from past experiences to learn quickly on target tasks, from only a few annotated samples. Meta-learning generally seeks performing the learning at a level higher than where conventional learning occurs, e.g. learning the update rule of a learner (Ravi & Larochelle, 2017) , or finding a good initialization point that can be easily fine-tuned (Finn et al., 2017) . Recently Li et al. (2018) propose a meta-learning method to train models with good generalization ability to novel domains. Our method follows the meta-learning paradigm that uses validation loss as the meta-objective. Our method also resembles the example reweighting method by Ren et al. (2018b) . However, they reweight samples in a batch for robust learning against noise, whereas we reweight source domain gradients layer-wise for transfer learning.

3. METHOD:



3.1. PROBLEM FORMULATION:

We first formally introduce the semi-supervised MS-DTT problem. We assume that there exists a set of k source domains S = {S 1 , S 2 , ..., S k }, and a target domain T . Each source domain S i contains N Si images, x Si ∈ X Si , with associated labels y Si ∈ Y Si . Similarly, the target domain consists of N T unlabeled images, x T ∈ X T , as well as M T labeled images, with associated labels y T ∈ Y T . We assume that the target domain is only sparsely labeled, i.e. M T N T . Our goal is to learn a strong target classifier that can predict labels y T given x T .

Different from standard domain adaptation approaches that assume a shared label space between source and target domain (Y S = Y T ), we study the problem of joint transfer across domains and tasks. Each source domain could have a partially overlapping label space with the target domain

(Y Si ∩ Y T ⊂ Y T and Y Si ∩ Y T = ∅), or a non-overlapping label space (Y Si ∩ Y T = ∅)

. However, we presume that at least one source domain should have the same label space as the target domain

(∃S i s.t. Y Si = Y T ).

3.2. META-LEARNING OBJECTIVE:

Let Θ denote the network parameters for our model. We consider a loss function L(x, y; Θ) = f (Θ) to minimize during training. For deep networks, SGD or its variants are commonly used to optimize the loss functions. At every step n of training, we forward a mini-batch of samples from each of the source domain {S i } k i=1 , and apply back-propagation to calculate the gradients w.r.t the parameters Θ n , ∇f si (Θ n ). The parameters are then adjusted according to the sum of the source gradients. For example, for vanilla SGD:

EQUATION

where α is the learning rate.

In semi-supervised MS-DTT, we also have a small validation set V that contains few labeled samples from the target domain. We want to learn a set of weights for the source gradients, w = {w si } k i=1 , such that when taking a gradient descent using their weighted combination k i=1 w si ∇f si (Θ n ), the loss on the validation set is minimized:

EQUATION

EQUATION

3.3. LAYER-WISE GRADIENT WEIGHTING:

Calculating the optimal w * requires two nested loops of optimization, which can be computationally expensive. Here we propose an approximation to the above objective. At each training iteration n, we do a forward-backward pass using the small validation set V to calculate the gradient, ∇f V (Θ n ).

We take a first-order approximation and assume that adjusting Θ n in the direction of ∇f V (Θ n ) can minimize f V (Θ n ). Therefore, we find the optimal w * by maximizing the cosine similarity between the combined source gradient and the validation gradient:

EQUATION

where a, b denotes the cosine similarity between vector a and b. This method is a cheap estimation for the meta-objective, which can also prevent the model from over-fitting to V.

Instead of using a global weight value for each source gradient, we propose a layer-wise gradient weighting, where the gradient for each layer of the network are weighted separately. This enables a finer level of gradient combination. Specifically, in our MS-DTT setting, the source domains and the target domain share the same parameters up to the last fully-connected (fc) layer, which is task-specific. Therefore, for each layer l with parameter θ l , and for each source domain S i , we have a corresponding weight w l si . We can then write Equation 4 as:

EQUATION

)

where L is the total number of layers for the ConvNet. We constrain w l s i ≥ 0 for all i and l, since negative gradient update can usually result in unstable behavior. To efficiently solve the above constrained non-linear optimization problem, we utilize a sequential quadratic programming method, SLSQP, implemented in NLopt (Johnson) .

In practice, we normalize the weights for each layer across all source domains so that they sum up to one:

wl si = w l si k i=1 w l si (6) 3.4 ADAPTIVE LEARNING RATE

Intuitively, certain mini-batches from the source domains contain more useful knowledge that can be transferred to the target domain, whereas some mini-batches contain less. Therefore, we want to adaptively adjust our training to pay more attention to the important mini-batches. To this end, we measure the importance score ρ of a mini-batch using the cosine similarity between the optimally combined gradient and the validation gradient:

EQUATION

Based on ρ, we calculate a scaling term η bounded between 0 and 1:

EQUATION

)

where β controls the rate of saturation for η, and γ defines the value of βρ where η = 0.5. We determine the value of β and γ empirically through experiments.

Finally, we multiply η to the learning rate α, and perform SGD to update the parameters:

EQUATION

3.5. PSEUDO-LABEL WITH ENSEMBLES:

In our semi-supervised MS-DTT setting, there also exists a large set of unlabeled images in the target domain, denoted as

U = {(x T n )} N T n=1 .

We want to learn target-discriminative knowledge from U. To achieve this, we propose a method to calculated pseudo-labels ŷT n for the unlabeled images, and construct a pseudo-labeled dataset

S u = {(x T n , ŷT n )} N p n=1 .

Then we leverage S u using the same gradient mixing method as described above. Specifically, we consider a loss L u (x, ŷ; Θ) to minimize during training, where (x, ŷ) ∈ S u . At each training iteration n, we sample a mini-batch from S u , calculate the gradient ∇f su (Θ n ), and combine it with the source gradients {∇f si (Θ n )} k i=1 using the proposed layer-wise weighting method.

In order to acquire the pseudo-labels, we perform a first step to train a model using the source domain datasets following the proposed gradient mixing method, and use the learned model to label U. However, the learned model would inevitably create some false pseudo-labels. Previous studies found that ensemble of models helps to produce more reliable pseudo-labels (Saito et al., 2017; Laine & Aila, 2017; Tarvainen & Valpola, 2017) . Therefore, in our first step, we train multiple models with different combination of β and γ in Equation 8. Then we pick the top R models with the best accuracies on the hyper-validation set (R = 3 in our experiments), and use their ensemble to create Here we propose two approaches to create pseudo-labels, namely hard label and soft label:

Hard label. In this approach, we assume that the pseudo-label is more likely to be correct if all the models can reach an agreement with high confidence. We assign a pseudo-label ŷ = C to an image x ∈ U, where C is a class number, if the two following conditions are satisfied. First, all of the R models should predict C as the class with the maximum probability. Second, for all models, the probability for C should exceed certain threshold, which is set as 0.8 in our experiments. If these two conditions are satisfied, we will add (x, ŷ) into S u . During training, the loss L u (x, ŷ; Θ) is the standard cross entropy loss.

Soft label. Let p r denote the output from the r-th model's softmax layer for an input x, which represents the probability over classes. We calculate the average of p r across all of the R pre-trained models as the soft pseudo-label for x: ŷ = 1 R R r=1 p r . Every unlabeled image x ∈ U will be assigned a soft label and added to S u . During training, let p Θ be the output probability from the model, we want to minimize the KL-divergence between p Θ and the soft pseudo-label for all pairs (x, ŷ) ∈ S u . Therefore, the loss is L u (x, ŷ; Θ) = D KL (p Θ , ŷ).

For both hard and soft label approach, after getting the pseudo-labels, we train a model from scratch using all available datasets {S i } k i=1 , S u and V. Since the proposed gradient mixing method relies on V to estimate the model's performance on the target domain, we enlarge the size of V to 100 samples per class, by adding hard-labeled images from S u using the method described above. The enlarged V can represent the target domain with less bias, which helps to calculate better weights on the source gradients, such that the model's performance on the target domain is maximized.

4. EXPERIMENT:



4.1. EXPERIMENTAL SETUP:

Datasets. In our experiment we perform MS-DTT across two different groups of data settings, as shown in Figure 1 . First we do transfer learning across different digit domains using MNIST (LeCun et al., 1998) and Street View House Numbers (SVHN) (Netzer et al., 2011) For our second setup, we study MS-DTT from human activity images in MPII dataset (Andriluka et al., 2014) and human action images from the Web (BU101 dataset) (Ma et al., 2017) , to video action recognition using UCF101 (Soomro et al., 2012) dataset. MPII dataset consists of 28,821 images covering 410 human activities including home activities, religious activities, occupation, etc. UCF101 is a benchmark action recognition dataset collected from YouTube. It consists of 13,320 videos from 101 action categories, captured under various lighting conditions with camera motion Table 1 : Classification accuracy (%) of the baselines and our method on the test split of MNIST 5-9. We report the mean and the standard error of each method across 10 runs with different randomly sampled V. Network and implementation details. For our first setting, we use the same ConvNet architecture as Luo et al. (2017) , which has 4 Conv layers and 2 fc layers. We randomly initialize the weights, and train the network using SGD with learning rate α = 0.05, and a momentum of 0.9. For fine-tuning we reduce the learning rate to 0.005. For our second setting, we use the ResNet-18 (He et al., 2016) architecture. We initialize the network with ImageNet pre-trained weights, which is important for all baseline methods to perform well. The learning rate is 0.001 for training and 5e-5 for fine-tuning.

Method:

4.2 SVHN 5-9 + MNIST 0-4 → MNIST 5-9

Experimental setting. In this experiment, we define four sets of training data: (1) labeled images of digits 5-9 from the training split of SVHN dataset as the first source S 1 , (2) labeled images of digits 0-4 from the training split of MNIST dataset as the second source S 2 , (3) few labeled images of digits 5-9 from the training split of MNIST dataset as the validation set V, (4) unlabeled images from the rest of the training split of MNIST 5-9 as U. We subsample k examples from each class of MNIST 5-9 to construct the unbiased validation set V. We experiment with k = 2, 3, 4, 5, which corresponds to 10, 15, 20, 25 labeled examples. Since V is randomly sampled, we repeat our experiment 10 times with different V. In order to monitor training progress and tune hyper-parameters (e.g. α, β, γ), we split out another 1000 labeled samples from MNIST 5-9 as the hyper-validation set. The hyper-validation set is the traditional validation set, which is fixed across the 10 runs.

Baselines. We compare the proposed method to multiple baseline methods:

(1) Target only: the model is trained using V.

(2) Source only: the model is trained using S 1 and S 2 without gradient reweighting.

(3) Fine-tune: the Source only model is fine-tuned using V. (4) MDDA (Mancini et al., 2018) : Multi-domain domain alignment layers that shift the network activations for each domain using a parameterized transformation equivalent to batch normalization. ( 5) DCTN (Xu et al., 2018 ): Deep Cocktail Network, which uses multi-way adversarial adaptation to align the distribution of multiple source domains and the target domain.

We also evaluate different variants of our model with and without certain component to show its effect: (6) GradMix w/o AdaLR: the method in Section 3.3 without the adaptive learning rate (Section 3.4). ( 7) GradMix: the proposed method that uses S 1 , S 2 and V during training. (8) GradMix w/ hard label: using the hard label approach to create pseudo-labels for U, and train a model with all available datasets. (9) GradMix w/ soft label: using the soft label approach to create pseudo-labels for U, and train a model with all available datasets.

Results. Table 1 shows the results for methods described above. We report the mean and standard error of classification accuracy across 10 runs with randomly sampled V. Methods in the upper part of the table do not use the unlabeled target domain data U. Among these methods, the proposed GradMix has the best performance. If we remove the adaptive learning rate, the accuracy would decrease. As expected, the performance improves as k increases, which indicates more samples in Ablation Study. In this section, we perform multiple ablation experiments to demonstrate the effectiveness of our method and the effect of different hyper-parameters. First, Figure 2 shows two examples of the hyper-validation loss as training proceeds. We show the loss for the baseline Source only method and the proposed GradMix, where we perform hyper-validation every 100 mini-batches (gradient descents). In both examples with different k, GradMix achieves a quicker and steadier decrease in the hyper-validation loss.

In Table 2 , we show the results using GradMix with different combination of β and γ when k = 3. We perform a grid search with β = [5, 6, ..., 10] and γ = [0, 0.1, ..., 0.8]. The accuracy is the highest for β = 10 and γ = 0.6. The top three models are selected for ensemble to create pseudo-labels for the unlabeled set U.

In addition, we perform experiments with different number of models used for ensemble. Table 3 shows the results for R = 1, 2, 3 across all values of k. R = 1 and R = 2 have comparable performance, whereas R = 3 performs better. This indicates that using the top three models for ensemble can create more reliable pseudo-labels. We experiment with k = 3, 5, 10 which corresponds to 303, 505, 1010 video clips. Each experiment is run two times with different V. We report the mean accuracy across the two runs for both per-frame classification and per-video classification. Per-frame classification is the same as doing individual image classification for every frame in the video, and per-video classification is done by averaging the softmax score for all the frames in a video as the video's score.

Baselines. We compare our method with multiple baselines described in Section 4.2, including (1) Target only, (2) Source only, (3) Fine-tune. In addition, we evaluate another baseline for knowledge transfer in action recognition, namely (4) EnergyNet (Li et al., 2017a) : The ConvNet (ResNet-18) is first trained on MPII and BU101, then knowledge is transfered to UCF101 through spatial attention maps using a Siamese Energy Network.

Results. Table 4 shows the results for action recognition. Target only has better performance compared to Source only even for k = 3, which indicates a strong distribution shift between source data and target data for actions in the wild. For all values of k, the proposed GradMix outperforms baseline methods that use S 1 , S 2 and V for training in both per-frame and per-video accuracy. GradMix also has comparable performance with MDDA that uses the unlabeled dataset U. The proposed pseudo-label method achieves significant gain in accuracy by assigning hard labels to U and learn target-discriminative knowledge from the pseudo-labeled dataset.

5. CONCLUSION:

In this work, we propose GradMix, a method for semi-supervised MS-DTT: multi-source domain and task transfer. GradMix assigns layer-wise weights to the gradients calculated from each source objective, in a way such that the combined gradient can optimize the target objective, measured by the loss on a small validation set. GradMix can adaptively adjust the learning rate for each mini-batch based on its importance to the target task. In addition, we assign pseudo-labels to the unlabeled samples using model ensembles, and consider the pseudo-labeled dataset as a source during training. We validate the effectiveness our method with extensive experiments on two MS-DTT settings, namely digit recognition and action recognition. GradMix is a generic framework applicable to any models trained with gradient descent. For future work, we intend to extend GradMix to other problems where labeled data for the target task is expensive to acquire, such as image captioning.

:



Figures and Tables:

Figure 1: An illustration of the two experimental settings (digit recognition and action recognition) for multi-source domain and task transfer (MS-DTT). Our method effectively transfers knowledge from multiple sources to the target task.

Figure 2: Loss on the hyper-validation set as training proceeds. We define 1 epoch as training for 100 mini-batches (gradient descents).

. MNIST is the popular benchmark for handwritten digit recognition, which contains a training set of 60,000 examples, and a test set of 10,000 examples. SVHN is a real-word dataset consisting of images with colored background and blurred digits. It has 73,257 examples for training and 26,032 examples for testing.

Table content: <table/>

We take the first split of UCF101 for our experiment. BU101 contains 23,800 images collected from the Web, with the same action categories as UCF101. It contains professional photos, commercial photos, and artistic photos, which can differ significantly from video frames.

Table content: <table><tr><td/><td>Datasets</td><td>k=2</td><td>k=3</td><td>k=4</td><td>k=5</td></tr><tr><td>Target only</td><td>V</td><td colspan="4">71.35±1.85 77.15±1.36 81.43±1.41 84.83±1.10</td></tr><tr><td>Source only</td><td>S 1 , S 2</td><td>82.39</td><td>82.39</td><td>82.39</td><td>82.39</td></tr><tr><td>Fine-tune</td><td>S 1 , S 2 , V</td><td colspan="4">89.94±0.35 89.86±0.46 90.89±0.48 91.96±0.39</td></tr><tr><td>GradMix w/o AdaLR</td><td>S 1 , S 2 , V</td><td colspan="4">90.10±0.37 90.22±0.62 92.14±0.43 92.92±0.29</td></tr><tr><td>GradMix</td><td>S 1 , S 2 , V</td><td colspan="4">91.17±0.37 91.45±0.52 92.14±0.40 93.06±0.46</td></tr><tr><td colspan="6">MDDA (Mancini et al., 2018) S 1 , S 2 , V, U 90.23±0.40 90.28±0.50 91.45±0.37 91.85±0.31</td></tr><tr><td>DCTN (Xu et al., 2018)</td><td colspan="5">S 1 , S 2 , V, U 91.81±0.26 92.34±0.28 92.42±0.39 92.97±0.37</td></tr><tr><td>GradMix w/ soft label</td><td colspan="5">S 1 , S 2 , V, U 94.62±0.18 95.03±0.30 95.26±0.17 95.74±0.21</td></tr><tr><td>GradMix w/ hard label</td><td colspan="5">S 1 , S 2 , V, U 96.02±0.24 96.24±0.33 96.63±0.17 96.84±0.20</td></tr><tr><td>and occlusion.</td><td/><td/><td/><td/><td/></tr></table>

Results of GradMix w/ hard label using different number of pre-trained models for ensemble. can help the GradMix method to better combine the gradients during training. The lower part of the table shows methods that use all available datasets including S 1 , S 2 , V and U. The proposed GradMix without U can achieve comparable performance with state-of-the-art baselines that use U (MDDA and DCTN). Using pseudo-label with model ensemble significantly improves performance compared to baseline methods. Comparing soft label to hard label, the hard label approach achieves better performance. More detailed results about model ensemble for pseudo-labeling is shown later in the ablation study.

Table content: <table><tr><td>5</td><td>90.92</td><td>90.96</td><td>90.95</td><td>90.58</td><td>90.75</td><td>90.75</td><td>90.51</td><td>90.63</td><td>91.12</td></tr><tr><td>β = 6</td><td>90.41</td><td>90.75</td><td>89.95</td><td>90.79</td><td>90.59</td><td>89.95</td><td>90.58</td><td>90.63</td><td>90.56</td></tr><tr><td>β = 7</td><td>89.76</td><td>90.44</td><td>90.42</td><td>90.94</td><td>90.28</td><td>90.40</td><td>90.52</td><td>90.70</td><td>90.66</td></tr><tr><td>β = 8</td><td>90.05</td><td>90.89</td><td>90.93</td><td>90.57</td><td>90.77</td><td>90.69</td><td>89.99</td><td>90.58</td><td>90.71</td></tr><tr><td>β = 9</td><td>90.32</td><td>90.70</td><td>90.48</td><td>90.94</td><td>90.47</td><td>90.92</td><td>90.20</td><td>90.23</td><td>90.86</td></tr><tr><td colspan="2">β = 10 90.52</td><td>90.03</td><td>89.67</td><td>90.01</td><td>89.84</td><td>90.51</td><td>91.45</td><td>90.58</td><td>90.70</td></tr><tr><td/><td colspan="2">Num. of Models</td><td>k=2</td><td/><td>k=3</td><td>k=4</td><td/><td>k=5</td><td/></tr><tr><td/><td colspan="2">R = 1</td><td colspan="6">95.80±0.28 95.92±0.28 96.29±0.20 96.46±0.25</td><td/></tr><tr><td/><td colspan="2">R = 2</td><td colspan="6">95.30±0.23 95.80±0.18 96.18±0.22 96.57±0.26</td><td/></tr><tr><td/><td colspan="2">R = 3</td><td colspan="6">96.02±0.24 96.24±0.33 96.63±0.17 96.84±0.20</td><td/></tr><tr><td>V</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Classification accuracy (%) of the baselines and our method on the test split of UCF101. We report the mean accuracy of each method across two runs with different randomly sampled V. In the action recognition experiment, we have four sets of training data similar to the digit recognition experiment, which include (1) S 1 : labeled images from the training split of MPII, (2) S 2 : labeled images from the training split of BU101, (3) V: k labeled video clips per class randomly sampled from the training split of UCF101, (4) U: unlabeled images from the rest of the training split of UCF101.

Table content: <table><tr><td>Method</td><td>Datasets</td><td/><td>per-frame</td><td/><td/><td>per-video</td></tr><tr><td/><td/><td>k=3</td><td>k=5</td><td>k=10</td><td>k=3</td><td>k=5</td><td>k=10</td></tr><tr><td>Target only</td><td>V</td><td colspan="5">42.58 53.31 63.05 43.74 55.50 64.74</td></tr><tr><td>Source only</td><td>S 1 , S 2</td><td colspan="5">41.96 41.96 41.96 43.46 43.46 43.46</td></tr><tr><td>Fine-tune</td><td>S 1 , S 2 , V</td><td colspan="5">55.86 60.55 66.77 58.57 66.01 70.21</td></tr><tr><td>EnergyNet (Li et al., 2017a)</td><td>S 1 , S 2 , V</td><td colspan="5">55.93 60.82 66.73 58.70 66.23 70.25</td></tr><tr><td>GradMix</td><td>S 1 , S 2 , V</td><td colspan="5">56.25 61.73 67.30 59.41 66.27 71.49</td></tr><tr><td colspan="7">MDDA (Mancini et al., 2018) S 1 , S 2 , V, U 56.65 61.58 67.65 60.00 65.14 71.54</td></tr><tr><td>GradMix w/ hard label</td><td colspan="6">S 1 , S 2 , V, U 68.92 68.76 69.25 72.58 72.34 73.48</td></tr><tr><td colspan="2">4.3 MPII + BU101 → UCF101</td><td/><td/><td/><td/></tr><tr><td>Experimental setting.</td><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 The paper is proposing a method which is aimed for multi-source domain adaptation with weak supervision. In addition to the classical setup, paper also consider the case in which some of the source domains have a possibly non-overlapping label space with the target task. Proposed idea is rather simple and effective. The paper consider a model in which shared parameters are updated with convex combination of gradients, computed for available sources. The mixture weights for the convex combination is chosen such that the resulting update decreases the loss function of the few labelled examples from the target domain. This defines a bilevel optimization problem and paper uses a simple proxy which is minimising the cosine distance between the mixture and the gradient for the few labelled examples. The biggest strength of the paper is its effectiveness in large set of experiments. Considering that it is also very easy to implement, I would expect the idea to be useful for the deep learning community. Idea of using adaptive weights for each batch and using different mixture weights for different layers are interesting and novel. Having said that, the paper is not ready to be published yet in my honest opinion as I summarize the issues below. MAJOR ISSUES: - Experimental Setup: Although the method is largely compared with domain adaptation methods, the setup is almost identical to few shot learning. In few shot learning, the test(target) and train(source) distributions typically have different label distribution. Hence, all few-shot learning methods are valid baselines for this paper. I think recent ones should be included in the experimental study. I would also suggest to use few-shot datasets like OmniGlot and CARS for the experiments since the performance of few-shot learning algorithms on them is already widely presented in the literature. - Related Work: Although the paper does a good job comparing with related work in domain adaptation, it misses two recent work which poses a bilevel optimization problem for meta-learning in order to solve a similar problem. These references are: [Bilevel Programming for Hyperparameter Optimization and Meta-Learning, ICML 2018] and [Deep Bilevel Learning, ECCV 2018]. Although these papers addresses slightly different problems with different optimization methods, fundamental idea is closely related. Authors should discuss these works. - Correctness: I think the way the optimization problem (defined in Equation 5) is solved is NOT correct. First of all, the inner product is a linear operator; hence the problem is a linear program in terms of w_{s_i}^l. Hence, the statement that it is a non-linear optimization problem is not correct. Second of all, as long as any of the inner-products is positive, the optimal value is infinity as the w is not bounded. Even if it was bounded, it would be a one hot vector such that the weight corresponding to largest inner product is 1 and others are zero. I do not think the paper solves (eq 5) correctly. Please correct me if I misunderstood some part. MINOR ISSUES -Implementation Details: Paper is missing majority of implementation details. For example, how the parameters are shared between different domains and tasks? This is rather crucial and should be discussed in details. SUMMARY: I think the fundamental idea in the paper is interesting although a similar ideas have been used for similar problems already. I think the paper would still be useful to the community even after considering the limited novelty. However, I do not think the paper is correct. Moreover, I think it needs a stronger empirical study comparing with few-shot learning methods. Hence, I suggest authors to fix the correctness issue. extend the experiments with few-shot learning setup, and re-submit.