Title: WHAT DATA IS USEFUL FOR MY DATA: TRANSFER LEARNING WITH A MIXTURE OF SELF-SUPERVISED EXPERTS

Abstract: Transfer learning has proven to be a successful way to train high performing deep learning models in various applications for which little labeled data is available.
In transfer learning, one pre-trains a model on a large dataset such as Imagenet, and fine-tunes its weights on the target domain. In our work, we claim that in the new era of ever increasing number of massive datasets, selecting the relevant pretraining data is a critical issue. We introduce a new problem in which available datasets are stored in one centralized location called a dataserver. We assume that a client, a target application with its own small labeled dataset, is only interested in fetching a subset of the server's data that is most relevant to its own target domain. We propose a novel method that aims to optimally select subsets of data from the dataserver given a particular target client. We perform data selection by employing a mixture of experts model in a series of dataserver-client transactions with a small computational cost. We show the effectiveness of our work in various transfer learning scenarios, demonstrating state-of-the-art performance on several target datasets and tasks such as image classification, object detection and instance segmentation. We will make our framework available as a web-service, serving data to users aiming to improve performance in their A.I. application.

1. INTRODUCTION:

In the recent years, we have seen an explosive growth in both, the number and variety of A.I. applications. These range from generic image classification tasks, to surveillance, sports analytics, clothing recommendation, early disease detection, and mapping, among others. Yet, we are only in the beginning of our exploration of what is possible to achieve with Deep Learning.

One of the critical components of the new age A.I. applications is the need for labeled data. To achieve high-end performance, typically a massive amount of data needs to be used to train deep learning models. One way to mitigate the need for large-scale data annotation for each target application is via transfer learning in which a neural network is pre-trained (Chen et al., 2016; He et al., 2017; Shelhamer et al., 2017) on existing large-scale datasets and then fine-tuned on the target downstream task. While transfer learning is a well studied concept that has been proven successful in many domains (Chen et al., 2016; He et al., 2017; Shelhamer et al., 2017) , deciding which data to pre-train the model on is an open research question that has received surprisingly little attention in the literature. We argue that this, however, is a crucial problem to be answered in light of the ever increasing scale of the available data.

A website of curated computer vision benchmarks1 currently lists 367 public datasets, ranging from generic imagery, faces, fashion photos, to autonomous driving data. The sizes of datasets have also massively increased: the recently released OpenImages (Kuznetsova et al., 2018) contains 9M labeled images (600GB in size), and is 20 times larger compared to its predecessor MS-COCO (Lin et al., 2014) (330K images, 30GB) . The video benchmark YouTube8m (Abu-El-Haija et al., 2016) (1.9B frames, 1.5TB), is 800 times larger compared to Davis (Caelles et al., 2018 ) (10k frames, 1.8GB), while the autonomous driving dataset nuScenes (Caesar et al., 2019) contains 100× the number of images than KITTI (Geiger et al., 2012) . It is evident that even downloading and storing all these datasets locally may not be affordable for everyone, let alone pre-training a model on this massive amount of data. Furthermore, for commercial applications data licensing may be a financial issue to consider. Recent works (He et al., 2018; Ngiam et al., 2018) have also shown that there is not a "the more the better" relationship between the amount of pre-training data and the downstream task performance. Instead, they demonstrated that selecting an appropriate subset of the pre-training data was important to achieve good performance on the target dataset.

In this paper, we envision a new scenario in which all (public) datasets are stored in one centralized location, i.e., a dataserver, and made available for download per request by a client. A client can be anyone with its own A.I. application in mind, and has a small set of its own labeled target data. We assume that each client is only interested in downloading a subset of the server's data that is most relevant to its own target domain, limited to a pre-defined budget (maximum allowed size).

We further want the transaction between the dataserver and the client to be both, extremely efficient computationally, as well as privacy-preserving. That is, the client's data should not be visible to the server, whereas the server aims to minimize the amount of computation per client, as it may serve possibly many clients in parallel.

We propose a novel method that aims to optimally select subsets of data from a large dataserver given a particular target client, in the context described above. In particular, we represent the server's data with a mixture of experts model trained with a simple self-supervised task. This allows us to distill all of the server's data, even when it consists of several datasets featuring different types of labels, into the weights of a small number of experts. These experts are then used on the client's side to determine the most important subset of the data that the server is to provide to the client. We show significant improvements in performance on all downstream tasks compared to pre-training on a randomly selected subset of the same size. Furthermore, we show that with only 20% or 40% of pre-training data, our method achieves comparable or better performance that pre-training on the entire server's dataset.

We implement our framework as a web platform, i.e., a dataserver that links to a variety of large datasets, and enables each client to only download the relevant subset of data. Our platform will be made available online upon acceptance. Tremblay et al. (2018) . Specifically in the context of neural networks, fine-tuning a pre-trained model in a new dataset is the most common strategy for knowledge transfer. Several works (Sun et al., 2017; Mahajan et al., 2018; Caron et al., 2019) have examined the idea of pre-training in an "enormous data" scenario. That is, pre-training on datasets that are 300× (JFT (Sun et al., 2017) ), and 3000× (Instagram (Mahajan et al., 2018) ) larger than the frequently used ImageNet Deng et al. (2009) hand, study how the choice of pre-training data impacts performance on fine-grained classification tasks. Specifically, they show that pre-training on only relevant examples is important to achieve good performance. Our work builds on top of these observations but presents a scalable and efficient way to select the most useful subset of data in a distributed scenario where the transactions between a datacenter and a client should be both, computationally efficient and privacy-preserving. Furthermore, unlike most previous works that focus on classification, our approach can be used in a variety of tasks.

2. RELATED WORK:

Federated Learning. (McMahan et al., 2016; Bonawitz et al., 2017) introduced a distributed ML approach with the goal of training a centralized model on decentralized data over a large number of client devices, (i.e mobile phones). Our work shares the similar idea of restricting the visibility of data in a client-server model. However, in our case the data is centralized in a server and the clients exploit the transfer learning scenario.

3. OUR APPROACH:

We define a new problem in which a dataserver, i.e., a centralized database that has access to a massive source dataset, aims to provide relevant subset of data to a client that wants to improve the performance of its model on a downstream task by pre-training the model on this subset. The dataserver's dataset may or may not be completely labeled, and the types of labels (e.g., masks for segmentation, boxes for detection, or scene attributes) across data points may vary. The client's dataset is considered to only have a small set of labeled examples, where further the task (and thus the type of its labels) may or may not be the same as any of the tasks defined on the dataserver' dataset(s). The main challenge is posed in requiring the dataserver-client transactions to be scalable (on the server side) with respect to the number of clients, and affordable for the resource-limited client (e.g., cannot pre-train on a massive dataset), as well as privacy preserving (client's data cannot be shared with the server, i.e., mimicking the case where the client has sensitive data such as hospital records). Only the most relevant data points should be transmitted from the server to the client.

In our approach, we represent the dataserver's data using a mixture of experts learned (only once) on a self-supervised task. This naturally partitions the datasets into K different subsets of data and produces specialized neural networks whose weights encode the representation of each of those subsets. These experts are cached on the server and shared with each client, and used as a proxy to determine the importance of data points for the client's task. In particular, the experts are downloaded by the client and fast-adapted on the client's dataset. We experimentally validate that the accuracy of each adapted expert indicates the usefulness of the data partition used to train the expert on the dataserver. The server then uses these accuracies to construct the final subset of its data that is relevant for the client. In Figure 2 , we present an illustration of our framework, and summarize the method in Algorithm 2.

In Section 3.1, we formalize our problem. In Section 3.2, we describe how we can obtain expert models through mixture of experts and analyze the different choices of representation learning algorithms for the experts (server side). In Section 3.3.1, we propose how to exploit the experts' performance on the client's task for data selection.

3.1. PROBLEM AND TASK DEFINITION:

Let X denote the input space (images in this paper), and Y a a set of labels for a given task a.

Generally, we will assume that multiple tasks, each associated with a different set of labels, are 

EQUATION

Here, h Ŝ∪T indicates that h is trained on the union of data Ŝ and T . Intuitively, we are trying to find the subset of data from S that helps to improve the performance of the model on the target dataset.

However, what makes our problem particularly challenging and unique is that we are restricting the visibility of the data between the dataserver and the client. This means that fetching the whole sample set S is prohibitive for the client, as it is uploading its own dataset to the server. We tackle this problem by representing the dataserver's dataset with a set of classifiers that are agnostic of the client (Sec. 3.2)., and use these to optimize equation 1 on the client's side (Sec. 3.3.1).

3.2. DATASERVER:

We here introduce our representation of the dataserver. This representation is computed once and stored on the server.

3.2.1. DATASET REPRESENTATION WITH A MIXTURE OF EXPERTS:

We choose to represent the dataserver's data S using the mixture of experts model (Jacobs et al., 1991) . In this model, one makes a prediction as:

EQUATION

Here, g θ denotes a gating function, e θi denotes the i-th expert model given an input x, θ are learnable weights of the model, and K corresponds to the number of experts. One can think of the gating function as softly assigning data points to each of the experts, which try to make the best guess on their assigned data points. In our work, we propose to choose the data relevant to the client by 1) estimating the relevance of each expert on the client's dataset, and 2) use the gating function as a means to measure relevance of the original data points. We explain this in more detail in Sec 3.3.1.

In this section, we focus our description on how we train the experts.

Learning the mixture of experts model is done by defining an objective L and using maximumlikelihood estimation (MLE):

EQUATION

We discuss the choices for the objective L in Sec 3.2.2, dealing with the fact that the labels across the source datasets may be defined for different tasks.

While this objective can be trained end-to-end, the computational cost of doing so on a massive dataset is extremely high, particularly when K is relatively large (we need to backpropagate gradients to every expert on every training example). A straightforward way to alleviate this issue is to associate each expert with a local cluster defined by a hard gating, as used in (Hinton et al., 2015; Gross et al., 2017) . In practice, we define a gating function g that partitions the dataset into mutually exclusive subsets, and train one expert per subset. This makes training easy to parallelize as each expert is trained independently on its own subset of data.

In our experiments, we use two simple partitioning schemes to determine the gating: (1) superclass partition, and (2) unsupervised partition. For superclass partition, we represent each class c in the source dataset as the mean of the image features f c for category c, and perform k-means clustering over {f c }. This gives a partitioning where each cluster is a superclass containing a subset of similar categories. For unsupervised partitioning, we partition the source dataset using k-means clustering on the feature space of a pretrained neural network (i.e. features extracted from the penultimate layer of a network pre-trained on ImageNet).

3.2.2. TRAINING THE EXPERTS:

We discuss two different scenarios to train the experts. In the simplified scenario, the tasks defined for both the server's and client's datasets are the same, e.g., classification. In this case we simply train a classifier for each subset of the data in S. We next discuss the more challenging case where the tasks are different.

Ideally, we would like to learn a representation that can generalize to a variety of downstream tasks and can therefore be used in a task agnostic fashion. To this end, we use a self-supervised method on a pretext task to train the mixtures of experts. In self-supervision one leverages a simple surrogate task that can be used to learn a meaningful representation. Furthermore, it does not require any manually labeled data to train the experts which means that the dataserver's dataset may or may not be labeled beforehand. This is useful if the client desires to obtain raw data and label the relevant subset on its own.

To be specific, we select image rotation as a pseudo-task for self-supervision. In particular, we follow (Gidaris et al., 2018) which demonstrated to be a simple yet powerful proxy for representation learning. Formally, given an image x, we define its corresponding label ŷ by performing a set of geometric transformations {r(•, j)} 3 j=0 on x, where r is an image rotation operator, and j defines a particular rotation by one of the following predefined degrees {0, 90, 180, 270}. We then minimize the following learning objective for the mixture of experts:

EQUATION

3.3. SERVER-CLIENT TRANSACTION:

In this section, we describe the transaction between the server and client that determines the relevant subset of the server's data. The client first downloads the experts and uses these experts to measure their performance on the client's dataset. Since there is likely a domain gap between the source and the target datasets, we perform a quick adaptation of the experts on the client's side (Sec 3.3.1). The performance of each expert is sent back to the server, which uses this information as a proxy to determine which data points are relevant to the client (Sec. 3.3.2). We describe these steps in more detail in the following subsections.

3.3.1. FAST ADAPT TO A TARGET DATASET (ON CLIENT):

Single Task on Server and Client: We first discuss the case where the dataset task is the same for both the client and the server, e.g., classification. While the task may be the same, the label set may not be (classes may differ across domains). An intuitive way to adapt the experts would be to remove their classification head that was trained on the server, and learn a small decoder network on top of the experts's penultimate representations on the client's dataset, as in (Zamir et al., 2018) . For classification tasks, we learn a simple linear layer on top of each pre-trained expert's representation for a few epochs. We then evaluate the target's task performance on a held-out validation set using the adapted experts. We denote the accuracy for each expert i as z i .

Diverse Tasks on Server and Client: To generalize to unseen tasks and be further able to handle cases where the labels are not available on the client's side, we propose to evaluate the performance of the common self-supervised task used to train the experts on the server's data. Intuitively, if the expert performs well in the self-supervised task on the target dataset, then the data it was trained on is likely relevant for the client. Specifically, we use the self-supervised experts trained to learn image rotation, and evaluate the proxy task performance of predicting image rotation angles on the target images:

z i = 1 |T | x∈T arg max j {e θi (r(x, j))} 3 j=0 = j (5)

Note that in this case we do not adapt the experts on the target dataset (we only perform inference).

3.3.2. DATA SELECTION (ON SERVER):

We now aim to assign a weighting to each of the data points in the source domain S to reflect how well the source data contributed to the transfer learning performance. The accuracies z i from the client's FASTADAPT step for each expert are normalized to [0, 1] and fed into a softmax function with temperature T = 0.1. These are then used as importance weights w i for estimating how relevant is the representation learned by a particular expert for the target task's performance. We leverage this information to weigh the individual data points x. More specifically, each source data x is assigned a probabilistic weighting:

p(x) = K i=1 w i g i θ (x) 1 |S i | (6)

Here, |S i | represents the size of the subset that an expert e θi was trained on. Intuitively, we are weighting the set of images associated to the i-th expert and uniformly sampling from it. We construct our dataset by sampling examples from S at a rate according to p.

3.4. RELATION TO DOMAIN ADAPTATION:

If we assume that the client and server tasks are exactly the same then our problem can be interpreted as doing domain adaptation in each of the subset Ŝ and the following generalization bound from Ben-David et al. ( 2009) can be used:

EQUATION

where ε represents the risk of a hypothesis function h ∈ H and d H∆H is the H∆H divergence Ben-David et al. (2009) , which relies on the capacity of H to distinguish between data points from Ŝ and T , respectively.

Let us further assume that the risk of the hypothesis function h on any subset Ŝ is similar such that ε Ŝ (h) ≈ ε S (h) for every Ŝ ⊂ P(S) and h ∈ H. Under this assumption, minimizing equation 1 is equivalent to finding the subset S * that minimizes the divergence with respect to T . Formally,

EQUATION

In practice, it is hard to compute d H∆H and this is often approximated by a so called proxy Adistance Ben-David et al. ( 2007); Chen et al. (2015) ; Ganin et al. (2015) . A classifier that discriminates between the two domains and whose risk ε is used to approximate the second part of the equation. dH ≈ dA ≈ 2(1 -2ε) (9) Note that doing so would require having access to S and T in at least one of the two sides (i.e to train the new discriminative classifier) and this is prohibitive in our scenario. In our case, we compute the domain confusion between Ŝ and T by evaluating the performance of expert e i on the target domain. We argue that this proxy task performance (or error rate) is an appropriate proxy distance that serves the same purpose but does not violate the data visibility condition. Intuitively, if the features learned on the subset cannot be discriminated from features on the target domain, the domain confusion is maximized. We empirically show the correlation between the domain classifier and our proposed proxy task performance in our experiments. To see how well the performance of the proxy task reflects the domain confusion, we perform an experiment comparing the proxy task performance and dA ( Ŝ, T ). To estimate dA , we follow the same idea from Ben-David et al. (2007) ; Chen et al. (2015) ; Ganin et al. (2015) and for each subset Ŝ, we estimate the domain confusion. Figure 3 shows the domain confusion vs the proxy task performance using OxfordIIIT-Pets (Parkhi et al., 2012) dataset as the target domain. In this plot, the highest average loss corresponds to the subset with the highest domain confusion (i.e., S i that is the most indistinguishable from the target domain). Notice that this correlates with the expert that gives the highest proxy task performance.

4. EXPERIMENTS 4.1 TOY EXPERIMENT -DOMAIN CONFUSION:



4.2. EXPERIMENTAL SETUP:

We perform experiments in classification, detection, and instance segmentation tasks on two server datasets and seven client datasets. In our experiments, we first train expert models on the server dataset S, and then use the experts to select an optimal S * for each target dataset as described in Section 3.3.1. We evaluate the performance on the target task by pre-training on the selected subset S * , and use this as an initialization for training over the target dataset. For all self-supervised experts, we use ResNet18 (He et al., 2015) , and train our models to predict image rotations.

4.2.1. IMAGE CLASSIFICATION SETUP:

For classification tasks, we use the Downsampled ImageNet (Chrabaszcz et al., 2017) as our server dataset. This is a variant of ImageNet (Deng et al., 2009) resized to 32×32 resolution, with 1,281,167 training images from 1,000 classes. We consider several small classification datasets to be used as target datasets (Nilsback & Zisserman, 2008; Wah et al., 2011; Parkhi et al., 2012; Krause et al., 2013; Khosla et al., 2011) . We use ResNet18 (He et al., 2015) as the base network architecture, and an input size of 32 × 32 for all classification datasets. Once the subsets are selected, we pre-train on the selected S * and evaluate the transfer performance by fine-tuning on client (target) datasets.

4.2.2. OBJECT DETECTION AND INSTANCE SEGMENTATION SETUP:

For detection and segmentation experiments, we use MS-COCO (Lin et al., 2014) as our server dataset. We evaluate the results using the standard metrics on Cityscapes (Cordts et al., 2016) and KITTI (Geiger et al., 2012) as the target datasets. We use Mask R-CNN models with ResNet-FPN-50 backbone, and follow the same training procedure as (He et al., 2017) for all experiments. We keep all hyperparameters fixed across all training runs and vary the choice of server data used for pre-training.

4.3. RESULTS AND ANALYSIS:

We begin by investigating the impact of pre-training data sampled using our approach on the downstream performance. In Table 1 , we summarize our result for classification, object detection, and instance segmentation tasks by subsampling 20%, 40% of the source dataset to be used for pretraining. By carefully selecting a similar subset of pre-training data using our approach, we see an improvement on all downstream tasks performance compared with pre-training on randomly selected subset of the same size. Moreover, when using 20% or 40% of pre-train data, we see comparable or improved performance of using the selected subset compared to pre-training on the entire 100% of pre-train data.

For classification tasks, we compare our method with the approach recently proposed by Ngiam et al. where they sample data based on the probability over source dataset classes computed by pseudo- labeling the target dataset with a classifier trained on the source dataset. Note that this approach is limited to the classification task, and cannot handle diverse tasks. Furthermore, it does not scale to a growing dataserver. Our approach achieves comparable results to Ngiam et al. in classification, and can be additionally applied to source datasets with no classification labels such as MS-COCO or even datasets which are not labeled.

Figure 4 shows the AP (average precision averaged over intersection-over-union (IoU) overlap thresholds 0.5:0.95) and AP@50 (average precision computed at IoU threshold 0.5) for object detection and segmentation after fine-tuning the Mask R-CNN on Cityscapes and KITTI dataset. A general trend is that performance is improved by pre-training for the instance segmentation task using COCO compared to ImageNet pre-training (COCO 0%). This suggests that a pre-training task other than classification is beneficial to improve transfer performance on localization tasks such as detection and segmentation, and shows the importance of training data. Next, we can see that pretraining using subsets selected by our approach is 2-3% better than the uniform sampling baseline, and that using 40% or 50% of COCO yields comparable (or better) performance to using 100% of data for the downstream tasks on Cityscapes. Table 2 further shows the instance segmentation performance on the 8 object categories for Cityscapes.

In Table 3 , we compare different instantiations of our approach on five classification datasets. For all instantiations, pre-training on our selected subset significantly outperforms the pre-training on a randomly selected subset of the same size. Our result in Table 3 shows that under the same superclass partition, the subsets obtained through sampling according to the transferability measured by self-supervised experts (SP+SS) yield a similar downstream performance compared to sampling according to the transferability measured by the task-specific experts (SP+TS). This suggests that self-supervised training for the experts can successfully be used as a proxy to decide which data points from the source dataset are most useful for the target dataset.

5. CONCLUSION:

In this work, we propose a novel method that aims to optimally select subsets of data from a large dataserver given a particular target client. In particular, we represent the server's data with a mixture of experts trained on a simple self-supervised task. These are then used as a proxy to determine the most important subset of the data that the server should send to the client. We experimentally show that our method is general and can be applied to any pre-training and fine-tuning scheme and that our approach even handles the case where no labeled data is available (only raw data). We hope that our work opens a more effective way of performing transfer learning in the era of massive datasets.

:

https://www.visualdata.io/

Figures and Tables:

Figure 1: Different clients(target) and datasets in the dataserver (source). Images are randomly chosen from S *

Figure 3: Relationship between domain classifier and proxy task performance on subsets Ŝ.

Figure 4: Transfer learning on object detection and instance segmentation. We report results on Cityscapes (top row) and KITTI (bottom row) when sampling {20%, 40%, 50%} of MS-COCO images (server).

. Other work has tried to understand transfer learning in neural networks. In particular, Yosinski et al. studied factors affecting the transferability of representations learned on ConvNets with respect to network architectures, network layers, and training tasks. Zamir et al. examines the relationship between visual tasks and proposes a computational method for modelling the transferability between these.Cui et al. and Ngiam et al., on the other

Table content: <table><tr><td>Client</td><td/><td/></tr><tr><td>Server: Mixture of Experts</td><td/><td>Server: Output Data for Client</td></tr><tr><td/><td>Target Data</td><td/></tr><tr><td/><td/><td>Source Dataset</td></tr><tr><td>Source Dataset</td><td>Fast Adapt</td><td>ComputeWeight</td></tr><tr><td>send experts to</td><td>send transfer</td><td/></tr><tr><td>client</td><td>performance to</td><td/></tr><tr><td/><td>server</td><td>subset to download</td></tr><tr><td/><td colspan="2">client download subset of source data with given budget</td></tr><tr><td colspan="2">Figure 2: Overview of our method.</td><td/></tr></table>

Initialize representation learning algorithm E, number of experts K 2: g θ ← HARDGATING(S, K)

Table content: <table><tr><td/><td/><td colspan="2">Algorithm 2 Overview of our framework.</td></tr><tr><td/><td/><td colspan="2">1: Input: S, T</td></tr><tr><td/><td/><td>2:</td><td>{e θ i } ← MOE(DS , E, K)</td></tr><tr><td/><td>Section 3.2: partition S</td><td>3:</td><td>z ← FASTADAPT(T , {e θ i })</td></tr><tr><td/><td>into local subsets to obtain gating</td><td>4:</td><td>S *  ← OUTPUTDATA(S, z, b)</td></tr><tr><td>3:</td><td/><td>5:</td><td>return S *</td></tr><tr><td colspan="2">4: procedure MOE(S, E, K):</td><td colspan="2">6: Output: S *  ∈ S to download</td></tr><tr><td>5:</td><td>For i = 1, ..., K</td><td/><td/></tr><tr><td>6:</td><td>Run E on {x ∈ S|g θ (x)i = 1} to ob-</td><td/><td/></tr><tr><td>7:</td><td>tain expert e θ i return {e θ i }</td><td colspan="2">Algorithm 3 Client module</td></tr><tr><td colspan="2">8: 9: procedure OUTPUTDATA(S, z): 10: w ← Sof tmax(N ormalize(z)) 11: p(x) = K i=1 wi gi θ (x) 1 |S i | 12: Sample S *  from S at rate according to p</td><td colspan="2">1: procedure FASTADAPT(DT , {e θ i }): 2: Initialize logits z ∈ R K 3: For i = 1, ..., K 4: zi ← PERFORMANCE(e θ i , T ) Section 3.3.1: Evaluate transfer performance of E i on T</td></tr><tr><td>13:</td><td>return S *</td><td>5:</td><td>return z</td></tr><tr><td colspan="2">the subset S</td><td/><td/></tr></table>

Transfer learning results on classification, object detection, and instance segmentation. Each row corresponds to data selection method, and we indicate the size of the subset (e.g., either 20% or 40% of the entire source dataset). Each column corresponds to a target dataset.

Table content: <table><tr><td/><td>Target Task</td><td colspan="2">Classification (% accuracy)</td><td colspan="4">Detection (% box AP) Segmentation (% mask AP)</td></tr><tr><td/><td>Source Dataset</td><td colspan="2">Downsampled ImageNet</td><td>COCO</td><td/><td>COCO</td><td/></tr><tr><td/><td colspan="4">Target Dataset Oxford-IIIT Pets CUB200 Birds Cityscapes</td><td>KITTI</td><td>Cityscapes</td><td>KITTI</td></tr><tr><td>0%</td><td>Random Initialization</td><td>32.4</td><td>25.1</td><td>36.2</td><td>21.8</td><td>32.0</td><td>17.8</td></tr><tr><td colspan="2">100% Entire Dataset</td><td>79.1</td><td>57.0</td><td>41.8</td><td>28.6</td><td>36.5</td><td>22.1</td></tr><tr><td/><td>Uniform Sample</td><td>71.1</td><td>48.6</td><td>38.1</td><td>22.2</td><td>34.3</td><td>18.9</td></tr><tr><td>20%</td><td>(Ngiam et al., 2018)</td><td>81.3</td><td>54.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>Ours</td><td>82.0</td><td>54.8</td><td>40.7</td><td>27.3</td><td>36.1</td><td>21.0</td></tr><tr><td/><td>Uniform Sample</td><td>76.0</td><td>52.7</td><td>39.8</td><td>23.4</td><td>34.4</td><td>18.8</td></tr><tr><td>40%</td><td>(Ngiam et al., 2018)</td><td>81.0</td><td>57.4</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>Ours</td><td>81.5</td><td>57.3</td><td>42.2</td><td>26.7</td><td>36.7</td><td>21.2</td></tr></table>



Table content: <table><tr><td>Size</td><td colspan="4">Selection Method box AP mask AP mask AP50</td><td>car</td><td colspan="4">truck rider bicycle person</td><td>bus</td><td colspan="2">mcycle train</td></tr><tr><td>0%</td><td>-</td><td>36.2</td><td>32.0</td><td>57.6</td><td colspan="2">49.9 30.8</td><td>23.2</td><td>17.1</td><td>30.0</td><td>52.4</td><td>17.9</td><td>35.2</td></tr><tr><td>20%</td><td>Uniform Sample Ours</td><td>38.1 40.7</td><td>34.3 36.1</td><td>60.0 61.0</td><td colspan="2">50.0 34.2 51.3 35.4</td><td>24.7 25.9</td><td>19.4 20.4</td><td>32.8 33.9</td><td>52.0 56.9</td><td>18.9 20.8</td><td>42.1 44.0</td></tr><tr><td>40%</td><td>Uniform Sample Ours</td><td>39.8 42.2</td><td>34.4 36.7</td><td>60.0 62.3</td><td colspan="2">50.7 31.8 51.8 36.9</td><td>25.4 26.4</td><td>18.3 19.8</td><td>33.3 33.8</td><td>55.2 59.2</td><td>21.2 22.1</td><td>38.9 44.0</td></tr><tr><td>50%</td><td>Uniform Sample Ours</td><td>39.5 41.7</td><td>34.9 36.7</td><td>60.4 61.9</td><td colspan="2">50.8 34.8 51.7 37.2</td><td>26.3 26.9</td><td>18.9 19.6</td><td>33.2 34.2</td><td>55.5 56.7</td><td>20.8 22.5</td><td>38.7 44.5</td></tr><tr><td colspan="2">100% -</td><td>41.8</td><td>36.5</td><td>62.3</td><td colspan="2">51.5 37.2</td><td>26.6</td><td>20.0</td><td>34.0</td><td>56.0</td><td>22.3</td><td>44.2</td></tr></table>

Ablation experiments on gating and expert training. SP stands for Superclass Partition, UP for Unsupervised Partition, TS for Task-Specific experts (experts trained on classif. labels), and SS for Self-Supervised experts (experts trained to predict image rotation). Results reported are top-1 accuracy for all datasets.

Table content: <table/>

Peer review (iclr2020):
This paper is focused on simplifying the use of larger datasets (via pretraining models) for the purpose of transfer learning onto smaller domains/datasets. An alternative view of this paper is that it is focused on a more privacy-friendly manner of doing data selection in a client-server manner. In particular the paper proposes an interesting client-server architecture which allows for servers to potentially hold on to large datasets and have pretrained models. On the other hand, clients can leverage these models while sending minimal information to the server so as to get the server to return a subset of the data -- which the client can in turn use for pre-training / joint training. In this case the proposed technique is composed of a few steps: On the client side, the data is partitioned into a few clusters from which pretrained models are trained. Next these models are shared with the client, and then used to *adapt* the model (e.g. one additional layer on top of the pretrained model) to figure out the best performing (pretrained) model (and thus effectively the best server data clusters). Lastly data from these good-performing clusters can be appropriately sampled and sent to the client. On the plus side I liked this vision of a server-client manner of interacting and pulling datasets. The basic skeleton of the overall infrastructure also makes sense to me. That said I had a few concerns which make me believe that the paper could do with more work and experiments before it can realize its potential impact. In no specific order: - In general I would have wanted a far more nuanced understanding of the efficacy of the proposed transfer learning / data selection methodology. There have been numerous works in this domain (not just restricted to vision) and it felt like there wasn*t really any comparison with any of the more common approaches to the problem. For example: One common family of approaches to performing such data selection is to run some clustering or PCA-like dimensionality reduction and then find clusters in the larger dataset closer to the clusters / basis vectors of the target set. Another set of techniques directly work in a common embedding space to find similar data points Why wasn*t any such approach considered / discussed? There are also many interesting *active* learning style approaches to the problem which allow for you to iteratively select data based on performance on some available target dataset. Those too would be valid in this setup and fair comparison points right? - The evaluations also felt discombobulated from the motivation/exposition of the approach. The paper motivated by saying there exist multiple large pretraining datasets than can be used, but in the evaluations only one single dataset was used for each task -- the two datasets weren*t even combined! To me the ability to combine different datasets was something that held significant appeal about the problem and the paper and I really would have wanted to see that showcased / some positive evidence towards the same. - On a somewhat related note, the paper motivated by saying that some of these datasets are so large that clients cannot afford to download them or pretrain on them. If that is the case is 20-40% really going to be that different? Again given the motivation in the paper I would have liked to see some deeper analysis on this. - Significance testing is a key empirical practice and one I would request the authors to add. - It also felt that the paper*s exposition and techniques were somewhat unclear -- they seemed to be focused on classification tasks (e.g. the superclass partitioning) but then were trying to generalize to non-classification problems without satisfactory explanations of how these approaches would generlize - On a more minor note, I felt the discussion in the paper is very specific to vision tasks since language understanding tasks have very different trends and techniques (e.g. BERT -- where more pretraining data only helps the model). I would actually try to clarify this scope accordingly earlier in the paper.