Scalable Neural Data Server: A Data Recommender for Transfer Learning
Authors: Tianshi Cao
Sasha Doubov
David Acuna
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transfer Learning
Computer Vision
Data Recommendation

Abstract: Absence of large-scale labeled data in the practitioner's target domain can be a bottleneck to applying machine learning algorithms in practice. Transfer learning is a popular strategy for leveraging additional data to improve the downstream performance, but finding the most relevant data to transfer from can be challenging. Neural Data Server (NDS), a search engine that recommends relevant data for a given downstream task, has been previously proposed to address this problem (Yan et al., 2020). NDS uses a mixture of experts trained on data sources to estimate similarity between each source and the downstream task. Thus, the computational cost to each user grows with the number of sources and requires an expensive training step for each data provider.

To address these issues, we propose Scalable Neural Data Server (SNDS), a large-scale search engine that can theoretically index thousands of datasets to serve relevant ML data to end users. SNDS trains the mixture of experts on intermediary datasets during initialization, and represents both data sources and downstream tasks by their proximity to the intermediary datasets. As such, computational cost incurred by users of SNDS remains fixed as new datasets are added to the server, without pre-training for the data providers.

We validate SNDS on a plethora of real world tasks and find that data recommended by SNDS improves downstream task performance over baselines. We also demonstrate the scalability of our system by demonstrating its ability to select relevant data for transfer outside of the natural image setting.

Scalable Neural Data Server: A Data Recommender for Transfer Learning
Tianshi Cao 1,2,3
Sasha Doubov 1,2 David Acuna 1,2,3
Sanja Fidler 1,2,3
University of Toronto' Vector Institute? NVIDIA {jcao cao, doubovs, ,dubovs,davidj fcs.eerressttoreeeeeerr
Abstract
Absence of large-scale labeled data in the practitioner's target domain can be a bottleneck to applying machine learning algorithms in practice. Transfer learning is a popular strategy for leveraging additional data improve the downstream performance, but finding the most relevant data to transfer from can be challenging. Neural Data Server (NDS) [45], a search engine that recommends relevant data for a given downstream task, has been previously proposed to address this problem. NDS uses mixture experts trained on data sources to estimate similarity between each source and the downstream task. Thus, the computational cost to each user grows with the number of sources. To address these issues, we propose Scalable Neural Data Server (SNDS), large-scale search engine that theoretically index thousands datasets to serve relevant ML data to end users. SNDS trains the mixture of experts intermediary datasets during initialization, and represents both data sources and downstream tasks by their proximity to the intermediary datasets. As such, computational cost incurred SNDS users remains fixed datasets are added to the server. We validate SNDS on a plethora of real world tasks and find that data recommended by SNDS improves downstream task performance over baselines. We also demonstrate the scalability of SNDS by showing its ability select relevant data for transfer outside of the natural image setting.
Introduction
In recent years, machine learning (ML.) applications have taken foothold in many fields. Method- ological and computational advances have shown a trend in performance improvements achieved by using larger and larger training datasets, confirming that data is the fuel of modern machine learning. This fuel, however, is not free of cost. While the raw data itself (such as images, text, etc.) can be relatively easy to collect, annotating the data labour intensive endeavour. A popular and effective approach to reduce the need of labeling in target application domain is to leverage existing datasets via transfer learning. Transfer learning is the re-purposing of ML models trained dataset towards different target task [22). The performance of transfer learning is predicated on Although there are numerous datasets available finding the right dataset that will most benefit transfer- transfer-learning performance on the target domain is not simple problem.
source
the relevance of the source to the target 46
36
through various data sharing platforms 41
5
38
20
Consider a startup looking to train a debris detector for use on road cleaning robot, and suppose that they have collected a small, labeled dataset. They are looking for data with which to augment their training set. Ideally, they would find a dataset of roadside debris that is captured from a sensor similar their own. However, keyword searches for "roadside debris". would likely find data geared towards autonomous driving, which might be from one of the many sensor types (rgb, lidar, radar). Searching for their specific sensor type might lead data from similar setups, but of different objects.
"Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Scalable Neural Data Server. SNDS a scalable data recommendation sys- tem for transfer learning data. It can theoreti- cally index thousands of datasets by using pool of experts, trained once on intermediary datasets, and used for data matching. SNDS uses these intermediary datascts to compute similarities between the consumer's data and the server's datascts. Similarities are com- puted on the user's side with a computation cost that does not grow with the size of the server's ' data. User's s data privacy is preserved.
Figure I:
Data Froviders
Scalable Neural Data Server
Jalocc Sicta
lest brons
Vecomenene
FouteN
Bownlood
Ted Eopars
Slet
Clarr
Recommended Data
Rather than using keywords, an ideal system would find relevant data based on the training set itself. In this work, we envision a data search engine that indexes source data from data providers, and recommends relevant transfer learning data to consumers who have target task.
Neural DataServer (NDS) [5] has been proposed as data recommendation system for transfer learning. Given a target task (as represented by a target dataset), it uses expert models trained on splits of source data to make predictions
the target, which in turn used as a measure of similarity between the source and the target. Directly indexing a large number of datasets with NDS expensive, it would require each data provider to upload a trained expert, and any data consumer looking to select data for purchase must download and run all of the trained experts in order to evaluate similarity. Hence, for M data providers in the server, the computational and bandwidth cost to the data consumer scales linearly with M.
At the same time, using an expert model per data provider in NDS ensures that the data recom- mendations are highly relevant to the consumer. A data search engine that operates in a large-scale setting must be able to preserve the quality of recommendations, while disentangling the consumer's computational cost from the number of providers. The system should also preserve the data privacy properties from NDS, where the client's data is never exchanged with the server.
We propose Scalable Neural Data Server (SNDS), a search engine for data recommendation at scale. This is achieved by using an intermediary, public dataset to train fixed number of experts, and similarity measure that evaluates the relevance of the server's data for a given consumer via these experts. Compared to NDS, the computational cost of SNDS is constant for given consumer with respect to the M data providers, and so is the bandwidth cost between the consumer and the server. We validate SNDS experimentally and find that 1) SNDS generalizes in the sense that it recommends useful data for downstream tasks that are not directly represented by the intermediary datasets, 2) SNDS improves performance on downstream tasks over baselines, and is competitive with NDS, 3) SNDS is able to generalize to domains that are very dissimilar to the intermediary datasets.
2
Background
In this section, we first formalize our problem setting which is similar to that of NDS, and then provide brief overview of the data selection method introduced in NDS.
2.1 Problem setting
Let X' denote a sample space and label space. Suppose that there currently M data providers (which we call "sources"), each with data available for use in transfer learning. Let the sources be denoted pairs (1x,,)))..... (Xm, Ym,)) E (X' y)m The target task by the data consumer is repre- sented by a target specific distribution D supported over X' V. from which the target dataset E (X' x y) is drawn. The goal of the data consumer is to find data sources suitable for transfer learning to their target task. Specifically, we seek a mixture of sources S* that minimizes the following risk for the data consumer:
= [S ......... JMr, where S, denote a set of m, sample-label
T= y()., (Xn,
R(S) Ex ( (x))
(1)
while satisfying some budget constraint of |S|S b, as specified by the data consumer. Here, hsur is a predictive model X + y that trained on and T. practice, h often first trained on during pretraining, and then trained on T during finetuning.

---

Desiderata 2.1. To meet the requirements of data search engine, a potential solution should satisfy:
DI: Data in S T are not stored on the server.
D2: Data & is not revealed to the consumers before S* is selected.
D3: Data in T is not revealed to any data provider or any other consumers throughout the process.
D4: The computational and bandwidth cost for the consumer should be fixed as M (i.e the number of data providers) grows.
What makes this problem challenging is that D1-D3 prevents the sources and the target from being on the same device, and the D4 prevents the server from iterating through every example in S for responding to every data consumer. Our problem setting differs from that of NDS in that we consider the number of sources M as an extrinsic variable, i.e. something outside the control of the server. This necessitates solution that scales cheaply with the number of sources, as stated in D4.
2.2 Neural Data Server
NDS partitions the source data through either (1) superclass partition (2) unsupervised partition. In either case, image features are extracted with a pretrained network. The image features are either (1) averaged over samples of the same class and then clustered with k-means, or (2) clustered with k-means directly. We refer to each of these clusters as S; (i ranges from 1 to K) as these analogous to the sources considered in our problem formulation. Then, expert networks trained for rotation prediction on each partition. Specifically, for each image x and a rotation angle I E {0, 90, 180, 270]. let r(x, 0) denote the image rotated clockwise by 0. An expert is a function f + 33, where A is probability mass function of n + 1 choices. The loss minimized by expert f: is: L = ExEs, Σj= log fi (r(x,)));
Once an expert is trained, accuracy of image rotation prediction is evaluated on the target set:
23 1
Σ
Σ
larg
m max(fa j))x] == j).
4
XET 1=0
The importance weight wi for each partition i is computed as softmax(z):- The sample weight for each example in the source is computed as (xx) = Σ 1(x E Si) [..] which is used to sample examples without replacement from the union of source data sets. Exposing the models directly trained on the source datasets in NDS to the client introduces privacy leakage, as the trained experts may memorize certain examples in the source data. It has been shown that possible to reconstruct training images from trained CNNs by optimizing images from noisy priors 19 47 [T]. We note that while NDS satisfies D1, it does not satisfy D4 and there may also be privacy leak on D2.
3 Scalable Neural Data Server
Scalable Neural Data Server (SNDS) is a data search engine for transfer learning. It is designed with application to large data sharing platforms in mind, where each data provider holds source dataset. The server's goal is to find the most relevant data to a data consumer who has budget constraint for the amount of data they could afford. Since is not feasible enumerate through every example indexed by the server when responding to every consumer, we simplify the search problem from "find the most relevant examples" to "find the most relevant source datasets". This simplification was also implied by NDS, who uses experts to represent examples of each source split.
SNDS is scalable in that the number of experts downloaded by each data consumer, well as the amount of compute used in the transaction, remains constant regardless of the number of data providers. This satisfies the computational cost constraint, making it scalable to a large number of source datasets. SNDS is also privacy preserving for both the data consumer and the data providers: no source or target data, nor any models trained either data, is moved until the recommendation is made. We introduce an intermediary public dataset for training a fixed set of experts. Outputs of these experts source and target data are used feature for matching sources targets.
3.1 SNDS Overview
In SNDS, we assume that the server has access to a public dataset P. The server partitions P into K disjoint subsets, which we denote as (Px 1K The server then optimizes an expert E, belonging to
3

---

Figure 2: Diagram our Scalable Neural Data Server for recommending data to single consumer. Step I occurs during initialization. Steps 2 & 3 occur during indexing of data providers. Steps 4-7 occur during data selection. Steps 8-9 are data transactions external to SNDS.
Soalable Noural Data Server
Data Stuiing
some hypothesis class E, for some objective L with learning algorithm Az. for each Pa. We thereby obtain K experts ] k-1 The server defines a performance function P(D, E) X x E + R, which computes some metric using E on a dataset D. The output of P is single real number, but can also be vector valued. Each data provider runs the performance function on their data (Pt.1..... Ps.k) for each S; in S, where PLAA P(S,, Ex). At the same time, data consumer also evaluates the experts on their target dataset T with P to obtain representation pr = (Pi. ....T.....P.rr). These representations are communicated back to the server, who then computes similarity between the target representation and that of each source. Let sim b) BK x R - R denote a similarity function. We then have z = (Zi,.... 2M [2; = sim (p:, PT)) as the similarity scores between the target and each source. Similarity scores are provided to the data consumer the server's ss recommendation of sources. The data consumer may then use the similarity scores to select data sources for transfer learning.
S,. This provides a K dimensional representation
3.2 Server Initialization
In our experiments, we use the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 training set as the public dataset. K- K-means on image features is used to partition the ILSVRC2012 training split into 50 partitions. Our main requirement for the experts is that their output as observed by P correlates with transfer performance from their partition of the public dataset. Following NDS, we use image rotation prediction as the task for training experts, as this task has been shown to correlate well with domain confusion, which is in turn empirical estimate of transfer performance. As such, we train Eg with loss function L(Pk, Ex), where L(D, E) -xxe Σ log Er l)), Since experts only need to be trained once during initialization, the computational cost of this step does not contribute the marginal cost of indexing queries.
3.3 Indexing Sources
To add source S, to SNDS, we evaluate experts {Ex) k= on S, and store the results pa on the server. First, the experts trained in section 3.2 are downloaded by the data providers. Then, since the experts are trained for image rotation prediction, we use the rotation prediction accuracy as the evaluation metric. Specifically, E) d ExED Σi= ( max, (axx/
== j).
The representation of a source S; in the server is thus P: = (P(S, Ex)) 1- The computational cost of this step for data provider is x Km,, which satisfies the complexity constraint of ((m).
3.4 Data Selection
A data consumer queries SNDS by first downloading experts {E+)E_ and then evaluating each expert on their target dataset T. Similar to Sec. 3.3, evaluation function P is used to obtain pr (P(T, Ex = which is communicated back to the server. The asymptotic computational complexity of producing Pr is Onz). which meets our requirements listed in Sec. 2 Furthermore, the number of downloaded experts and evaluations is independent of M sources, and so SNDS satisfies D4 on the consumer-side
The server needs to weigh each source. For simplicity and computational efficiency, we choose normalized variant of cosine similarity because it is shift invariant in its input arguments and has a fixed output range. This alleviates the need to tune the selection strategy based on the performance metric. Specifically, we compute the channel mean as p
M Σisl Pi, and use this to obtain
1 Σ
M
centered performance vectors p'; -P; -p and p'r D' Pr P. We define the similarity function as sim(a, b)
||abb||" so that z (IP' [[pp' 1=1 ||]|||]a
|p''''''1'' M

---

We normalize via softmax with temperature т to obtain source weights w RM, The temperature parameter determines the trade-off between sampling greedily from the most relevant clusters and sampling uniformly from all clusters. Rather than assuming a constant value for T, we find the 7 that satisfies a target entropy value for w. Intuitively, this allows us to maintain the same trade-off between greedy vs. uniform sampling across different consumer datasets, and this approach outperforms fixed practice. More details on this in the supplementary material.
Finally, the server outputs w to the data consumer, who uses it to select source data. Computing w from the expert scores incurs an asymptotic cost of O(MK) to the server. This is the main operation that scales with the number of data providers in the system. We note that millions of dot products between low (K) dimensional vectors can be performed in seconds on modern hardware, and the computational cost in both NDS/SNDS is vastly overshadowed by the expert model evaluations on the consumer's s data which is precisely the cost SNDS is trying to minimize.
3.5 Differences with NDS
SNDS implements many similar techniques as NDS, but has significant differences to NDS conceptu- ally. We highlight the main differences between SNDS and NDS:
1. NDS is designed to find data from few datasets. SNDS is designed to find data from many datasets.
2. NDS trains expert for each source. SNDS trains an expert for each intermediary public dataset.
3. NDS uses performance of experts the target dataset as proxy for similarity sources. SNDS measures performance of (public) experts on sources and targets, and then uses similarity in perfor- mances as proxy for data similarity.
Querying with a target dataset of size n on NDS incurs a bandwidth cost x M and computational cost M X x C. Querying with target dataset of size SNDS incurs bandwidth cost K and computational cost x K n x C. Here C' is the cost of evaluating an expert on a data point.
3.6 Theoretical Analysis
The benefits of SNDS are achieved by disentangling the training of experts from sources: the computational and privacy drawbacks are due to the 1-to-1 correspondence between sources and experts. This "transfer by proxy" approach works because intuitively, if tasks A and B are both "similar" to C, then A and are also "similar" to each other. Indeed, theory in domain adaptation For example, Mansour et al. [28) have provided generalization
supports this intuition 8 128 4 bound on the target domain.
Theorem 3.1. Mansour et al. (28): Let S and T be the source and target domains over X X Y, H. be a hypothesis class, and let 1: x + IR+ be symmetric loss cossfunction function that obeys the triangle inequality. Further, let his argmingen (h) and ht argmin (b) denote the optimal hypothesis for their respective domains, we have
Vh E H. R) s (2,(h, hz) + disci Tx) + (l + (hg,,
where (k,)),)) marginalized over X.
Ex-s [((()), ((((x), [x))) Sx and Tx are source and target distributions
We make the simplifying assumption that the source task is the same as the target task, since it is not computationally feasible for the server to identify the task of every source and target. By the same vein, we assume that the risk on any S is similar. Thus the main quantity we look to optimize is discy(Sx) Tx). Which is defined as:
disc;(S,T)
sup (K
R' h) R4 (h)))))))))))
An important property of disc; is that it obeys triangle inequality independently of the choice of l. That is: disc T) < dSsci(S, P) + (P. T) for some intermediate domain P over the same subspace X. A simple observation then that we minimize the upper bound, we are indeed minimizing the disc(S, T). This observation fuels the motivation for our approach.
5

---

4 Experiments
We first empirically verify that our similarity function output is well correlated with domain confusion, validating our choices of the proxy task and similarity function in the data selection process. We then experimentally evaluate SNDS in the task of classification. We use accuracy of classifier trained through transfer learning on target datasets to gauge the usefulness of data recommended by SNDS. We also extend the experiments into domains dissimilar to ImageNet, to test the generality of SNDS.
4.1 Datasets
Source data We simulate data providers with partitions of the OpenImages dataset (2)). The Open- Images dataset contains 9 million images with various modes of annotations. We split Openlmages into 50 partitions using K K-means on image features extracted by an ImageNet pretrained network. Each image in Openlmages is tagged with multiple object bounding boxes with labels. We convert OpenImages into classification dataset by cropping the images by the bounding boxes and assigning the label of that bounding box as label of the cropped image. Thus, we obtain a classification dataset of 15.9 million cropped images, each with label from the 600 object bounding box categories.
Public data and expert training We use the training split of ILSVRC2012 containing 1.2 million images, as the public dataset. We split it using superclass partition described in [45]: the mean features of each class are extracted using an ImageNet pretrained network, and then clustered using K Memmns.. Experts are trained on the task of image rotation prediction. Experts use R as backbone with a input size of 224x224 and output dimension of 4 (corresponding to the 4 rotations). We use K 50 in our experiments, and study the effect of reducing K in an ablation. Further details on transfer learning are found in the appendix. Experiments are implemented using Py PyTorch|
Target Datasets We use nine finegrained classification datasets as target datasets. They are: FGVC- Aircraft 27]. Stanford Cars [25), CUB200 43). Stanford Dogs [24]. DTD 13, Flowers [1]
Food100 9 Oxford Pets (33, and SUN397 44
4.2 Domain confusion and SNDS
Arcwn
trc.
time
Since we want SNDS to select sources that are similar to the target domain, the scores of each source returned by SNDS should intu- itively be correlated with disc; (S; T) for source S; and target T. Domain confusion 88 21 has been used an empirical approx- imation of disci (A A, C). In this experiment, we compare domain confusion and score provided by SNDS for target datasets.
-2.3
600
Figure 3: Relationship between the domain confusion and similarity met- ric of the source and target domains.
We use an MLP with 1 hidden layer of 512 neurons to classify between each pair of source and target. We use weighted sample of 50k images from the mixed source/target dataset to ensure that the samples are balanced, as done in NDS. We report the test loss a held-out set of images from the source and target domains.
As seen in Figure the score returned by SNDS is positively correlated with the domain confusion between the source and target domains. This shows that SNDS indeed outputs higher scores for sources similar to the target domain. This can also be observed qualitatively in Figure
4.3 Data Recommendation with SNDS
Implementing Data Consumers Evaluation of SNDS on downstream tasks requires us to experi- mentally simulate data consumers. For target domain data, use the finegrained classification tasks described earlier- each classification task represents one data consumer. Once a recommendation of sources has been made by the server, it is up to the data consumer to decide their budget, data selection strategy, and transfer learning method. Similar to NDS, our simulated data consumer sample from all available data in S without replacement, weighing the likelihood of each data point w. We pre-train on the selected data using supervised learning, and then | finetune on the downstream dataset. Details about our transfer learning procedure are in the appendix. Experiments performed on a Tesla P100 with 12 GB memory in an internal cluster.
as π (x) M 1(x S.

---

Table 1: Downstream task performance with pretraining data recommended by SNDS.
Method
% Images
CUR200
Plowers102
101
Stanford Dogs
SUNN99
Avemge
No pretraining
0%
27.25
52.42
42.21
71.75
39.35
30.16
18.78
45,80
31.80
39.95
Random Sample
(292K)
67.02
52.93
SNDS-50
2% (292K)
50.59
$1.66
71.06
73.73
$5.07
56.28
17.37
55,91
39.79
59.07
NDS
2% (292K)
49.91
78.32
70.71
73.35
$4.64
55.32
52.95
57.95
42.27
59.53
Random Sample
5% (730K)
53.11
83.53
73.79
74.86
57.93
57.87
67.34
62.70
44.42
63.95
SNDS-50
(730K)
75.91
74.87
60.59
43.99
NDS
33% (730K)
59.04
85.10
75.18
75.58
$8.99
60.27
66.56
63.76
45.98
65.61
Random Sample
10% (1-46M)
$8.40
86
76.40
76.22
60.81
60.74
71.58
65.24
47.69
67.10
SNDS-50
NDS
10% (1/16M)
61.27
89.88
78.32
77.03
61.28
60.96
73.77
65.57
46.81
68.33
Client Dataset
Selected images from Openimages dataset
Top 10 Classes Sampled Distribution
CUB200 Birds
Pets
Flowers 102
Figure 4: From left to right: Image from client dataset, images from "most similar" public split (ImageNet). selected images from data sources (Openimages).
4.3.1 Main Experimental Results
In this experiment, we select 2% (292K images), 5% (730K images) and 10% (1 .6M images) of total source data us- ing SNDS and compare with randomly selected data and NDS. We implement NDS with image-rotation experts, as per Sec. Our results demonstrate that data selected by SNDS is more useful in al- most every downstream task than ran- domly selected data. This advantage is most pronounced when using tight budget of 2%, since the chance of ran- domly selecting useful data higher
Table 2:
Downstream performance on Pets CUB200 Birds, using pretraining data recommended by SNDS with no target class in public data
Target Dataset Oxford Pets
Selection Method
% Images
CUB200
Random Sample
2% (292K)
41.49
67.02
SNDS
2% (292K)
50.59
71.06
SNDS (no target data)
2% (292K)
47.67
70.35
Random Sample
5% (730K)
53.11
73.79
SNDS
5% (730K)
57.35
75.91
SNDS (no target data)
5% (730K)
58.69
76.83
when more data is being selected.
Furthermore, we find that SNDS is competitive with NDS even though NDS trains experts directly on source data, whereas SNDS trains experts on the public dataset. As such, SNDS promises cheaper computation cost with growing number of sources without sacrificing much in performance.
Fig. 4 shows retrieval results from SNDS for several target datasets. Similarity between the target and the public data split is determined by the accuracy of that split's expert on the target dataset. Since OpenImages provides multiple class labels per image, the top classes in the sampled distribution may be ones that co-occur with more relevant classes, such as the Plant class for the Pets dataset.
We ablate the use of 50 experts in the appendix, and find that with just 10 experts, SNDS is able to out-perform the baseline, though having more experts allows for more specific recommendations that improve the downstream performance.
7

---

Computational Cost Per Query with Growing Server
Total Computational Cost with Sources and Queries
SNDS
NDS
Ngiarn al. Chokraborty et al.
SNDS
Ngiam at al. Chairaborty at al.
300
800k
Pe-Quarry Compussion Time
500k
200
400k
Towe 20mmmaaaaoo
800k
100
2004
100%
25
50
75
100
125
150
175
250
500
750
1000
1250
1600
1750
Number of Sources
Number Sources + Number Guertes
Figure 5: Comparison of simulated computational cost.
Impact of missing public data critical factor determining the feasibility of SNDS whether generalize to types of images that are not represented in the public dataset. That is, it should be able to recommend source images similar those in the target even if no such images are in the public dataset. We emulate this by dropping images in the public dataset with classes corresponding to the target task. Two target tasks: Oxford Pets and CUB200, are used. After we split the public dataset into 50 superclass clusters, we remove clusters that contain overlap with target domain classes. For the Oxford Pets experiment, we removed eight clusters that included images containing dogs and cats, and for the CUB200 experiment, remaining clusters are used to train experts and in turn recommend data. From Table we can see that dropping experts trained on classes of images in the target set did not have significant effect on performance. Hence, SNDS can be used in the real world without frequent updates to the public dataset once it is deployed. As a side note, source dataset, similar to that used in NDS, for SNDS results reported in this experiment.
removed nine clusters that contained images of birds. The
sampling without replacement over the entire
Simulating Computational Cost We simulate the computational cost of SNDS and compare it to that of Yan al. [5), Ngiam et al. [30]. and Cui et al. (15) in Fig. 5 We assume that the number of sources and the number of queries both grow over time, while the number of images per source stays constant. Detailed modeling assumptions can be found in the supplementary material. For the data consumer, SNDS has a flat per-query computational cost that does not grow with additional sources. Over time lifetime of server, SNDS's total computational cost (incurred by all parties) also the lowest among the methods compared.
4.3.2 Scaling beyond the Natural Image Setting
The computational benefits of SNDS stem from using a fixed set of experts for performing data recommendation. For the system to scale, these experts must be able to provide clients with rec- ommendations in domains beyond the ones that they has been trained on. To test this, we explore the performance of the server, both qualitatively and quantitatively, when recommending data for domains that are highly dissimilar to natural images and find that SNDS is able to provide useful recommendations using experts that have been trained ImageNet. Note that in these experiments the public experts are kept the same, i.e. experts trained on ImageNet.
Medical data Here, we focus on whether SNDS is able to recommend meaningful medical data with experts trained only on ImageNet. We add the chest x-ray dataset CheXpert [23] and the diabetic retinopathy dataset provided by Kaggle 22 [4] to the original source datasets consisting of OpenImages clusters. We examine the recommendations that the server provides when using different targets datasets of the same modalities as the source datasets: the NIH ChestX-ray 14 dataset [42), and the Aptos 2019 diabetic retinopathy dataset M. We do not perform any additional pre-processing to these datasets when incorporating them within SNDS.
In Fig. 66] we show the softmax scores of the top 5 recommended clusters for both the NIH ChestX- ray 14 and Aptos2019 datasets. both cases, the most relevant cluster in the source data corresponds to the image type in target data. Furthermore, SNDS does not conflate the recommendations between the chest x-ray and diabetic retinopathy, suggesting that the server not grouping all dissimilar data together and maintains its ability to provide valuable recommendations in highly dissimilar data.
8

---

Selected Frowiders (Openimages Chelpert
KI
Samping probati Hyy private detusets Gliere
MNl Cllen Data
Recinopaa yy
#15
Opentmages #27
Top c155: Humao 10c2
Dpan maces #10
Oparimages #20 class
Oparrmages Woman
Top dess: Bottle
maceimaces #36 Top case Goehing
Openimages #30
Top dass Food
Figure 6: Recommended data and output probabilities for satellite and x-ray data
In practice, the best method for pre-training with medical images remains an open question and may not provide consistent benefits [6)), though recent works in self super-ueeeeieed learning have shown improvements when using a multi-step training process [6] In our experiments, a naive supervised learning approach did not provide significant transfer benefits, even when directly transferring from one x-ray dataset to another (the oracle recommendation scenario). Still, the relevant recommenda- tions provided by server directly usable by the consumer for improving performance using the latest techniques in this area as they emerge.
Sketch and satellite data evalua- tion Due to the challenges asso- ciated with transfer learning the medical setting, we evaluate the ben- efits of transfer on more balanced datasets consisting of satellite and sketch datasets. We add the Quick-
Table 3: Task Performance for Satellite & Sketch Data
Target Dataset NWPU RESISC45 -ini
Selection Method
% Images
Sketch-Mint
Rnd. Sample (+ sketch/sat)
00.5% (73K)
11.79
52.40
SNDS (Openlmages only)
0..5% (73K)
13.60
53.60
(+ sketch/ktt data) 0.5% (73K)
22.50
Draw dataset split from (35) and the PatternNet dataset 48 to the source sets of our server. As the
client, we evaluate the performance of our method when using 1k subsets of the Sketch [18] and NWPU-RESISC45 datasets [12,, which denote Mini versions of those datasets. We use these subsets rather than the full datasets to better observe the effects of pretraining, as was done NDS. To perform pretraining with the recommended data, mix the various datasets together and perform standard supervised training with cross-entropy loss, predicting the class of the image from the union of all the dataset classes.
Table 3 provides performance for the satellite and sketch datasets. both domains, SNDS shows performance benefits over random sample of data from the server. SNDS even shows benefits over a random sample without satellite and sketch dataset in the server, which suggests that our server is able to find natural images that help with downstream performance, experts used for recommendation have been trained exclusively on ImageNet clusters, demonstrating that SNDS is truly able to scale and improve task performance in highly dissimilar domains.
in dissimilar domains. The
see transfer benefits using our simple pre-training approach, which leads us believe that consumers will be able to benefit even more from the data recommendations when using more complex approaches from multi-task learning. This highlights the strength and robustness of the SNDS' recommendations, which provides immediate benefits with a simple transfer algorithm.
Limitations While SNDS can be applied to many task types, our experiments have focused on fine fine-grained classification tasks, allowing to illustrate its behaviours across different data domains

---

and in several important ablations. We also omit transfer experiments in the medical domain, as mentioned in Sec. 4.3.2
Related Work
Our work is built upon the rich literature of transfer learning and domain adaptation 49 32
g
4 28 Of most resemblance to ours are the "instance weighing" approaches [39 40
7
16
15
30 3 [0]. These methods compute the weight of instances or sets of instances based
some measure of usefulness to the target task. Our work differs from these methods in data separation and computational constraints: our problem setting requires method that does can operate with the source and target dataset on separate devices, and the computational cost for indexing new source data must be low.
Cui et al. [5] uses similarity between mean features of each class as extracted by a NN trained on source data to match classes of the source to target dataset. Ngiam et al. [30] instead labels the target dataset with a NN trained on source data, and recommends source classes that appear most frequently as pseudo-labels. These methods requires retraining of the NN whenever the source data is updated, thus they are computationally expensive in terms total cost.
Achille et al. 3 find the Fisher information matrix for each source and target dataset to compute a similarity measure between sources and targets. This approach requires access to the consumer's labels and is challenging to scale beyond classification tasks, and thus not immediately applicable to our setting where we focus on being label-agnostic to allow scalability to arbitrary tasks.
Chakraborty et al. [10 proposed to train NN on the target dataset, and compute similarity between the target dataset and each source in the feature space of this NN. Finally, Yan al. 445 proposed NDS, reviewed in Sec. 2 These two methods have a per-query computational cost that grows with the number of sources. Furthermore, all existing methods require training of NNs on either sources or targets. This violates our privacy constraints (DI and D2 in desiderata). SNDS protects the privacy of source and target data, and has a lower computational cost than existing methods.
Conclusion
We proposed SNDS, scalable data recommendation method that selects sets of useful pretraining data for each given downstream task. In SNDS, sources and targets are represented by their degree of similarity to public datasets. The biggest advantage of SNDS over existing methods is computational efficiency: the computational cost for data consumers remains constant with the number of sources indexed by the server, and sources already indexed by SNDS do not need to be re-indexed when adding new sources. Experiments show that SNDS can work well even if there is data in the public datasets that is dissimilar to target, thereby allowing SNDS to generalize beyond what is contained in the public datasets. Lastly, we demonstrated the robustness of SNDS by applying it to a variety of domains dissimilar to natural images and show transfer benefits in that setting.
SNDS is a large-scale search engine for transfer learning data. Users can use SNDS to find relevant pretraining data for their dataset, which allows practitioners with smaller datasets to leverage the latest models designed for larger datasets, and to select pretraining data according to their computational needs. This is especially beneficial for consumers for whom collecting large dataset is cost-prohibitive, such as in hospitals or non-profit organizations. As data recommendation search engine, our system is susceptible to misuses. In a completely open system, data providers may upload inappropriate datasets or biased data to intentionally unintentionally cause harm to the consumer. This can be mitigated by having a dataset approval process for SNDS and by including the latest tools explore dataset biases part of the server.
Potential Societal Impact
Acknowledgment This work was partially supported by LG and NSERC. SF acknowledges the Canada CIFAR AI Chair award the Vector Institute. TC acknowledges support from Vector Scholarship for Artificial Intelligence. DA acknowledges support from the Microsoft Ada Lovelace Fellowship.
10

---

References
[1] Aptos 2019 diabetic retinopathy dataset.
URL htt:://wwww.. http://wwwwggc..oooo //
[2] Diabetic retinopathy detection, 2015.
URL Sttps://www.kagee
[3] A. Achille, M. Lam, R. Tewari, A. Ravichandran, S. Maji, C. Fowlkes, S. Soatto, and Perona. Task2vec: Task embedding for meta-learning, 2019.
[4] D. Acuna, G. Zhang, M. Law, and S. Fidler. ddlominn-adversaae learning: Theory and
algorithms. In Proceedings of the 38th International Conference on Machine Learning, pages 66-75. PMLR, 2021.
[5]
AlMultiple Data marketplaces: What, why. how, types, benefits, vendors. 2020. URL https://research.. aimuttippeeim hts//ddaaa-aaaaa con./aaaaammarkctplaee// Adatamarketplacese hwg.aaaaeeepp.aaaaaaaaaaaaaattta
[6] S. Azizi, B. Mustafa, F. Ryan, Z. Beaver, J. Freyberg, J. Deaton, A. Loh, A. Karthikesalingam,
S. Kornblith, T. Chen, Natarajan, and M. Norouzi. Big self-supervised models advance medical image classification, 2021.
[7] M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research, 7(Nov): 2399-2434, 2006.
[8] S. Ben-David, J. Blitzer, K. Crammer, and Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pages 137-144, 2007.
[9] I.. Bossard, M. Guillaumin, and L.. Van Gool. Food-101-mining discriminative components with random forests. In European conference on computer vision, pages 446-461. Springer, 2014.
[10] S. Chakraborty, B. Uzkent, K. Ayush, K. Tanmay, E. Sheehan, and S. Ermon. Efficient conditional pre-training for transfer learning. arXiv preprint arXiv:2011.1023/ 2020.
[11] S. Chen, R. Jia, and G.-J. Qi. Improved techniques for model inversion attack. arXiv preprint WxXi:2200004099 2020.
[12] G. Cheng. J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105 165(10):1865-1883,, Oct 2017. ISSN 2556. doi: 10.1 20.1009/pooc.2217726566
/ doi www.oo.go///./... JPROC. 2017 2675999
[13] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2014.
[14] Cuadros and G. Bresnick. Eyepacs: an adaptable telemedicine system for diabetic retinopathy screening. Journal of diabetes science and technology. 3(3):509 -160999-56666 2009.
[15] Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie. Large scale fine- fine-graine grained categorization and domain- specific transfer learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4109-4118, 2018
[16] W. Dai, Q Yang, G.-R. Xue, and Y. Yu. Boosting for transfer learning. In Proceedings of the 24th international conference on Machine learning. pages 193 200, 2007.
[17] Deng, W. Dong, Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[18] M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? ACM Trans. Graph. (Proc. SIGGRAPH), 31(4):44: 10, 2012.
[19] M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communicattonn Security, pages 1322- 1333, 2015.
[20] Fysical. The next data frontier isn't digital. it's fysical, 2020. URL BRLLtttss///yyecaa..oo
[21] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and Lempitsky. avversiadv training of neural networks. The journal of machine learning research, 17(1):2096-203) 2016.

---

[22] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning, volume 1. MIT Press, 2016.
[23] Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-licus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones, D. B. Larson, C. P. Langlotz, B. N. Patel, M. P. Lungren, and A. Y. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, 2019.
[24] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
[25] Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[26] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Tuse-,,set, S. Kamali, S. Poppo.. M. Malloci, A. Kolesnikov, T. Duerig, and V. Ferrari. The open images dataset v4; Unified image classification, object detection, and visual relationship detection at scale. IJC,, 2020.
[27] S. Maji, Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[28] Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Proceedings of The 22nd Annual Conference on Learning Theory (COLT 2009), Montréal, Canada, 2009. URL hhttp://wwr cs .n... www.:n...uuuuuuuunmmm.uu. edu/ edtn//mobrr//pottscrrp. nadap.pdl pdf
[29] B. Neyshabur, H. Sedghi, and C. Zhang. What is being transferred in transfer learning?, 2021.
[30] J. Ngiam, D. Peng, V. Vasudevan, S. Korbllih, Q. V. Le, and R. Pang. Domain adaptive transfer learning with specialist models. arXiv preprint 1811. 1.07556, 2018.
[31] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
[32] S. J. Pan and Q. Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 1345-13345-135, 2009.
[33] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
[34] A. Paszke, S. Gross, F. Massa, A. Lever, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. Vito,, M. Raison, A. Tejani, S. Chilamkurthy,, B. Steiner, L. Fang, Bai, and S. Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and Gamett, editors, Advances in Neural Information Processing Systems 32, pages 802480044--00 8035. Curran Associates, Inc., 2019. http: neur www.nninrii..o cc/paper/ style---ttthhhhhh hhgggppprrrrrmmo-----e---ee-oaaa -ibbrrr
9015-pytorch
pdf
[35] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEFE International Conference on Computer Vision, pages 1415, 2019.
[36] M. Raghu, C. Zhang, J. Kleinberg, and S. Bengio. Transfusion: Understanding transfer learning for medical imaging. arXiv preprint arXiv: 1902.07208, 2019.
[37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. doi:
1410077/11223-015---21
[38] Snowflake. Snowflake data marketplace, 2020. data marketplace/
URL https://www.snowft
com/
[39] M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4):699-746, 2008.
12

---

[40] Q. Sun, R. Chattopadhyay, S. Panchanathan, and J. Ye. A two-stage weighting framework for multi-source source domain adaptation. Advances neural information processing systems, 24: 505-513, 2011.
[41] VisualData. Visualdata search engine for computer vision datasets, 2020. URL https
www..iisallata. ioddvvcvery
[42] X. Wang. Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Ohestx-ray& Hospital-scale chest x-ray database and benchmarks on seakly-superrised classification and localization of common thorax discases. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR). Jul 2017. doi: 10. 10110000
URL http::////> doi www.:ii........... 1019//C/I.1///VP
2017.369
[43] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSS) Birds 200. Technical Report
California Institute of Technology. 2010.
[44] Xiao, Hays, recognition from abbey to Zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492 IEEE, 2010.
Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene
[45] X. Yan, D. Acuna, and S. Fidler. Neural data server: A large-scale search engine for transfer learning data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3893-3902, 2020.
[46] A. R. Zamir, A. Sax, W. B. Shen, L. J. Guibas, J. Malik, and S. Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018.
[47] Y. Zhang, R. Jia, H. Pei, W. Wang, B. Li, and D. Song. The secret revealer: Generative model- inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 253-261, 2020.
[48] W. Zhou, S. Newsam, C. Li, and Z. Shao. Patternnet: A benchmark dataset for performance evaluation of remote sensing image retrieval. ISPRS Journal of Photogrammetrr and Remole URL
Sensing, 14: Nov 2018. ISSN 0924-2716. doi:
/ doi orgoorg///// 1016/ isprsjprs. 2012 01.0044
2018.002800
[49] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu. H. Zhu, H. Xiong, and Q. He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We describe main claims in the in the introduction (Section[] I in terms of the generalization and performance of the system, both in natural image and dissimilar image domains, which we demonstrate experimentally in Sections 4.3. and 4.3.2. We also describe the scalability of the system in detail in terms of computational cost within simulate the computational growth Fig.
(b) Did you describe the limitations of your work? [Yes] See Section 4.3.2
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] In addition to reading the guidelines, we discuss the broader implications of our work in Section aas mentioned in the previous question.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [No] This is proprietary information that we will not be releasing at this time.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We will be addressing these in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Due to the large-scale experiments required for transfer learning, we were unable get these results for the paper, as this requires pre-training with a large set of data for variety of datasets and finetuning on downstream tasks. This would require at least 3 times the current computational resources, making this computationally infeasible for the current submission.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See section 4.3 for the type of resource. Given the amount of experiments performed during the process (with different percentages of data across a wide variety of datasets), we were unable to properly quantify the total amount of resources used.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We mention the datasets in Section 4.1 and within the relevant parts of 4.3.2
(b) Did you mention the license of the assets? [Yes] This is included in the supplementary
material.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We did not curate any data ourselves, but used pre-existing datasets curated by others. Details are included in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] This is included the supplementary material.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rZykisK-4Ha", "reviewer": "Reviewer_2egD", "summary": "This paper presents an idea of a data recommender system called Scalable Neural Data Server (SNDS) to support transfer learning for target customers. The interesting point is that SNDS does not store any source or target dataset on the server, but only maintains the benchmark intermediary datasets, and the pools of experts that are trained on the intermediary datasets. SNDS shares the pool of experts to both data providers and consumers to compute similarities between the consumer's data and the data provider's datasets. Image rotation prediction is used to measure the expert model's performance on the source and target datasets. In this way, SNDS indirectly measures the data similarity between the sources and targets. Compared to its predecessor NDS, SNDS trains experts for an intermediary public dataset, instead of training an expert for each source. Using this method, SNDS achieves scalability as well as securing datasets from both sides. Extensive experiments are done to gauge the usefulness of data recommended by SNDS. With SNDS-recommended source, classification performance on target datasets improved significantly. To support this, the authors show that domain confusion exists between target datasets and recommended sources. The performance gain using SNDS is comparable to that of NDS, even though SNDS trains experts on the public datasets and promises cheaper computation costs. Besides, SNDS generalize to types of images that are not represented in the public dataset. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Recommending datasets without having it seems like a great innovation to me who came across the Neural Data Server concept through this paper. But, reviewers who already knew of NDS could look at this as incremental work. Though there is a huge concept change between the two works and it is worth noticing. The major change is not to provide experts trained from source datasets to clients. By using expert models trained from intermediary datasets, SNDS can guarantee extra anonymity between the customers and data providers. For me, it is not very clear why this solution scales cheaply with the number of sources. In Appendix D, since the first term (K x n x iter time) dominates the second term (K x M x match time), does SNDS is more efficient than [44] if M (# of data providers) >> K (public splits)? \n\nJust wanted to clarify these again:\n- Does the number of split K mean how many clusters the public dataset is divided into through k-means?\n- Does the expert model is a model to classify the rotation amount of each partition?\n\nThen, here is my question. Appendix F shows the performance change by reducing the number of splits. For some target datasets, the performance decreases. But overall, it increases on average. By connecting this observation to the computation cost difference between NDS and SNDS (M and  K), can you say it is okay to split the public dataset up to the number of source datasets (considering the match time is negligible)? Also, Have you tried a theoretical approach to prove a certain number of splits (K) is needed to approximate the source and target similarity when the number of source datasets is M?\n\nAdditional questions and typos:\n- Should you denote \\sigma_{i=1}^K ? instead of \\sigma_{i}^K ?\n- In Table 1, NDS is mentioned as an oracle probably because it provides experts trained on source data. But, why SNDS sometimes perform better than NDS? Would like to see standard deviation together with the averaged results.\n- In Table 2, why SNDS (no target dataset) somehow outperforms SNDS in the 5% scenario?\n- Figure 6: In the caption, Satellite data?\n- Page 9: does can \n- Page 13: Are you planning to open-source SNDS, or open it in commercial form?\n\nOverall, I enjoyed reading this manuscript. The paper is well-written and the suggested framework is backed up with extensive experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impacts are mentioned in the manuscript. As mentioned in the paper, providing a data approval process to block uploading inappropriate datasets can be needed to mitigate the potential negative impact of the data-sharing platform.\n\nSNDS can be further improved for some domains such as medical data. From Figure 6, I felt SNDS recommends relevant images on source datasets. Was the problem that the performance didn't increase?  \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {"ethical_concerns": "There are no ethical concerns. The reviewer can guess that this work is done by the authors of the neural data server (CVPR 2020) due to the similar style of Figure 1 and some figures in the experiments section. This evidence alone does not break anonymity, so it is good to go."}}, {"review_id": "CpmbitnCYyA", "reviewer": "Reviewer_eRqo", "summary": "This paper extends Neural Data Server (NDS), which recommends relevant data points for the target task, to be more scalable, by employing a fleet of a mixture of experts. Each expert is trained on data slices, which is based on predefined assumptions, and generates representations that will be used to select relevant data slices for a given task.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, this paper fits more appropriate to distributed systems or data mining conferences than a ML conference. \n\n**Originality** This paper extends a prior work, NDS, by training experts for the subsets of datasets rather than training experts per each data source entirely. Each data source is partitioned by a predefined metric. When the actual task is given, the consumer pipeline will select the subsets by computing cosine similarities using the output of each expert from each slice. Hence, the whole system has some innovation in constructing the distributed system; however, I do not see a clear novelty for the machine learning community.\n\n**Quality** Few questions arose after reading this paper. First, each expert needs to be trained with prior knowledge (or hypothesis) for the upcoming task. Combined with the simple, not adaptive similarity metric to select the subsets, the system seems to be limited for only a few tasks that are very similar to the original task. The generalizability of the proposed algorithm is quite limited in that sense. Second, similarly, I am a bit confused about the claim at Line 200. Assuming the source task is the same as the target task is a bit too strong an assumption. What’s the point of having a large system if we can only subset the dataset based on the known assumptions. Can we just train multiple experts over the whole dataset and ensemble them if we already have knowledge of the target task? I may have missed something, but I am not fully convinced of this setup. Third, the gap between the random sample and the proposed method seems to be a bit too thin, and it is essentially the only baseline. Authors should have considered some other baselines such as simple cosine similarity based subset selection (since they are using cosine similarities to the output of experts).\n\n**Clarity** The paper was mostly easy to follow and clear.\n\n**Significance** As mentioned earlier, the machine learning contribution of this paper is not substantial.\n\nOther comment:\nThere are multiple other selection algorithms ([1, 2] for example -- just did the Scholar search) that can find the most relevant data points. Authors may want to add discussion to the work.\n\n[1] Bengio, Yoshua, et al. \"Curriculum learning.\" Proceedings of the 26th annual international conference on machine learning. 2009.\n[2] Bruno, Andreis, et al. \"Stochastic Subset Selection for Efficient Training and Inference of Neural Networks.\" (2020).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No suggestions. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "8U9aLvnJb2n", "reviewer": "Reviewer_yKh5", "summary": "The paper presents a scalable approach to recommend datasets from data providers to consumers looking to assemble datasets to pre-train models for downstream tasks. The approach does not leak source data from data providers and target data from consumers, and instead uses an intermediary public dataset and expert models trained on subsets of this public dataset to evaluate the data in sources that are likely to benefit the target tasks. The approach is scalable (independent of the number of source datasets), does not transfer source or target datasets to a server, transfers only expert models, and evaluates these expert models only on the source datasets and target datasets in the provider and consumer servers alone to keep computation costs low. The experiment results demonstrate the efficacy of this method, and show (somewhat surprisingly) that it is able to identify relevant source datasets even when the intermediate public dataset does not contain data relevant to the target downstream tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper extends an earlier method - neural data server to recommend datasets for pre-training models for downstream tasks, making it more scalable and secure for consumers. It relies on the fact that self-supervised training of experts on intermediate public datasets on a generic task (predicting rotation of images) captures some representations of images, and that these experts can find datasets with similar representations in source (from data providers) and targets (consumers who want to assemble pre-training data). While the idea is relatively common, the application to scale up the neural data server is novel and appears to be original. \n\nQuality: The paper motivates the problem well, explains the solution approach, demonstrates the results using reasonable baselines and experiments, and addresses the limitations. The tests are rigorous (within the computational limits the authors mention) and results are convincing. \n\nClarity: I think the paper can use one round of clarifying edits. For example:\n1)  it took a while to even realize that the authors focus on vision tasks. \n2) sec 3.2: claim that image rotation prediction is correlated with domain confusion can use some references of related papers\n3)  sec 3.3:  what if the source dataset happens to be rotated versions of the public dataset? \n4) sec 4.3.1: I guess the section refers to Table 1, but nowhere is this mentioned\n5) sec 4.3.1: In Table 2, Why do some metrics go up for SNDS compared to NDS at 5% sampling? What metrics are shows in Table 2?\n6) Fig 5. left panel. Where is the result for Ngiam et al (green line)?\n7) Table 3 - is the difference in metrics for NWPU-RESISC45 dataset significant? \n\nSignificance: With the growing availability of source datasets from providers who are however not willing to share the datasets publicly, and the increased use of pretrained models on large datasets for downstream tasks through transfer learning (or fine tuning), this approach offers a scalable way to identify specific source dataset relevant to a target downstream task, thus establishing a marketplace that helps both providers and consumers understand the value of a dataset without having to share it a-priori. This is a significant problem, the solution of which can open up broader applicability of machine learning. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper proposes the scalable data recommendation method for the task of image classification. From the paper, it is not clear how well this approach generalizes to other domains (like NLP, time series, speech/audio etc.) and other task (like object detection, segmentation etc.). I suspect it can generalize to any domain/task that uses pre-trained models, but this is neither claimed or dismissed in the paper.  \n\nAs the authors say, the approach can benefit society by making it easier for consumers to identify source datasets likely to benefit their downstream tasks, without having to exchange data a-priori with the provider. \n\n", "review_text": "", "extra_scores": {"ethical_concerns": "As the authors say, malicious actors can manipulate the source datasets. This risk can be mitigated by a review system. \n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Quite a borderline paper with a high variance of scores. I'm recommending acceptance though the average score is probably below what would correspond to an accepted paper at NeurIPS. Mostly I'm siding with the positive reviewers because:
-- The negative score/review is somewhat of an outlier compared to the other two.
-- The two positive reviewers voiced their support of acceptance in the discussion, whereas the negative reviewer did not weigh in. This caused me to downweight their assessment a little.
-- The rebuttal seems reasonable, and was at least persuasive to the most positive reviewer. Again, the negative reviewer was rebutted though did not respond (which I take as somewhat implicit support).

Ultimately a tough call. The positive reviewers found the problem important, the solution effective, and praised the introduction of new datasets for the task. The negative reviewer criticized the novelty of the work, though again this was not a consensus view. Most reviewers raised issues of clarity, though these can likely be addressed in a revision (or have already been addressed in the rebuttal).

Still may be borderline given that the paper doesn't quite have a "champion" and likely will have lower scores than other accepted papers. But there is reasonable endorsement by the reviewers (especially weighted by participation in the discussion), and the criticisms seem like not major red flags.
Decision: Accept (Poster)