Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper extends Neural Data Server (NDS), which recommends relevant data points for the target task, to be more scalable, by employing a fleet of a mixture of experts. Each expert is trained on data slices, which is based on predefined assumptions, and generates representations that will be used to select relevant data slices for a given task.
main_review: First of all, this paper fits more appropriate to distributed systems or data mining conferences than a ML conference. 

**Originality** This paper extends a prior work, NDS, by training experts for the subsets of datasets rather than training experts per each data source entirely. Each data source is partitioned by a predefined metric. When the actual task is given, the consumer pipeline will select the subsets by computing cosine similarities using the output of each expert from each slice. Hence, the whole system has some innovation in constructing the distributed system; however, I do not see a clear novelty for the machine learning community.

**Quality** Few questions arose after reading this paper. First, each expert needs to be trained with prior knowledge (or hypothesis) for the upcoming task. Combined with the simple, not adaptive similarity metric to select the subsets, the system seems to be limited for only a few tasks that are very similar to the original task. The generalizability of the proposed algorithm is quite limited in that sense. Second, similarly, I am a bit confused about the claim at Line 200. Assuming the source task is the same as the target task is a bit too strong an assumption. What’s the point of having a large system if we can only subset the dataset based on the known assumptions. Can we just train multiple experts over the whole dataset and ensemble them if we already have knowledge of the target task? I may have missed something, but I am not fully convinced of this setup. Third, the gap between the random sample and the proposed method seems to be a bit too thin, and it is essentially the only baseline. Authors should have considered some other baselines such as simple cosine similarity based subset selection (since they are using cosine similarities to the output of experts).

**Clarity** The paper was mostly easy to follow and clear.

**Significance** As mentioned earlier, the machine learning contribution of this paper is not substantial.

Other comment:
There are multiple other selection algorithms ([1, 2] for example -- just did the Scholar search) that can find the most relevant data points. Authors may want to add discussion to the work.

[1] Bengio, Yoshua, et al. "Curriculum learning." Proceedings of the 26th annual international conference on machine learning. 2009.
[2] Bruno, Andreis, et al. "Stochastic Subset Selection for Efficient Training and Inference of Neural Networks." (2020).

limitations_and_societal_impact: No suggestions.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a scalable approach to recommend datasets from data providers to consumers looking to assemble datasets to pre-train models for downstream tasks. The approach does not leak source data from data providers and target data from consumers, and instead uses an intermediary public dataset and expert models trained on subsets of this public dataset to evaluate the data in sources that are likely to benefit the target tasks. The approach is scalable (independent of the number of source datasets), does not transfer source or target datasets to a server, transfers only expert models, and evaluates these expert models only on the source datasets and target datasets in the provider and consumer servers alone to keep computation costs low. The experiment results demonstrate the efficacy of this method, and show (somewhat surprisingly) that it is able to identify relevant source datasets even when the intermediate public dataset does not contain data relevant to the target downstream tasks.
main_review: Originality: The paper extends an earlier method - neural data server to recommend datasets for pre-training models for downstream tasks, making it more scalable and secure for consumers. It relies on the fact that self-supervised training of experts on intermediate public datasets on a generic task (predicting rotation of images) captures some representations of images, and that these experts can find datasets with similar representations in source (from data providers) and targets (consumers who want to assemble pre-training data). While the idea is relatively common, the application to scale up the neural data server is novel and appears to be original. 

Quality: The paper motivates the problem well, explains the solution approach, demonstrates the results using reasonable baselines and experiments, and addresses the limitations. The tests are rigorous (within the computational limits the authors mention) and results are convincing. 

Clarity: I think the paper can use one round of clarifying edits. For example:
1)  it took a while to even realize that the authors focus on vision tasks. 
2) sec 3.2: claim that image rotation prediction is correlated with domain confusion can use some references of related papers
3)  sec 3.3:  what if the source dataset happens to be rotated versions of the public dataset? 
4) sec 4.3.1: I guess the section refers to Table 1, but nowhere is this mentioned
5) sec 4.3.1: In Table 2, Why do some metrics go up for SNDS compared to NDS at 5% sampling? What metrics are shows in Table 2?
6) Fig 5. left panel. Where is the result for Ngiam et al (green line)?
7) Table 3 - is the difference in metrics for NWPU-RESISC45 dataset significant? 

Significance: With the growing availability of source datasets from providers who are however not willing to share the datasets publicly, and the increased use of pretrained models on large datasets for downstream tasks through transfer learning (or fine tuning), this approach offers a scalable way to identify specific source dataset relevant to a target downstream task, thus establishing a marketplace that helps both providers and consumers understand the value of a dataset without having to share it a-priori. This is a significant problem, the solution of which can open up broader applicability of machine learning. 

limitations_and_societal_impact: The paper proposes the scalable data recommendation method for the task of image classification. From the paper, it is not clear how well this approach generalizes to other domains (like NLP, time series, speech/audio etc.) and other task (like object detection, segmentation etc.). I suspect it can generalize to any domain/task that uses pre-trained models, but this is neither claimed or dismissed in the paper.  

As the authors say, the approach can benefit society by making it easier for consumers to identify source datasets likely to benefit their downstream tasks, without having to exchange data a-priori with the provider.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an idea of a data recommender system called Scalable Neural Data Server (SNDS) to support transfer learning for target customers. The interesting point is that SNDS does not store any source or target dataset on the server, but only maintains the benchmark intermediary datasets, and the pools of experts that are trained on the intermediary datasets. SNDS shares the pool of experts to both data providers and consumers to compute similarities between the consumer's data and the data provider's datasets. Image rotation prediction is used to measure the expert model's performance on the source and target datasets. In this way, SNDS indirectly measures the data similarity between the sources and targets. Compared to its predecessor NDS, SNDS trains experts for an intermediary public dataset, instead of training an expert for each source. Using this method, SNDS achieves scalability as well as securing datasets from both sides. Extensive experiments are done to gauge the usefulness of data recommended by SNDS. With SNDS-recommended source, classification performance on target datasets improved significantly. To support this, the authors show that domain confusion exists between target datasets and recommended sources. The performance gain using SNDS is comparable to that of NDS, even though SNDS trains experts on the public datasets and promises cheaper computation costs. Besides, SNDS generalize to types of images that are not represented in the public dataset. 
main_review: Recommending datasets without having it seems like a great innovation to me who came across the Neural Data Server concept through this paper. But, reviewers who already knew of NDS could look at this as incremental work. Though there is a huge concept change between the two works and it is worth noticing. The major change is not to provide experts trained from source datasets to clients. By using expert models trained from intermediary datasets, SNDS can guarantee extra anonymity between the customers and data providers. For me, it is not very clear why this solution scales cheaply with the number of sources. In Appendix D, since the first term (K x n x iter time) dominates the second term (K x M x match time), does SNDS is more efficient than [44] if M (# of data providers) >> K (public splits)? 

Just wanted to clarify these again:
- Does the number of split K mean how many clusters the public dataset is divided into through k-means?
- Does the expert model is a model to classify the rotation amount of each partition?

Then, here is my question. Appendix F shows the performance change by reducing the number of splits. For some target datasets, the performance decreases. But overall, it increases on average. By connecting this observation to the computation cost difference between NDS and SNDS (M and  K), can you say it is okay to split the public dataset up to the number of source datasets (considering the match time is negligible)? Also, Have you tried a theoretical approach to prove a certain number of splits (K) is needed to approximate the source and target similarity when the number of source datasets is M?

Additional questions and typos:
- Should you denote \sigma_{i=1}^K ? instead of \sigma_{i}^K ?
- In Table 1, NDS is mentioned as an oracle probably because it provides experts trained on source data. But, why SNDS sometimes perform better than NDS? Would like to see standard deviation together with the averaged results.
- In Table 2, why SNDS (no target dataset) somehow outperforms SNDS in the 5% scenario?
- Figure 6: In the caption, Satellite data?
- Page 9: does can 
- Page 13: Are you planning to open-source SNDS, or open it in commercial form?

Overall, I enjoyed reading this manuscript. The paper is well-written and the suggested framework is backed up with extensive experiments. 
limitations_and_societal_impact: Limitations and societal impacts are mentioned in the manuscript. As mentioned in the paper, providing a data approval process to block uploading inappropriate datasets can be needed to mitigate the potential negative impact of the data-sharing platform.

SNDS can be further improved for some domains such as medical data. From Figure 6, I felt SNDS recommends relevant images on source datasets. Was the problem that the performance didn't increase?
review_title: null

Meta-review: Quite a borderline paper with a high variance of scores. I'm recommending acceptance though the average score is probably below what would correspond to an accepted paper at NeurIPS. Mostly I'm siding with the positive reviewers because:
-- The negative score/review is somewhat of an outlier compared to the other two.
-- The two positive reviewers voiced their support of acceptance in the discussion, whereas the negative reviewer did not weigh in. This caused me to downweight their assessment a little.
-- The rebuttal seems reasonable, and was at least persuasive to the most positive reviewer. Again, the negative reviewer was rebutted though did not respond (which I take as somewhat implicit support).

Ultimately a tough call. The positive reviewers found the problem important, the solution effective, and praised the introduction of new datasets for the task. The negative reviewer criticized the novelty of the work, though again this was not a consensus view. Most reviewers raised issues of clarity, though these can likely be addressed in a revision (or have already been addressed in the rebuttal).

Still may be borderline given that the paper doesn't quite have a "champion" and likely will have lower scores than other accepted papers. But there is reasonable endorsement by the reviewers (especially weighted by participation in the discussion), and the criticisms seem like not major red flags.
Decision: Accept (Poster)