Scalable Neural Data Server: A Data Recommender for Transfer Learning

1 Introduction . In recent years , machine learning ( ML ) applications have taken a foothold in many fields . Methodological and computational advances have shown a trend in performance improvements achieved by using larger and larger training datasets , confirming that data is the fuel of modern machine learning . This fuel , however , is not free of cost . While the raw data itself ( such as images , text , etc . ) can be relatively easy to collect , annotating the data is a labour intensive endeavour . A popular and effective approach to reduce the need of labeling in a target application domain is to leverage existing datasets via transfer learning . Transfer learning is the re-purposing of ML models trained on a source dataset towards a different target task [ 22 ] . The performance of transfer learning is predicated on the relevance of the source to the target [ 46 , 36 ] . Although there are numerous datasets available through various data sharing platforms [ 41 , 5 , 38 , 20 ] , finding the right dataset that will most benefit transfer-learning performance on the target domain is not a simple problem . Consider a startup looking to train a debris detector for use on a road cleaning robot , and suppose that they have collected a small , labeled dataset . They are looking for data with which to augment their training set . Ideally , they would find a dataset of roadside debris that is captured from a sensor similar to their own . However , keyword searches for “ roadside debris ” , would likely find data geared towards autonomous driving , which might be from one of the many sensor types ( rgb , lidar , radar ) . Searching for their specific sensor type might lead to data from similar setups , but of different objects . ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Rather than using keywords , an ideal system would find relevant data based on the training set itself . In this work , we envision a data search engine that indexes source data from data providers , and recommends relevant transfer learning data to consumers who have a target task . Neural DataServer ( NDS ) [ 45 ] has been proposed as a data recommendation system for transfer learning . Given a target task ( as represented by a target dataset ) , it uses expert models trained on splits of source data to make predictions on the target , which is in turn used as a measure of similarity between the source and the target . Directly indexing a large number of datasets with NDS is expensive , as it would require each data provider to upload a trained expert , and any data consumer looking to select data for purchase must download and run all of the trained experts in order to evaluate similarity . Hence , for M data providers in the server , the computational and bandwidth cost to the data consumer scales linearly with M . At the same time , using an expert model per data provider in NDS ensures that the data recommendations are highly relevant to the consumer . A data search engine that operates in a large-scale setting must be able to preserve the quality of recommendations , while disentangling the consumer ’ s computational cost from the number of providers . The system should also preserve the data privacy properties from NDS , where the client ’ s data is never exchanged with the server . We propose Scalable Neural Data Server ( SNDS ) , a search engine for data recommendation at scale . This is achieved by using an intermediary , public dataset to train a fixed number of experts , and a similarity measure that evaluates the relevance of the server ’ s data for a given consumer via these experts . Compared to NDS , the computational cost of SNDS is constant for a given consumer with respect to the M data providers , and so is the bandwidth cost between the consumer and the server . We validate SNDS experimentally and find that 1 ) SNDS generalizes in the sense that it recommends useful data for downstream tasks that are not directly represented by the intermediary datasets , 2 ) SNDS improves performance on downstream tasks over baselines , and is competitive with NDS , 3 ) SNDS is able to generalize to domains that are very dissimilar to the intermediary datasets . 2 Background . In this section , we first formalize our problem setting which is similar to that of NDS , and then provide a brief overview of the data selection method introduced in NDS . 2.1 Problem setting . Let X denote a sample space and Y a label space . Suppose that there are currently M data providers ( which we call “ sources ” ) , each with data available for use in transfer learning . Let the sources be denoted as S = { S1 , . . . , SM } , where Si denote a set of mi sample-label pairs ( ( x1 , y1 ) , . . . , ( xmi , ymi ) ) ∈ ( X × Y ) mi . The target task by the data consumer is represented by a target specific distribution D supported over X × Y , from which the target dataset T = ( ( x1 , y1 ) , . . . , ( xn , yn ) ) ∈ ( X × Y ) n is drawn . The goal of the data consumer is to find data sources suitable for transfer learning to their target task . Specifically , we seek a mixture of sources S∗ that minimizes the following risk for the data consumer : R ( S ) = Ex , y∼D [ l ( hS∪T ( x ) , y ) ] ( 1 ) while satisfying some budget constraint of |S|≤ b , as specified by the data consumer . Here , hS∪T is a predictive model h : X → Y that is trained on S and T . In practice , h is often first trained on S during pretraining , and then trained on T during finetuning . Desiderata 2.1 . To meet the requirements of a data search engine , a potential solution should satisfy : • D1 : Data in S or T are not stored on the server . • D2 : Data in S is not revealed to the consumers before S∗ is selected . • D3 : Data in T is not revealed to any data provider or any other consumers throughout the process . • D4 : The computational and bandwidth cost for the consumer should be fixed as M ( i.e the number of data providers ) grows . What makes this problem challenging is that D1-D3 prevents the sources and the target from being on the same device , and the D4 prevents the server from iterating through every example in S for responding to every data consumer . Our problem setting differs from that of NDS in that we consider the number of sources M as an extrinsic variable , i.e . something outside the control of the server . This necessitates a solution that scales cheaply with the number of sources , as stated in D4 . 2.2 Neural Data Server . NDS partitions the source data through either ( 1 ) superclass partition or ( 2 ) unsupervised partition . In either case , image features are extracted with a pretrained network . The image features are either ( 1 ) averaged over samples of the same class and then clustered with k-means , or ( 2 ) clustered with k-means directly . We refer to each of these clusters as Si ( i ranges from 1 to K ) as these are analogous to the sources considered in our problem formulation . Then , expert networks are trained for rotation prediction on each partition . Specifically , for each image x and a rotation angle θ ∈ { 0 , 90 , 180 , 270 } , let r ( x , θ ) denote the image rotated clockwise by θ . An expert is a function f : X → ∆3 , where ∆n is a probability mass function of n+ 1 choices . The loss minimized by a expert fi is : L = − ∑ x∈Si ∑3 j=0 log fi ( r ( x , j ) ) j . Once an expert is trained , accuracy of image rotation prediction is evaluated on the target set : zi = 1 4|Si| ∑ x∈T 3∑ j=0 1 ( arg max k [ fi ( r ( x , j ) ) k ] == j ) . ( 2 ) The importance weight wi for each partition i is computed as softmax ( z ) i . The sample weight for each example in the source is computed as π ( x ) = ∑K i 1 ( x ∈ Si ) wi |Si| , which is used to sample examples without replacement from the union of source data sets . Exposing the models directly trained on the source datasets in NDS to the client introduces privacy leakage , as the trained experts may memorize certain examples in the source data . It has been shown that is possible to reconstruct training images from trained CNNs by optimizing images from noisy priors [ 19 , 47 , 11 ] . We note that while NDS satisfies D1 , it does not satisfy D4 and there may also be privacy leak on D2 . 3 Scalable Neural Data Server . Scalable Neural Data Server ( SNDS ) is a data search engine for transfer learning . It is designed with application to large data sharing platforms in mind , where each data provider holds a source dataset . The server ’ s goal is to find the most relevant data to a data consumer who has a budget constraint for the amount of data they could afford . Since it is not feasible to enumerate through every example indexed by the server when responding to every consumer , we simplify the search problem from “ find the most relevant examples ” to “ find the most relevant source datasets ” . This simplification was also implied by NDS , who uses experts to represent examples of each source split . SNDS is scalable in that the number of experts downloaded by each data consumer , as well as the amount of compute used in the transaction , remains constant regardless of the number of data providers . This satisfies the computational cost constraint , making it scalable to a large number of source datasets . SNDS is also privacy preserving for both the data consumer and the data providers : no source or target data , nor any models trained on either data , is moved until the recommendation is made . We introduce an intermediary public dataset for training a fixed set of experts . Outputs of these experts on source and target data are used as feature for matching sources to targets . 3.1 SNDS Overview . In SNDS , we assume that the server has access to a public dataset P . The server partitions P into K disjoint subsets , which we denote as { Pk } Kk=1 . The server then optimizes an expert E , belonging to some hypothesis class E , for some objective L with learning algorithm AL , for each Pk . We thereby obtain K experts { Ek } Kk=1 . The server defines a performance function P ( D , E ) : Xn × E → R , which computes some metric using E on a dataset D. The output of P is a single real number , but can also be vector valued . Each data provider i runs the performance function on their data Si . This provides a K dimensional representation pi = ( pi,1 , . . . , pi , K ) for each Si in S , where pi , k = P ( Si , Ek ) . At the same time , a data consumer also evaluates the experts on their target dataset T with P to obtain a representation pT = ( p1 , T , . . . , pK , T ) . These representations are communicated back to the server , who then computes similarity between the target representation and that of each source . Let sim ( a , b ) : RK × RK → R denote a similarity function . We then have z = ( z1 , . . . , zM |zi = sim ( pi , pT ) ) as the similarity scores between the target and each source . Similarity scores are provided to the data consumer as the server ’ s recommendation of sources . The data consumer may then use the similarity scores to select data sources for transfer learning .

Summary: The paper presents a scalable approach to recommend datasets from data providers to consumers looking to assemble datasets to pre-train models for downstream tasks. The approach does not leak source data from data providers and target data from consumers, and instead uses an intermediary public dataset and expert models trained on subsets of this public dataset to evaluate the data in sources that are likely to benefit the target tasks. The approach is scalable (independent of the number of source datasets), does not transfer source or target datasets to a server, transfers only expert models, and evaluates these expert models only on the source datasets and target datasets in the provider and consumer servers alone to keep computation costs low. The experiment results demonstrate the efficacy of this method, and show (somewhat surprisingly) that it is able to identify relevant source datasets even when the intermediate public dataset does not contain data relevant to the target downstream tasks.