Outsourcing Training without Uploading Data via Efficient Collaborative Open-Source Sampling
Authors: Junyuan Hong
Lingjuan Lyu
Jiayu Zhou
Michael Spranger
Venue: NeurIPS 2022 Accept (2022)
Keywords: outsource training
open-source sampling

Abstract: As deep learning blooms with growing demand for computation and data resources, outsourcing model training to a powerful cloud server becomes an attractive alternative to training at a low-power and cost-effective end device. Traditional outsourcing requires uploading device data to the cloud server, which can be infeasible in many real-world applications due to the often sensitive nature of the collected data and the limited communication bandwidth. To tackle these challenges, we propose to leverage widely available open-source data, which is a massive dataset collected from public and heterogeneous sources (e.g., Internet images). We develop a novel strategy called Efficient Collaborative Open-source Sampling (ECOS) to construct a proximal proxy dataset from open-source data for cloud training, in lieu of client data. ECOS probes open-source data on the cloud server to sense the distribution of client data via a communication- and computation-efficient sampling process, which only communicates a few compressed public features and client scalar responses. Extensive empirical studies show that the proposed ECOS improves the quality of automated client labeling, model compression, and label outsourcing when applied in various learning scenarios. Source codes will be released.

Outsourcing Training without Uploading Data via Efficient Collaborative Open-Source Sampling
Junyuan Hong"
Lingjuan Lyu
Michigan State University hongju120man edu
Sony AI lingjuan www.svooy.c.om com
Jiayu Zhou
Michael Spranger
Michigan State University j jiayuz@msu.co edu
Sony AI
michael. setaaell.sppanggreesonoo com
Abstract
As deep learning blooms with growing demand for computation and data resources, outsourcing model training to powerful cloud server becomes an attractive alterna- tive to training low-power and cost-effective end device. Traditional outsoure- ing requires uploading device data to the cloud server, which can be infeasible in many real-world applications due the often sensitive nature of the collected data and the limited communication bandwidth. To tackle these challenges, we propose to leverage widely available open-sonee source data, which is massive dataset collected from public and heterogeneous sources (e.g.. Internet images). We develop novel strategy called Efficient Collaborative Open-souree Sampling (ECOS) to construct a proximal proxy dataset from open-source data for cloud training, in lieu of client data. ECOS probes open-source data on the cloud server to sense the distribution of client data via communication- and computation-effiiie which only communicates a few compressed public features and client scalar re- sponses. Extensive empirical studies show that the proposed ECOS improves the quality automated client labeling, model compression, and label outsourcing when applied in various learning scenarios.
sampling process,
Introduction
Nowadays, powerful machine learning services are essential many devices that supports our daily routines. Delivering such services is typically done through client devices that are power-efficient and thus very restricted in computing capacity. The client devices can collect data through built-in sensors and make predictions by machine learning models. However, their stringent computing power often makes the local training prohibitive, especially for high-capacity deep models. One widely adopted solution is to outsource the cumbersome training cloud servers equipped with mas- sive computational power, using machine-learning -ea-niin--a----seviicc a--eeviee (MLaaS). Amazon Sagemaker [29], Google ML Engine [6], and Microsoft Azure ML Studio [4] are among the most successful industrial adoptions, where users upload training data to designated cloud storage, and the optimized machine learning engines then handle the training. One major challenge of the outsourcing solution in many applications that the local data collected are sensitive and protected by regulations, therefore prohibiting data sharing. Notable examples include General Data Protection Regulation (GDPR) [1] and Health Insurance Portability and Accountability Act (HIPPA) [2].
Work done during internship Sony AI. Corresponding to: Lingjuan Lyu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Open-source data
Efficiency Cluster festures
Proximity
Diversity
Weightad naighbor sampling
Remove reclundant samples
Cloud
OoD
Random samples w/ distributional bins
Centroids 0
Coverage scores
ECOS samples w/ reduced bias
Client
Features
Figure 1: Illustration of the proposed ECOS framework. Instead of uploading local data for cloud training, ECOS downloads the centroids of clustered open-source features efficiently sense the client distribution, where the client counts the local neighbor samples of the centroids as the coverage score. Based on the the scores of centroids, the server adaptively samples proximal and diverse data for training transferable model on the cloud.
On the other hand, recent years witnessed a surging amount of general-purposs and massive datasets authorized for public use, such ImageNet [15]. CelebA [31], and MIMIC [26]. Moreover, many task-specific datasets used by local clients can be well considered as biased subsets of these large public datasets [41, 28]. Therefore, the availability of these datasets allows us use them to model confidential local data, facilitating training outsourcing without directly sharing the local data. One approach is to use the private client dataset to craft pseudo labels for public dataset in a confidential manner [56, 40], assuming that the public and local data are identically-an annttialll-an--ddeeednnntt----- disstiibbed (iid). In addition, Alon et al. showed that iid public data can strongly supplement client learning, which greatly reduces the private sample complexity [3]. However, the iid assumption can often be too strong for general-purpose open-source datasets, since they are usually collected from heterogeneous sources with distributional biases from varying environments. For example, a search of "digits' online yields digits images from handwriting scans, photos, to artwork of digits.
In this paper, we relax the iid assumption in training outsourcing and instead consider the availability of open-source dataset. We first study the gap between the iid data and the heterogeneous open- source data in training outsourcing, and show the low sample efficiency of open-source data. We show that in order to effectively train a model from open-source data that is transferable to the client data, the open-source data needs to communicate more samples than those of iid data. The main reason behind such low sample efficiency is that we accidentally included -utto--distribution (OoD) samples, which poison the training and significantly degrade accuracy at the target (client) data distribution [5]. We propose a novel framework called Efficient Collaborative Open- source Sampling (ECOS) to tackle this challenge, which filters the open-source dataset through an efficient collaboration between the client and server and does not require client data to be shared. During the collaboration, the server sends compressed representative features (centroids) of the open-source dataset to the client. The client then identifies and excludes OoD centroids and returns their privately computed categorical scores to the server. The server then adaptively and diversely decompresses the neighbors of the selected centroids. The main idea illustrated in Fig.
Our major contributions are summarized as follows:
New problem and insight: Motivated by the strong demands for efficient and confidential outsourc- ing, using public data in place of the client data is an attractive solution. However, the impact of heterogeneous sources of the public data, namely open-source data, is rarely studied in existing works. Our empirical study shows the potential challenges due to such heterogeneity.
New sampling paradigm: We propose a new unified sampling paradigm, where the server only sends very few query data to the client and requests very few responses that efficiently and privately guide the cloud for various learning settings on open-source data. To our best knowledge, our method enables efficient cloud outsourcing under the most practical assumption of open-source public data, and does not require accessing raw client data or executing cumbersome local training.
Compelling results: In all three practical learning scenarios, our method improves the model accuracy with pseudo, manual or pre-trained supervisions. Besides, our method shows competitive efficiency in terms of both communication and computation.

---

2
Related Work
There are a series of efforts studying how to leverage the data and computation resources on the cloud to assist client model training, especially when client data cannot be shared [53, 48]. We categorize them as follows: 1) Feature sharing: Methods like group knowledge transfer [21], split learning [47] and domain adaptation [18, 17] transfer edge knowledge by communicating features extracted by networks. To provide a theoretical guarantee of privacy protection, [37] proposed an advanced information removal to disentangle sensitive attributes from shared features. the notion of rigorous privacy definition, Liu ef al. leveraged public data to assist private information release [30]. Earlier, data encryption was used for outsourcing, which however is too computation-intesiiyy for client and less applicable for large-scale data and deep networks [12, 27]. Federated Learning (FL) [34] considers the same constraint on data sharing but allocates the burdens of training [23] and communication [57] to clients and opens up series of challenges
privacy [11], security [33, 10] and knowledge transfer [24]. 2) Private labeling: PATE and its variants were proposed to generate client-ap dieett-pproximated labels for unlabeled public data, on which model can be trained [39, 40].
Without training multiple models by clients, Private kNN was a more efficient alternative which explored the private neighborhood of public images for labeling [56]. These approaches a strong assumption of the availability of public data that is iid as the local data. This paper considers more practical yet challenging setting where public data are from multiple agnostic sources with heterogeneous features.
based on
Sampling from public data has been explored in central settings. For example, Xu et al. [51] used few target-domain samples as a seed dataset to filter the open-domain datasets by positive-unlabeled learning [32]. Yan et al. [52] used model to find the proxy datasets from multiple candidate datasets. In self self-super contrastive learning, model-aware K-center (MAK) used a model pre-trained on the seed dataset to find desired-clate samples from open- world dataset [25]. Though these methods provided effective sampling. they are less applicable when the seed dataset is placed at the low-energy edge, because the private seed data at the edge cannot be shared with the cloud for filtering and the edge device is incapable of computation-intensvy training. To address these challenges, we develop new sampling strategy requiring only light-weight computation at the edge.
3 Outsourcing Model Training With Open-Source Data
3.1 Problem Setting and Challenges
Motivated in Section 1, we aim to outsource the training process from computation-conssrainee devices the powerful cloud server, where proxy public dataset without privacy concerns is used in place of the client dataset for cloud training. One solution is (private) client labeling by k-rearest-neighbors (kNN) [56], where the client and cloud server communicate the pseudo-label of a public dataset privately and the server trains a classifier by the labeled and unlabeled samples in a semi-supervised manner. The success of this strategy depends
the key assumptions that public data in the cloud and private data in the client are iid, which are rather strong in practice and thus prevent it from many real-world applications. In this work, we make a more realistic assumption that the public datasets are as accessible as open- source data. An open-source dataset consists of
Table 1: Test accuracy (%) with different client domains (columns). Cloud data are identically distributed as the client data (ID) or including more data from distinct domains (ID+OoD) without overlapped samples. We first label number of randomly selected cloud examples (i.e., sampling budget) privately by client data [56). and then train a classifier to recognize digit images. The privacy cost is accounted for in the notion of differential privacy. Larger budgets imply more privacy and communication costs. More results on different settings are enclosed in Appendix B.3.
Cloud Data
Sampling Budget
MNIST Acc (5)
SVHN
USPS
SynthDigits
MNIST-M
Average (%)
c Acc (S6)
c Acc (S))
Acc (%) Acc (R) d
ID
1000
84.3=24
4.48 51 61.621.4
4.08
87.1205
4.51
73.2115
4.57
55.52100 4.46
70.4
4.42
1000
78.0=3.5
4,30 40.6+1.6
3.75
82.2+27
4.32 62.1+16
4.41 49.1+1.
4.27
62.4
4.21
ID+OoD
8000
82.2
47.9+18
85.4205
64.453.
53.3
5,89
66.6
16000
82,6+14
7.17
48.5+1.7
7.17
86.7419
7.17
67.5e23
7.17
52.043.
7.17
67.4
7.17
32000
84, 14.1
9.32
49.4
9.32
86.842.
9.32
68.5g0.1
9.32
53.0+2.7
9.32
68.4
9.32
3

---

biased features from multiple heterogeneous sources (feature domains), and therefore includes not only in-distribution (ID) samples similar to the client data but also multi-domain OoD samples.
The immediate question is how the OoD samples affect the outsourced training. In Table 1, we empirically study the problem by using 5-domain dataset, Digits, where 50% of one domain is used on the client and the remained 50% together with the other 4 domains serve as the public dataset on the cloud. To conduct the cloud training, we leverage the client data generate pseudo labels for the unlabeled public samples, following [56]. It turns out that the presence of OoD samples in the cloud greatly degrades the model accuracy. The inherent reason for the degradation that the distributional shift of data [43] compromised the transferability of the model to the client data [50].
Problem formulation by sampling principles. Given client dataset D and an open-source dataset D9, the goal of open-source sampling is to find a proper subset S from D', whose distribution matches DP. In [25], Model-Aware K-center (MAK) formulated the sampling principled optimization:
minsgo A(S, DP) - H(SU U DP; D*),
(1)
where DP) == DD [minges (lx(x) x(x')1² measures proximity as the set differ-
ence between S and D using a feature extractor Φ, and the latter H(S U DP; D*) == maX minzesuDe llØ(z) - =(2')ll measures diversity by contradicting S U D and D (sup- pose D9 is the most diverse set)². Solving Eq. (1) results in an NP-hard problem that is intractable [13], and MAK provides an approximated solution by a coordinate- wise greedy strategy. It first pre-trains the model representations on D2 and finds large candidate set with the best proximity to extracted features. Then, it incrementally selects the most diverse samples from the candidate set until the sampling budget is used up.
Though MAK is successful in the central setting, it is not applicable when DP is isolated from cloud open-source data and is located at resource-constrained client for two reasons: 1) Communication inefficiency. Uploading client data may result in privacy leakage, sending public data to the client is a direct alternative but the cost can be prohibitive. 2) Computation inefficiency. Pre-training a model on DP or proximal sampling (which computes the distances between paired samples from D and D2) induces unaffordable computation overheads for the low-energy client.
3.2 Proposed Solution: Efficient Collaborative Open-Source Sampling (ECOS)
To address the above challenges, we design a new strategy that I) uses compressed queries to reduce the communication and computation overhead and 2) uses novel principled objective to effectively sample from open-source data with the client responses of the compressed queries.
Construct communication-efiiee
and an informative query set &8 at cloud. Let d be the
number of pixels of an image, the communication overhead of transmitting De to the client is given by (d D) For communication efficiency, we optimize the following two factors:
i) Data dimension d. First, we transmit extracted features T = {<{=)] D*} instead of images to reduce the communication overhead to (d,,) D))), where d, is a much smaller embedding dimension.
For accurate estimation of the distance A, pre-defined discriminative feature space is essential without extra training on the client. Depending on resources, one may consider hand-crafted features such as HOG [14], or an off-the-shelf pre-trained model such as ResNet pre-trained on ImageNet. ii) Data size (D).. Even with compression, sending all data for querying is inefficient due to the huge size of open-source data [ Meanwhile, too many queries would cast unacceptable privacy costs to the client. As querying on similar samples leads to redundant information in querying, we propose to reduce such redundancy by selecting informative samples. We use the classic clustering method KMeans [20] for compressing similar samples by clustering them, and collect the R mean vectors or centroids into के = {cr r=11 We denote R as the compression size and πi as the set of original samples corresponding to $
New sampling objective. We note that sending the compact set के in place of Da prohibits the client from optimizing ASS, DP) in Eq. (1) for S E Dt, Instead, we sample set of centroids Š E 69 and decompress them by the cluster assignment into corresponding original samples with rich features
Note that we use Lz-norm distance instead of normalized cosine similarity in A(S, DP) in contrast to MAK, since normalized cosine similarity is not essential if the feature space is not trained under the cosine metric. We also omit the tailedness objective which is irrelevant in context.

---

afterwards. In principle, we leverage the inequality ASS, DP) & A(S, DP) + S) to attain
communication-efine efficcen surrogate objective as follows:
sgs, ASS, D + A(S
H(S; D
peoximity
diversity
where (or D is the compressed centroid substitute of (or DO). Different from Eq. (I), we decompose the proximity term into two in order facilitate communication efficiency leveraging an informative subset 1>9 We solve the optimization problem in greedy manner by steps the client and the cloud, respectively:
i) At the client step, we optimize A(5, DP) to find a subset of centroids (Š C ②i9 that are proximal to the client set DP. Noticing that 1i contains the cluster centroids, we take advantage of the property define a novel proximity measure of the cluster r: Centroid Coverage (CC), denoted as ey- Upon receiving centroids from the cloud, the client uses them to partition the local data into {C Tr=1 where O denotes the r-th cluster partition local data. We compute the CC score by the cardinality of the neighbor samples of the centroid i.e., Ur |Cp). To augment the sensitivity to the proximal clusters, we scale the CC score by non-linear function e/- = ei (ICP)), where the scale function (x)) x² is parameterized by îi) At the cloud step, we optimize the proximity of S w.r.t. the proxy set i.e. A(Š,S). and remove redundant and irrelevant samples from the candidate set to encourage diversity, i.e. -H (S; D9). As samples among clusters are already diversified by KMeans, we only need promote the in-cluster diversity. To this end, we reduce the sample redundancy per cluster at cloud by K-Center [44], which heuristically finds the most diverse samples. Such design transfers the diversity operation to cloud and thus reduces the local computation overhead. To maintain the preximiry, the K K-Center is applied within each cloud cluster and the sampling budget per cluster is proportional to their vote numbers and the original cluster sizes. With the normalized scores, we compute the sampling budget per cluster which upper bounded by the ratio of the cluster in the cloud set.
Algorithm Efficient collaborative open-source sampling (ECOS)
Input:
Client dataset DP, cloud query dataset D sampling budget B, compression size R, feature extractor Φ(-), distance function Azz, S) = minges [lő(z) <by)l,, initial sample set S score scale function (x) = x²
Extract features 4 = {0(x)) D9}:
2: Cloud creates a compressed dataset q KMeans
P> Compress R Centroids
3: I> > b Client End >
4: Download the feature extractor e and $4;
5: Use centroids के to partition Φ {o(#)) (ar DP} into clusters (CF)
6: Compute the Centroid Coverage (CC) scores; u, = (CP), Vr
7: Upload scaled cluster scores {or = (o,))
b Proximity
8: < <4 4 Cloud End < 44
9: Partition D into clusters {C#) by centroids Φ2;
10: Compute per-cluster sampling budget 6, = min
B;
11: for in {l..... R} do Initialize 5' {a} by a sample randomly picked from C,; 13: while | < b, do
Decompress Centroids
b Diverse Sampling
== arg max (xx,S);
15:
5' US:;
16:
= S'
17:
return
We summarize our algorithm in Algorithm 1, which enables the clients to enjoy better computation efficiency than local training and better communication efficiency than the centralized sampling (e.g.. MAK). 1) Computation efficiency. Since our method only requíres inference operations on the client device, which should be efficiently designed for the standard predictive functions of the device, and is training-fre free for the client, the major complexity of ECOS is on computing centroid coverage and much lower than cradient-based algorithms whose complexity scales with the model size and training
5

---

iterations. As computing the CC scores only requires the nearest centroid estimation and ranking, the filtering can be efficiently done. The total time complexity OC (d,++dd++ 1) DPI), + dominated by the first term, where Co is the complexity of extracting features depending on the specific method.
The second term (de + 1) | is for computing the pair-wise distances between Φ and 49 and estimating the nearest centroids per sample (or partitioning client data). In a brief comparison, the complexity of local 7-iteration gradient-descent training could be approximately (f which is much more expensive since typically TC3 >> de. To complete the analysis. the space complexity OCC² + (d + DPI + d.R. + | for the memory footprint of Φ, the images (d) features (d.) of client and cloud centroid data, and the distance matrix. 2) Communication efficiency. The downloading complexity will be (dg]l) for It d,-dimensional centroid features and the uploading complexity is (R) including indexes of samples. Thus, the data that will be communicated between the client and cloud is approximately of O(d, R+ Rt) complexity in total. In comparison, downlouding the whole open-source dataset by central sampling (e.g. MAK) requires O(d) - B) complexity. As R < R.R < del < d our method can significantly scale down the computation cost.
Privacy protection and accountant. When the cloud server is compromised by an attacker, uploading CC scores leak private information of the local data samples, for example, the presence of an identity [45]. To mitigate the privacy risk, we protect the uploaded scores by a Gaussian noise mechanism, i.e., G = Ur + +N(00, o²), and account for the privacy cost in the notion of differential privacy (DP) [19]. DP quantifies the numerical influence of the absence of a private sample on [vi,..., [R]. which is connected the chance of exposing the sample to the attacker. To obtain tight bound on the privacy cost, we utilize the tool of Rényi Differential Privacy (RDP) [36] and leverage the Poisson sampling to further amplify the privacy [55]. With the noise mechanism governed by a, the resultant privacy cost the sense of (c, 5-D can be accounted as = O0 log(1 / / where y is the Poisson subsampling rate and 8 is a user-specified parameter. A larger e implies higher risks of privacy leakage in the probability of 8. Formal proofs can be found in Appendix A.3.
Generalization error of models trained on cloud. Our work can be viewed as knowledge transfer from the open-source domain to the private client domain. Therefore, we present Theorem 3.1 based on prior comain-adappationn theoretical results [7, 18, 17].
Theorem 3.1. Assume that open-source dataset S is induced from a mixture of cluster distributions,
R Σx= 0, 1. Suppose client data are i.e., Σm= a, DY with cluster distribution Dj, a, [0, 1] and
sampled from DP. Let L(-,-) be a loss function on a hypothesis and a dataset (for empirical error) or a distribution (for generalization error). If f is governed by the parameter A trained on S and belongs to a hypothesis space H of VC-dimension d, then with probability at least - p over the choice of samples, the inequality holds,
T( DP) S S) + 1
-n_,, n
2d 2dlog(2|SS) + log(4/p) E,
2 p-= ondHAn (D2, DP) + V
(3)
[S]
where & ming (I(fe, + ((fe,D"), and dHan(D) (D, D') denotes the distribution divergence.
The proof of Theorem 3.1 is deferred to Appendix A.4. Theorem 3.1 shows that given model trained on cloud, its generalization error on client data hinges on the quality and size of the ECOS-sampled subset. Informally, if the sampled set has enough samples (the 3rd term) and follows similar distribution as the client dataset (the 2nd term), then the model generalizes better via only training on the proxy cloud dataset (the 1st term). In Theorem 3.1, the model fe can be trained by any task-specific optimization method. For example, minimizing a p-strongly-convex and G-smooth loss function Lffo. S) w.r.t. 0 via T-iteration gradient descent leads to L(fe, S) = (1 where L is the optimal loss.
S) L'I
Given Eq. (3), it can be shown that minimizing the upper bound requires a dedicated off between sample size and quality. Consider a simple case, only minimizing Σ/= a,duax DP). The solution is that a, equals 1 if dHAu(D") (D), DD)) is smallest among all choices of clusters, otherwise zero. However, the number of samples in the cluster r is limited as (C) which is much smaller than the whole open-source dataset [D'], and therefore |S] will be too small to enlarge the third term in Eq. (3). Thus, a, should be smoothed to increase sample size by trading in some quality (distribution divergence). In our implementation, we approximate o, via the CC score of,, where an < oo smooths the a, to trade off sample size and quality. When 8 vanishes, all clusters will be selected at the same chance and the divergence could be large.

---

Libe's
Large f
model,
Unlassled doud 6323
Prodiral Dora
Lebelos prodinel
Som supervised Learning
Lebaled doud dots
Selected proeinal
Intebsted diant datts
KODS
Uniabared dl dlent dete
DOGS
(a) Selective manual labeling.
(b) Adaptive model compression.
Uniabaled doul dota
prodiaal
riaen
(c) Automated client labeling.
Figure 2: Our method applicable to various cloud training cases, where ECOS filters the open- source samples to improve the model performance trained on (a) manual, (b) pre-trained model (teacher), and (c) pseudo supervisions.
4 Empirical Results
Datasets. We datasets from two tasks: digit recognition and object recognition. Distinct from prior work [56], in our work, the open-source data contains samples out of the client's 's distribution. With the same classes as the client dataset, we assume open-source data are from different environments and therefore include different feature distributions. for example, DomainNet [41] and Digits [28]. DomainNet includes large-sized 244 244 everyday images on domains and Digits consists of 28 x 28 digit images on 5 domains. Instead of using an overly large volume of data from a single domain like [56]. we tailor each-domain subset to contain fewer images than standard digit datasets, for instance, MNIST with 7438 images, which was previously adopted in the distributed learning setting [28] and mitigates the hardness of collecting enormous data. In practice, collecting tens of thousands of images by single client, e.g.. 50000 images from MNIST domain, will be unrealistic. Similarly, DomainNet will be tailored to only include 10 classes with 2000-5000 images per domain. domain for the client and the rest domains
Splits of client and cloud datasets. For Digits, for the cloud as open-source set. For DomainNet, we randomly select 50% samples from one domain for the client and leave the rest samples together with all other domains to the cloud. The difference of configurations for the two datasets is caused by their different domain gaps. Even without ID data, it is possible for Digits to transfer the knowledge across domains.
Baselines. For a fair comparison, we compare our method to baselines with the same sampling budget. Each experiment case is repeated for three times with seeds {1, 2, 3]. We account for the privacy cost by Poisson-subsampling RDP [55] and translate the cost to the general privacy notion, (e, 6DP when 8 10 15-5 Here, we use the random sampling
naive baseline. We also adopt a coreset selection method, K- K-Center [44], to select informative samples within the limited budget. Both baselines are perfectly private, because they do not access private information from clients. Details of
are deferred to Appendix B.I.
4.1 Evaluations on Training Outsourcing
To demonstrate the general applicability of ECOS, we present three practical use cases of outsourcing. categorized by the form of supervisions. The conceptual illustrations are in Fig. 2. Per use case, we the client test set
train model on partially labeled cloud data (outsourced training) and accuracy is reported together with standard deviations. We present the results in Tables 2 to 4 case by case, where we vary the domain of the client by columns. In each column, we highlight the best result unless the difference is not statistically significant.
Case 1: Selective manual labeling. We assume that the cloud will label the filtered in in-domain samples by using third-party label service, e.g., Amazon Mechanical Turk [38], or by asking the end clients for manual labeling. As the selected samples are non-private, they can be freely shared with third parity. But the high cost of manual labeling or service the pain point, which should be carefully constrained within a finite budget of demanded labels.
7

---

Table 2: Test accuracy by selective labeling on Digits (top) and DomainNet (bottom).
Sampling
MNIST
SVHN
USPS
SynthDigits
MNIST-M
Average Acc (%) f
Budget
Method
(%)
(%)
(5)) f 4 Acc (86) f Acc (%) f <
Ours 97.3.001 0.22 68 7.0.3 0.22
90.810.1 0.22 84. 610.6 0.22 70.110.6 0.22
82.3
2000
K-Center
96.7=0.3 0.00 65.1+1a 0.00
90.1+0.7 0.00 80.2s1. 0.00
70.1+0.3 0.00
80.4
Ramdom
96.520.3 0.00
64.020.3 0.00
91.6+1.0
0.00
83.8+0.3
0.00
70.9#0.6
0.00
81.4
Ours
98.1 20.2 0.22 74. 6...6:
0.22 93. 33.5
0.22
01.2
0.22
74.520.5
0.22
86.4
5000
K-Center
97.910.2 0.00
72.310.8 0.00
92.710.9
0.00
89.610.3
0.00
74.010.5
0.00
85.3
Random
97.6.0.3 0.00 70.0
0.00
93.0a0.6
0.00
89.7.0.4
0.00
73.9
0.00
81.8
Local 99. 11..1 0.00 88 810.2 0.00 98. 98.910.1
0.00 96.110.2 0.00
88.810.2 0.00
941.1
Sampling
Clipart
Infograph Acc (%) 1
Painting
Quickdraw
Real
Sketch
Averago Acc (16 f
Budget
Mothod
Acc (9) 4
4 Acc (5)
Acc (%) e
Acc (98) 1 <
Acc (18 1
Ours 88.4
0,58 52.6+0.9 0.58 90.4+1. 0.58 84.3#16 0.58 92.1+1.2 0.58 87.2=0.6 0.58
82.5
1000
K-Center
86.8:0.3
0.00 50.520. 0.00 89.1 141. 0.00
27.2#1. 0.00 92.52201 0.00 85.6=1.4 0.00
72.0
Random
86.910.8
0.00
47.412.7
0.00
88.610.1
0.00
77.912.4
0.00
91.410.3
0.00
86.9.0.3
0.00
79.9
Ours
93.2+04
0.58
58.1++0.0
0.58
92.5+1.1
0.58
89. 2:0.
0.58
94.440.2
0.58
02.8=0.2
0.58
86.7
3000
K-Center
93.5#1.11
0.00
56.3+0.3
0.00
92.9=0.3
0.00
60.5+8.7
0.00
94.1+0.2
0.00
92.1=0.7
0.00
81.6
Random
92.5+0.0
0.00
53.6+1.4
0.00
91.7+00.
0.00
86.1 0.00
93.5+0.3
0.00
93.0=0.2
0.00
85.1
Local 87. 0.00 51.7+0.7 0.00 85.9#0.4 0.00 83.5#0.3 0.00 93.5+001 0.00 81.7=0.6 0.00
80.6
Given a specified sampling budget, we compare the test accuracy of semi-supervised learning (FixMatch) on sampled data in Table 2. Since the ECOS tends to select in-distribution samples, it cases the transfer of dlouddtraiiee model to the client data. On the Digits dataset, we find that our method attains more accuracy gains as budget increases, demonstrating that more effective labels are selected. On the DomainNet dataset, our method outperforms baselines on 5 out of 6 domains and is stable in most domains given a small budget (1 (1000) and is superior average. Given higher budgets, the accuracy of all methods are improved, and our method is outstanding on Infograph and Quickdraw domains and is comparable to the baselines on other domains.
Table 3: Test accuracy by adaptive model compression on DomainNet.
Sampling
Infograph Ace (%) t Acc (8)
Clipart
Painting
Sketch Ace (S) f J Acc (56 J Acc (56 f
Quickdraw
Real
Average Acc (16)
Budget
Method
Ace (
Ours
82.9=0.7 0.58 48 48.5
0.58
85.4+1.0
0.58
81.3+2.5 0.58 91.4+0.6 0.58 82, 82.4+1.1 0.58
78.6
1000
K-Center
81.2209
0.00
44.620.9
0.00
84.5222
0.00
41.7+19
0.00 92. 52.4 0.00
80.1 0.00
70.8
Random
83.810.7 0.00
41.11211
0.00
83.811.8
0.00
76.3122
0.00
90.110.8
0.00
80.110.7
0.00
76.1
Ours
90. 60.6 0.58
51.4#1.9
0.58
89 6:12
0.58
87 640.3
0.58
93.610.7
0.58
88 4+1.5
0.58
83.5
3000
Keneter
88.9=2.3 0.00
51.2+0.4
89.4+0.9
0.00
57.6+44
0.00
94.5+0.5
0.00
86.9+0.
0.00
78.1
Random
88.4208 0.00 47.6+1.9 0.00
89.421.0 0.00
84.7+00.
0.00
93.021.0 0.00
86.3#1.2 0.00
81.6
Adaptive model compression. Due to the large volume of the open-source dataset, a larger model is desired for better capturing the various features, which however is so inefficient to fit into the resource-constraing Confronting this challenge, model compression [9] is a conventional idea to forge a memory-efficiem model by transferring knowledge from large models to small ones. Specifically, we first pre-train a large teacher model fr on all cloud data by the supervised learning, assuming labels are available model to initialize the feature extractor o(-) of a
Case 2:
client devices or specialize for the data distribution of the client.
in advance. Still, we use an weangeeee--re-ttriinne student model f.. Then the client will use the downloaded feature extractor é to filter samples. Here, we utilize knowledge distillation [22] to finetune the fa with an additional classifier head upon the ø. On the selected samples, we train a linear classifier head for 30 epochs under the supervision of true labels and the teacher model ft, and then fine-tune the full network fa for 500 epochs. The major challenge comes from distributional biases between the multi-source open-source source data and the client data. Leveraging ECOS, we may sample data near the client distribution and reduce the bias in the follow-up compression process.
We simulate the case on the large-sized image dataset, NomainNet, which is demanding for large-scale networks, e.g., ResNet50. to effectively learn the complicated features. Here, we compress ResNet50 into a smaller network, ResNet18, 18, by using an adaptively selected subset of the cloud dataset. We omit the experiment for digit images where a large model may not be necessary for such a small image size. In Table 3, we present the test accuracy of compressed ResNet18 using 3000 samples from DomainNet in finetuning. With a small portion of privacy cost (e < 0.6), our method imprrves the accuracy on Clipart and Real domains against the baselines. Note that the model accuracy here is
8

---

lower than label outsourcing in Table 2, and reason is that the supervisions from the larger models are just an approximation of the full dataset. Without using the full dataset for compression, the training can be completed fast and responsively on the demand of client.
Table 4: Test accuracy of client labeling two datasets: Digits (top) and DomainNet (bottom).
Sampling
MNIST Ace (56) 1 +
SVIIN
USPS
SymhDDigits Acc (9) f
MNIST-M
Average Acc (9) f
Budget
Method
Acc (96) f 4 Acc (96)
Acc (%) <.
Ours
84.2 2.23 5.35
47.913.1 5,32 86. 1.10 5.35 68.6116 5.35 58. 41.9 5.35
69.0 67.1 65.9
5000
K-Ceater
81.9 933,4 5.34
18. 1112 5.33
82.1112 5,34
69.4119 5.34
55.4120 5.34
Rundom
81.8=4.1 5.34
45.3+3.0
5.29 81.242. 5.34 65.9+27 5.34 55.5+26 5.34
Sampling
Clipurt Acc (9) f
Infograph Acc (9)
Painting
Quickdraw
Real
Sketch
Average
Budget
Method
Ace (9)
Acc (8)
Acc ( + Acc (%) 1 c Acc (9) f
Ours
33.2=55 4.46 23.8+22 3.50 47.4*a. 4.51 39.8477 4.87 62.9e19 4,92 51.7+12 4,94
43.2 42.0
3000 K Centter
39.3 3.6 4.57
2827 3.61
66.6 4.52 36. 36.1 4.94 63.8 33.8 4.96 47.7=28 4.96
Random
30.7=222 4.41 21.6+4. 3.43 44.0a3.1 4.53 39.045.6 4.82 59.9sa.6 4.75 47.2+3. 4.94
40.4
Case 3: Automated client labeling. When the client obtained labeled samples, for example, photos labeled by phone users, the cloud only needs to collect an unlabeled public dataset. Therefore, we may automate the labeling process leveraging the client supervision knowledge reduce cost or users' efforts on manual labeling. To be specific, we let the client generate pseudo labels for the cloud data based on their neighbor relation, as previously studied by [56] (private kNN). However, the private kNN assumes that the client and the cloud follow the same distribution, which weakens its applicability confronting the heterogeneity and the large scale of open-sourge data. Thus, we utilize ECOS as pre- pre-processing of open-source data before being labeled by private kNN. Therefore, we have two rounds of communication for transferring client knowledge: proximal-date sampling by the ECOS and client labeling by the private KNN [56]. To compose the privacy costs from these two steps, we utilize the analytical moment accountant technique to get a tight privacy bound [49]. Interested readers can refer to Appendix A. for brief introduction to private KNN and our implementations.
In experiments, we use the -fate-of-the-- art private lseuui-labeling method, private kNN [56]. to label the subsampled open open-source data with the assistance from the labeled client dataset. To reduce the sensitivity of private kNN w.r.t. the threshold, we instead enforce the number of the selected labels to be 600 and balance the sizes by selecting the top-60 samples with the highest confidence per class. Then, we adopt the popular semi-supervised learning method, FixMatch [46], to train the classifier. In Table 4, we report the results when cloud features are distributionally biased from the client ones but they share the same class set. For each domain choice of client data, we will use the other domains as the cloud dataset. Distinct from prior studies, e.g., in [56] or [40], we assume 80-90% of the cloud data are out of the client distribution and are heterogeneously aggregated from different domains, casting greater challenges in learning. On the Digits, we eliminate all ID data from the cloud set to harden the task. Both on Digits and DomainNet datasets, our method consistently outperforms the two sampling baselines under the similar privacy costs. The variance of privacy costs is mainly resulted from the actual sampling sizes. Though simply adopting K- K-Center outperforms the random sampling, it still presents larger gaps compared to our method in multiple domains. For instance, in Quickdraw domain, given 3000 sampling budget, the K-Center method performs poorly and is even worse than random sampling.
4.2 Qualitative Study
To better understand the proposed method, we conduct a series of qualitative studies on client labeling. We use two DomainNet datasets as a representative benchmark in the studies.
Table Ablation study of the proposed method on DomainNet. Test accuracy of the client labeling case is reported.
Proximity Diversity
Clipart
Infograph
Painting
Quickdraa
Real
Sketch
Average
x
30.719.3
21 .1.61499
44.013.1 39.015.6
59.913.
47.213-4
40.4
25.5 #5.7
140.5
41.455a 31.3aLa
60.3+2.
31.9220
35.2
39.8
18.229.7
46.619.5
119.
811.9
47.7
42.0
33.215.9
28.8
47.413.
89.81777
62.911.9
61.711.3
48.2

---

1) Ablation study. In Table 5, we conduct ablation studies to evaluate the effect of different objectives introduced by ECOS, following the client labeling benchmark with a 3000 sample budget. Without proximity and diversity objectives, we let the baseline be the naive random sampling. We first include the proximity objective, where we greedily select samples from top-scored clusters until the budget is fulfilled. However, we find that the naive proximity objective results in a quite nega- tive effect compared to the random baseline. The failure can be attributed to the nature of clustering, which includes more similar samples and thereby lacks diversity. When diversity is encouraged and combined with the proximal votes, we find the performance is improved significantly in multiple domains and on average. Now with diversity, the proximity objective can further improve the sampling in Painting, Quickdraw, and Sketch domains significantly. 2) Visualize cluster selection. In Fig. we demonstrate that the CC can effectively reject OoD centroids. Also, it is interesting to observe that when multiple centroids are distributed closely, then they will compete with each other and reject the redundant ones consequently. 3) Effi- ciency. In Section 3, we studied the communication efficiency theoretically. Empirically, ECOS only needs to upload 100 bytes of the CC scores in all experiments, while traditional outsourcing needs to upload the dataset, which is about 198MB for the lowest load in DomainNet (50% of Sketch domain data for client). Even counting the download load, ECOS only needs to download 45MB of the pre-trained ResNet18 feature extractor together with 51KB data of centroid features, which is much less than data uploading. More detailed evaluation of the efficiency is placed in Appendix B.4 and we study how sample budgets and privacy budgets trade off the accuracy of ECOS models in Appendix B.3.
Fed ded
sentro
Figure 3: Demonstration of the cen- troids qualified by private CC. We use blue dots to represent the client data from Real domain of Domain- Net. For the data of the cloud do- mains, larger circles represent cen- troids with higher CC and orange crosses rejected OoD centroids.
5 Conclusion
In this paper, we explore the possibility of outsourcing model training without access to the client data. To reconcile the data shortage from the target domain, we propose to find a surrogate dataset from the source agnostic public dataset. We find that the heterogeneity of the open-source data greatly compromises the performance of trained models. To tackle this practical challenge, we propose a collaborative sampling solution, ECOS, that can efficiently and effectively filter open-source samples and thus benefits follow-up learning tasks. We envision this work as a milestone for the private and efficient outsourcing from low-power and cost-effective end devices.
We also recognize open questions of the proposed solution for future studies. For example, the public dataset may require additional data processing, e.g., aligning and cropping for improved prediction accuracy. In our empirical studies, we only consider the computer vision tasks, though no assumption was made the data structures. We expect the principles to be adapted to other data types with minimal efforts. More data types, including tabular and natural-language data, will be considered in the follow-up works. Take the language data for example: BERT is among the most popular pre-trained models for extracting features from sentences [16], upon which ECOS can assistant to sample proxy data from massive public dataset, for instance, WikiText- 103 [35] based on 28,475 475 Wikipedia articles. Alternative to the Lz-norm based proximity objectives, advanced semantic features could also enhance the sampling effectiveness of ECOS in language data [54]. More discussions on the social impacts of this work are enclosed in Appendix A.5.
Acknowledgments and Disclosure of Funding
This research was funded by Sony AI. This material is based in part upon work supported by the National Science Foundation under Grant IIS-2212174, 17499749,, Office of Naval Research N00014-20-1- 1--22-2 and National Institute on Aging (NIA) RFI RFIAG072449.
10

---

References
[T] General data protection regulation. https://gdpr-innoo eu/-
[2] A. Act. Health insurance portability and accountability act of 1996. Public law, 104:191 1996.
[3] N. Alon, R. Bassily, and S. Moran. Limits of private learning with access to public data. Advances in neural information processing systems, 32, 2019.
[4] J. Barnes. Azure machine learning. Microsoft Azure Essentials. 1st ed, Microsoft, 2015.
[5] B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. arXiv 2w...2006,66.9 2012.
preprint
[6] E. Bisong. Google cloud machine learning engine (cloud mle). In Building Machine Learning and Deep Learning Models on Google Cloud Platform, pages 545-579. Springer, 2019.
[7] J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman. Learning bounds for domain adaptation. Advances in neural information processing systems, 20, 2007.
[8] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[9] C. Buciluă, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining, pages 535-541, 2006.
[10] C. Chen, Y. Liu, X. Ma, and L. Lyu. Calfat: Calibrated federated adversarial training with label skewness. In Advances in Neural Information Processing Systems, 2022.
[11] C. Chen, L. Lyu, H. Yu, and G. Chen. Practical attribute reconstruction attack against federated learning. IEEE Transactions Big Data, 2022.
[12] Chen, T. Xiang, X. Lei, and Chen. Highly efficient linear regression outsourcing to cloud. IEEE transactions cloud computing, 2(4):499-508, 2014.
[13] W. Cook, W. H. Cunningham, W. Pulleyblank, and A. Schrijver. Combinatorial optimiza- tion. Oberwolfach Reports, 5(4):2875-2942, 2009.
[14] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 055, volume 1, pages 886 893. Ieee, 2005.
[15] Deng, W. Dong, Socher, L.-J. K. Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018
[17] Dong, Y. Cong, G. Sun, 7.. Fang, and Ding. Where and how to transfer: Knowledge angregation-indueed transferability perception for unsupervised domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2021.
[18] J. Dong, Y. Cong, G. Sun, B. Zhong, and X. Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
June 2020.
[19] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise sensitivity in private data analysis. In Theory of Cryptography, Lecture Notes in Computer Science, pages 265-284. Springer Berlin Heidelberg. 2006.
[20] E. W. Forgy. Cluster analysis of multivariate data: efficiency versus interpretability of classifica- tions. biometrics, 21:768-766 -666, 1965.

---

[21] C. He, M. Annavaram, and S. Avestimehr. Group knowledge transfer: Federated learning of large cnns the edge, Advances Neural Information Processing Systems, Nov. 2020.
[22] G. Hinton, O. Vinyals, Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 02531, 2(7), 2015.
[23] J. Hong, H. Wang. Z. Wang, and J. Zhou. Efficient split-mix federated learning for on-demand and in-situ customization. arXiv preprint arXiv:2203.0977 2022.
[24] Hong, Z. Zhu, S. Yu, Z. Wang, H. H. Dodge, and Zhou, Federated adversarial debiasing for fair and transferable representations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 617-627, 2021.
[25] Z. Jiang, T. Chen, T. Chen, and Z. Wang. Improving contrastive learning on imbalanced data via open-world sampling. In Advances in Neural Information Processing Systems, May 2021.
[26] A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, Szolovits, L. Anthony Celi, and R. G. Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[27] X. Lei. X. Liao, T. Huang, H. Li, and C. Hu. Outsourcing large matrix inversion computation to a public cloud. IEEE Transactions on cloud computing, 1(1):111, 2013.
[28] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q Dou. Fedbn: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, Sept. 2020.
[29] E. Liberty, Z. Karnin, B. Xiang, L. Rouesnel, B. Coskun, R. Nallapati, J. Delgado, A. Sadoughi, Y. Astashonok, P. Das, et al. Elastic machine learning algorithms in amazon sagemaker. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pages 731-737. 2020.
[30] T. Liu, G. Vietri, T. Steinke, J. Ullman, and S. Wu. Leveraging public data for practical private query release. In International Conference on Machine Learning, pages 6968-6977. PMLR, 2021.
[31] Z. Liu, P. Luo. X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[32] M. R. Loghmani, M. Vineze, and T. Tommasi. Positive-unllbeled learning for open domain adaptation. Pattern Recognition Letters, 136, June 2020.
[33] L. Lyu, H. Yu, X. Ma, C. Chen, L. Sun, J. Zhao, Q. Yang, and P. S. Yu. Privacy and robustness in federated learning: Attacks and defenses. arXiv preprint arXiv:2012.06337, 2020.
[34] B. McMahan, F. Moore, D. Ramage, S. Hampson, and B. A. y Areas. Communication-effcien learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[35] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint arXiv: (77447744, 2016.
[36] I. Mironov. Rényi differential privacy. 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263-275, Santa Barbara, CA, USA. Aug. 2017. IEEE.
[37] S. A. Osia, A. Taheri, A. S. Shamsabadi, K. Katevas, H. Haddadi, and H. R. Rabiee. Deep private-feate feature extraction. JEEE Transactions on Knowledge and Data Engineering, 32(1):54- 66, Jan. 2020.
[38] G. Paolacci, J. Chandler, and P. G. Ipeirotis. Running experiments amazon mechanical turk. Judgment and Decision making, 5(5):41 1-499, 2010.
[39] N. Papernot, M. Abadi, Ó. Erlingsson, I. Goodfellow, and K. Talwar. Semi-supervised knowl- edge transfer for deep learning from private training data. In International Conference on Learning Representations, 2016.
12

---

[40] N. Papernot, S. Song, I. Mironov, A. Raghumathan, K. Talwar, and U. Erlingsson. Scalable private learning with pate, International Conference on Learning Representations, Feb. 2018.
[41] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4405-1415, 2019.
[42] X. Peng, Z. Huang, Y. Zhu, and K. Saenko. Federated adversarial domain adaptation. arXiv preprint arXiv: Tte.:/9111.0000 1.02054, 2019.
[43] Quinonero-Candela. M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. Mit Press, 2008.
[44] O Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. International Conference on Learning Representations, 2018.
[45] R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18.
IEFF, 2017.
[46] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-600 608, 2020.
[47] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. ICLR 2019 Workshop on Al for social good, Dec. 2018.
[48] X. Wang. Y. Han, C. Wang, Q. Zhao, X. Chen, and M. Chen. In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning. IEEE Network. 3365)1156-16, 2019.
[49] Y.-. Wang, B. Balle, and S. P. Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1226-1235. PMLR, 2019.
[50] K. Weiss, T. M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big data, 3(1):1 10111-40222 2016.
[51] Y. Xu, Y. Wang. H. Chen, K. Han, C. XU, D. Tao, and C. Xu. Positive-unlaheled compression on the cloud. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[52] X. Yan, D. Acuna, and S. Fidler. Neural data server: A large-scale search engine for transfer learning data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3993-3902, 2020.
[53] J. Yao, S. Zhang, Y. Yao, F. Wang. J. Ma, J. Zhang, Y. Chu, L. Ji, K. Jia, T. Shen, A. Wu, Zhang, Z. Tan, K. Kuang, C. Wu, F. Wu, J. Zhou, and H. Yang. Edge-cloud polarization and collaboration: A comprehensive survey. arXiv:2111.0666 [es]. Nov. 2021.
[54] S. Zad, M. Heidari, P. Hajibabace, and M. Malekzadeh. A survey of deep learning methods on semantic similarity and sentence modeling. In 2021 IEEE 12th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pages 0476.0472. IEEE, 2021.
[55] Y. Zhu and Y.-.. Wang. Poission subsampled rényi differential privacy. International Conference Muchine Learning, pages 7634-7642. PMLR, 2019.
[56] Y. Zhu, X. Yu, M. Chandraker, and Y.-X. Wang. Private-knn: Practical differential privacy for computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11854-11 11862, 2020.
[57] 7. Zhu, J. Hong, S. Drew, and J. Zhou. Resilient and communication efficient learning for heterogeneous federated systems. In Proceedings of Thirt-ninth International Conference on Machine Learning (ICML 2022), 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We include the instructions but not codes for reproducing results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tZJ6TCynCrJ", "reviewer": "Reviewer_5x3V", "summary": "The work proposes to outsource model training from low-power and cost-effective end devices to powerful and cloud-based servers, where no access to the private data is given and instead a surrogate dataset is sampled from public data to obtain a proxy dataset. The sampling process was designed to be communication and computation efficient. \n\nIn the ECOS (Efficient Collaborative Open-source/world Sampling - there is a disagreement what ECOS stands for - compare lines 9 and 50) framework, the low-power clients do not send their private data but share features of their data that are used by the high-power server to adaptively find proximal public data for training the model (as a service). This facilitates the outsourcing of the model training without sharing local data by clients. \n\nIt is argued that public data have low sample efficiency in training the model that could be useful on a client's local data. This is due to the fact of including OOD (Out-Of-Distribution) samples from the public domain that degrade the accuracy on the client's data distribution. \n\n", "questions": "- It is very surprising in Table 5 that the random sampling performs better than the proximity objective. Could the proximity part from Equation 2 be further improved? \n\n- How is the privacy cost computed in Table 4. How is the privacy loss from returning the coverage scores combined with the privacy loss incurred by labeling the proximal data with Private knn?\n\n- Lack of future work or more potential methods. What are the other possible approaches to find which data the server should train the model? Wouldn't it be more practical if a client would point the server to the public data samples that should be used by the server to train the model? These can be also public datasets with labeled examples. The client could simply send links to the public data samples so the communication cost would be limited to the minimum. It seems that the best way to compare images is by applying a self-supervised model and then by comparing the representations from the private set with the representations from the public images. \n\n", "limitations": "The work assumes that there exist public data samples similar to the client's private data samples.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors tackle the important problem of outsourcing model training, which is of practical use. They incorporate the framework of differential privacy to account for the privacy leakage when a client instructs the server on the selection of the proximal dataset.\n\nWeaknesses:\n- Regarding the related work, the idea to \"generate client-approximated labels for unlabeled public data, on which a model can be trained\" was first proposed in https://arxiv.org/abs/1610.05755 and not in Private kNN.\n- The definitions of proximity and diversity are omitted but this hinders the understanding of the paper. One has to refer to the related work [19] to find out the corresponding definitions - they should be included in this paper as they are the core of the method. \n- In Tables 2 and 3, the Random method outperforms K-Center in all but 1 case. Isn't there any better baseline to compare with?\n- It is not clear how the adaptive model compression works (e.g., presented as subplot (b) in Figure 2).\n\n\nTypos:\n- Line 154: the client uses\n- Line 218: repeated three times with the following seeds {1,2,3}\n- Line 224: three practical use cases\n- Line 241: given higher budgets (not more)\n- Line 316: ECOS only needs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pIpl29nZxq", "reviewer": "Reviewer_YJH7", "summary": "This paper proposes a method to efficiently sample from open-source data in order to train models that are then transferable to private local datasets. This enables the use of machine-learning-as-a-service (MLaaS) to train models on the cloud without the need to upload confidential data to a remote server. The authors develop a new sampling method, by scoring centroids of clustered features from open-source data samples, which then guides the selection of a proper subset of the open-source dataset for model training. The uploaded scores can further be protected by adding Gaussian noise, in order to account for a desired privacy cost in the context of differential privacy (DP). This then leads to a technique with minimal computational/bandwidth requirements for the client. The proposed method is evaluated for digit and object recognition tasks, trained using pseudo, manual or pre-trained supervisions, and is shown to outperform baseline methods on average. ", "questions": "- Do the authors have any insight as to why the random baseline outperforms the authors’ method in some cases? This seems counterintuitive especially since this baseline is perfectly privacy-preserving, and by using random sampling one is polluting the training dataset with out-of-distribution samples.\n- Does the K-center baseline require uploading the client dataset (or a transformed version of it) to the server? If so, this should be clarified since Tables 2-4 suggest that it is perfectly privacy-preserving.\n- For DomainNet, roughly what portion of the selected proximal data corresponds to the same domain as the client dataset? This could show the effectiveness of the sampling method in picking in-distribution samples.\n", "limitations": "The main limitation of this method is that it requires the utilized public dataset to contain samples that are similar in distribution to the private dataset. The authors have adequately discussed this in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper addresses an important problem, especially with the growing popularity of MLaaS. The proposed method is sound and well-described. I also appreciate the analysis of the algorithm with regard to different privacy, which allows a user to fine-tune the method according to their privacy cost. In most cases, the performance of the proposed method (in Tables 2-4) is within the confidence interval of another baseline method, and therefore the performance gains are not necessarily statistically significant. Nevertheless, configurable privacy and the computational/bandwidth efficiency of this method are solid contributions.\n\nI think the evaluation can be improved in the following regards:\n- While the baselines (K-center and random) provide a useful comparison between the authors’ method and more naive approaches, it would also help to include accuracies when a model is trained on the client dataset itself. This would show how much accuracy one would lose (if any) by using a privacy-preserving proxy dataset.\n- I also think a sensitivity analysis with respect to the privacy cost (\\epsilon) could be very informative. How does changing \\epsilon affect the achieved accuracy?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kmoGC7mfTPm", "reviewer": "Reviewer_tiVh", "summary": "Facing the importance of privacy in machine learning, the paper proposes an efficient and private outsourcing strategy to find surrogate datasets for training classifiers to be adapted to private datasets. The authors identify the limitation of traditional outsourcing strategies in the iid assumption of open-source data or inefficient or non-private data sharing. In addition, they reveal the challenges of outsourcing with non-iid open-source data and therefore propose a private, efficient and compelling solution regardless of the availability of labels in the open-source datasets. Extensive and various experiments demonstrate the effectiveness and flexibility of the proposed methods in practice.", "questions": "Can the proposed method be extended to other data domains, e.g., language models?", "limitations": "No obvious ethic issues in the paper. The authors have adequately discussed the limitation of the methods, like the data processing. Though more discussions on automating the data processing could be appreciated, it is an acceptable limitation within the scope of the paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1. (Novelty) To mitigate the high cost of model training, traditional outsourcing methods tends to transfer the local training to public data that are identically distributed to the local dataset. The idea of outsourcing training without uploading reference private data is quite novel with the relaxed non-iid assumption on the public dataset and provides an interesting direction for outsourcing that is both efficient and private. \n2. (Impact) The paper is well motivated by the non-iid nature of open-source data and focuses on improving proximity, diversity and efficiency. Such practical considerations and goals make the method quite applicable in a wide range of real scenarios (with or without labels, label supervisions or model supervisions), and are promised with a high impact on privacy-preserving learning and efficient training outsourcing.\n3. (Methodology) The proposed method is intuitive and is clearly guided by the importance of proximity, diversity and efficiency in improving the quality of selected open-source data. The intuition naturally leads to the principled objectives in sampling. Later experiments also demonstrate the importance of the three components. The privacy and efficiency (in terms of communication and computation) are solidly studied by theoretical analysis.\n4. (Technical Quality) The paper is technically sound presenting significant improvements in privacy, computation and communication efficiency. \n - The experiments are extensively executed in various scenarios with different supervisions and the proposed method exhibits outstanding performance in various benchmarks. Due to the high cost of labels in practice, the authors clarifies that it is usually unrealistic to assume the availability of labels in open-source datasets. In the well-designed experiments, the authors provide comprehensive evaluations regarding the forms of training supervisions. It is shown that the method is flexible in multiple scenarios, including the sample selection for reducing the cost of manual labeling, adaptive model compression on calibrated open-source data distributions, and automated client labeling with improved accuracy. \n - In addition, the qualitative studies are quite helpful for understanding how principled objectives can lead to better performance. The visualization in Fig 3 provides another interesting perspective of the success, as well. The effects of important parameters like compression size and score scale s are well studied.\n - The privacy and efficiency of the method are evaluated not only theoretically but also empirically, which are quite solid.\n - Details of the implementation, including theorem proofs, datasets, and network architectures, are provided in detail meriting the reproducibility.\n\nWeaknesses:\n1. The experiments are limited to the domain of image recognition. The method could be considered in more general data assumptions.\n2. The measurement of privacy could be described in detail early rather than in the appendix, for the convenience of readers who are not familiar with the notion of differential privacy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6SKx5sWU8XF", "reviewer": "Reviewer_tw1P", "summary": "The authors considered utilizing public open-source data rather than sensitive private data when outsourcing the computation of machine learning under strong privacy constraints. To tackle the problem that the heterogenous public data from various domains, which may be not IID with the target-domain data and may degrade the model performance, the authors proposed the ECOS framework, which filters the public data according to the local private data.", "questions": "1. On the one hand, more data benefit training from reducing overfitting risk. On the other hand, out-of-distribution (OoD) samples degrade the model performance. The authors consider filtering public data to get rid of OoD samples, which, however, may reduce the amount of training data. Does the proposed ECOS framework strike a balance between more training data and high-quality training data, theoretically and empirically?\n\n2. Federated learning considers a similar scenario of privacy constraints. We recommend the authors to compare their setting with federated learning in the related work. \n", "limitations": "See my comments on more theoretical analysis and comparison with federated learning.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n**Originality:** The authors considered a new problem of outsourcing machine learning under privacy constraints. This proposed framework clusters the public data and distributes the cluster centroids to the clients. The clients then use the private local data to score the centroids and then use the scores to filter the heterogeneous public data. This new framework works intuitively to alleviate the problem of data heterogeneity.\n\n**Quality:** Compared with existing baselines, the proposed ECOS framework gains accuracy improvement in the evaluation. The authors also provided visualization which illustrate the effects of the ECOS framework.\n\n**Clarity:** The paper is well organized and the proposed method is clear.\n\n### Weaknesses\n\n**Quality:** The theoretical analysis is not very sufficient, e.g., about how much accuracy the data filtering process can improve (compared with using all public data).\n\n**Significance:** Federated learning considers the similar problem: users cannot upload raw data due to privacy concerns. There has been many research work in this field, which may also help address the problem considered by the authors. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "457cVo-Nrii", "reviewer": "Reviewer_PrwV", "summary": "This paper proposes a method of outsourcing client training in a privacy-preserving manner by effectively building and utilizing open-source data in the cloud. Problems with data heterogeneity is tackled by using a collaborative sampling solution. A series of experiments demonstrate the performance of this method.", "questions": "- Could ECOS be applied to a Federated Learning scenario, where multiple clients can learn from the proximal/diverse data on the cloud?", "limitations": "- No theoretical proofs on the convergence of models using ECOS.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed method appears to perform well under multiple learning scenarios\n- Arguments on privacy show that ECOS could provide a viable method for training models in a safe manner.\n- Extensive experiments demonstrate ECOS provides empirically well compared to other methods.\n\n**Weaknesses**\n- See *limitations* below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept