GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
Authors: Enmao Diao
Jie Ding
Vahid Tarokh
Venue: NeurIPS 2022 Accept (2022)
Keywords: Assisted Learning
Decentralized Machine Learning

Abstract: Collaborations among multiple organizations, such as financial institutions, medical centers, and retail markets in decentralized settings are crucial to providing improved service and performance. However, the underlying organizations may have little interest in sharing their local data, models, and objective functions. These requirements have created new challenges for multi-organization collaboration. In this work, we propose Gradient Assisted Learning (GAL), a new method for multiple organizations to assist each other in supervised learning tasks without sharing local data, models, and objective functions. In this framework, all participants collaboratively optimize the aggregate of local loss functions, and each participant autonomously builds its own model by iteratively fitting the gradients of the overarching objective function. We also provide asymptotic convergence analysis and practical case studies of GAL. Experimental studies demonstrate that GAL can achieve performance close to centralized learning when all data, models, and objective functions are fully disclosed.

GAL: Gradient Assisted Learning for Decentralized Multi-Organization Collaborations
Enmao Diao
Jie Ding School of Statistics University of Minnesota-Twiie Cities Minneapolis, MN 55455, USA Zingj@umn. edu
Department of Electrical and Computer Engineering
Duke University
Durhm, NC 27705, USA
enmao www.dduke..e....o edu
Vahid Tarokh
Department of Electrical and Computer Engineering Duke University
Durhm, NC 27705, USA
vahid.tarokheee tar.khhdukee edu
Abstract
Collaborations among multiple organizations, such as financial institutions, medical centers, and retail markets in decentralized settings are crucial to providing im- proved service and performance. However, the underlying organizations may have little interest in sharing their local data, models, and objective functions. These requirements have created new challenges for multiioorgaiizatinn collaboration. In this work, propose Gradient Assisted Learning (GAL), multiple organizations to assist each other in supervised learning tasks without sharing local data, models, and objective functions. In this framework, all par- ticipants collaboratively optimize the aggregate of local loss functions, and each participant autonomously builds its own model by iteratively fitting the gradients of the overarching objective function. We also provide asymptotic convergence analysis and practical case studies of GAL. Experimental studies demonstrate that GAL can achieve performance close to centralized learning when all data, models, and objective functions are fully disclosed. Our code is available here
method for
1 Introduction
One of the main challenges in harnessing the power of big data is the fusion knowledge from numerous decentralized organizations that may have proprietary data, models, and objective functions. Due to various ethical and regulatory constraints, it may not be feasible for decentralized organizations to centralize their data and fully collaborate to learn shared model. Thus, large-scale autonomous decentralized learning method that can avoid data, models, and objective functions transparency may be of critical interest.
As illustrated in Figure D medical institute may be helped by multiple clinical laboratories and pharmaceutical entities to improve clinical treatment and facilitate scientific research 2 Financial organizations may collaborate with universities and insurance companies to predict loan default rates (4). The organizations can match correspondence with common identifiers, such as user identification associated with the registration of different online platforms, timestamps associated with different
Cooperative learning may have various scientific and business applications [T]
Resources related Assisted Learning (AL) can be found http: httt::://www... http///www....ieeeeeeaaarraaaa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

$
Date
Proprtebery
Figure 1: Decentralized organizations form community of shared interest to provide better Machine- aaaiiin--a-a-Servie
climics and health providers, and geo-locations associated with map-related traffic and agricultural data. With the help of our framework, they form community of shared interest to provide better (MLaaS) 6] without transmitting their local data, models, and
Machine- sarning-as-a-Sert objective functions.
The main idea of Gradient Assisted Learning (GAL) is outlined below. In the training stage, the organization to be assisted, denoted by Alice, will calculate set of 'residuals' and broadcast these to other organizations. These residuals approximate the fastest direction reducing the training loss in hindsight. Subsequently, other organizations will fit the residuals using their local data, models, and objective functions and send the fitted values back to Alice Alice will then assign weights to each organization to best approximate the fastest direction of learning. Next, Alice will line-search for the optimal gradient assisted learning rate along the calculated direction of learning. The above procedure is repeated until Alice accomplishes sufficient learning. In the inference stage, other organizations will send their locally predicted values to Alice who will then assemble them to generate the final prediction. We show that the number of assistance rounds needed attain the centralized performance is often small (e.g., fewer than ten). This is appealing since GAL is primarily developed for large organizations with rich computation resources. A small number of interactions will reduce communication and networking costs. Our main contributions are summarized below.
We propose a Gradient Assisted Learning (GAL) algorithm that is suitable for large-scale autonomous decentralized learning. It can effectively exploit task-relevant information preserved by vertically decentralized organizations. Our method enables simultancous collaboration among organizations without sharing data, models, and objective functions.
We provide asymptotic convergence analysis and practical case studies of GAL. For the case vertically distributed data, GAL generalizes the classical Gradient Boosting algorithm.
Our proposed method significantly outperform learning baselines and achieve near- oracle performance on various benchmark datasets. Compared with existing works, GAL does not need frequent synchronization of organizations. also significantly reduces the computation and communication overhead.
2
Related work
Multimodal Data Fusion Vertically distributed data can be viewed as multimodal data with modali- ties provided in a distributed manner to different pearnes/organiizatios.. Standard multimodal data fusion methods include the early, intermediate, and late data fusions (78). These methods concatenate different modes of data at the input, intermediate representation, and final prediction levels. However, these data fusion methods in decentralized settings often require organizations to share the task labels to train their local models synchronously. In contrast, our method presented below only requires that organizations asynchronously some task-related statisties named pseudo-residuals approximate the direction of reducing the global training loss in hindsight.

---

Gradient Boosting Our approach was inspired by Gradient Boosting sequentially trained from the same dataset and aggregated into strong earner. In our learning context, each organization uses side information from heterogeneous data sources to improve a particular learner's performance. Our method can be regarded as generalization of Gradient Boosting to address decentralized learning with vertically distributed data.
100,
where weak learners are
Federated Learning Federated learning [1+116 is a popular distributed learning framework. Its main idea is to learn joint model by averaging locally learned model parameters. It avoids the need for the transmission of local training data. Conceptually, the goal of Federated Learning is to exploit the resources of edge devices with communication efficiency. Vertical Federated Learning methods split sub-networks for local clients to jointly optimize a global model [17)22]. These methods can be viewed as federated learning with an intermediate data fusion method, and the central server will have access to the true labels. In order to converge, these methods typically require very frequent batchwise synchronization of backward gradients (22]. Frequent batchwise synchronization critical for vertical federated learning because the local model at each client constitutes a part of the globally backpropable model, and one client's 's local update may not decrease the overall loss. In contrast, our proposed method trains multiple autonomous local models with pseudo-residuals, each contributing to a small portion of the overarching loss. Each round of updates will decrease the loss. Consequently, our method can achieve desirable performance with significantly fewer communication rounds without global backpropable model.
Assisted Learning Assisted Learning (AL) (23) a decentralized collaborative learning framework for organizations to improve their learning quality. In that context, neither the organization being as- sisted nor the assisting organizations share their local models and data. The original AL methodology applies to regression tasks. It is derived from a linear projection perspective, and its convergence to the oracle performance was theoretically justified for linear regression models with quadratic loss.
Inspired by Gradient Boosting, the proposed Gradient Assisted Learning (GAL) is a general method for multiple organizations to assist each other in supervised learning scenarios. Overall, AL and GAL share similar motivations and concepts but significantly differ from methodological and theoretical perspectives. More specifically, the novelties of GAL include: 1) generalization from regression loss to any differentiable loss for supervised learning, 2) allowing for local objective functions at each organization, 3) generalization from a sequential protocol between two organizations to paraliel aggregation across multiple organizations, 4) introduction of deep model sharing for reducing computation and memory costs, 5) introduction of the assisted learning rate for fast convergence, and 6) theoretical analysis of GAL's convergence properties.
3 Gradient Assisted Learning
3.1 Notation
Suppose that there are N data observations independently drawn from joint distribution Pay =
PaPylas where y E ) and x E Rd respectively represent the task label and feature variables, and is the number of features. For regression tasks, we have y R. For K-class classification tasks, y = =ei,.... ) where ej is the canonical vector representing the class k, k = L...., K.
Let E and Ex denote the expectation and empirical expectation, respectively. Thus, ENg x) e -i= ,'' (V,, 2².) for any measurable function g. where (y,, are i.i.d. observations from Pay Suppose that there are M organizations. Each organization m only holds Xm, a sub-vector of X (illustrated in F). In general, we assume that the variables in X X..... XM are disjoint in the presentation of our algorithm, although our method also allows the sharing of some variables. For example, one organization may observe demographic features for a mobile user cohort, and another organization holds health-related features of that cohort. Without loss of generality, we suppose that Alice, the organization be assisted, has local data 21 and task label y1. while other M 1 organizations are collaborators which assist Alice and have local data Z2 2 M- We use 1 m and 1 / to represent from the first to the th organization and the 4th assistance round, respectively.
3.2 Problem
For m = 11 M, let (fim J.V. denote the available data to the organization Thus, N objects are simultaneously observed by M organizations, each observing a subset of features from the E Rd. Alice also has local task labels (yr1 J.V for training purposes.
3

---

features
N samples
Xi
X
XM
Figure 2: Illustration of organizations' vertically distributed data.
Let Fm and Lm respectively denote supervised function class (such as generalized linear functions or neural networks) and the local objective function of organization m. We will assume that Lm. is differentiable. Without loss of generality, we assume that Alice denotes organization 1, who will be assisted. Without assistance from other organizations, Alice would learn model that minimizes the following empirical risk,
F Alooee argmin E,y L1 (1())))
(1)
FIFFi
Note that the above formulation only involves Alice's local data and local model (as represented by Fi and L1). an oracle case, Alice would be able to operate on other organizations' data 422, JM as well. Recall that represents the ensemble of all the available data variables. In general, the ideal case for Alice is to minimize the following empirical risk,
Foint = argmin Li
PEF
where F is a supervised function class defined with the space of x as its domain.
In reality, Alice has access to the complete data and model resources of other organizations. In this light, she shall be happy to crowdsource the learning task to other organizations to cooperatively build model in hindsight, without the need to share any organization's local data models. In the prediction stage, Alice can collect the information needed to form a final prediction to hopefully achieve performance that significantly improves over her single-organization performance. To this end, we will develop a solution for Alice to achieve such a goal.
We will include detailed derivations and discussions of our solution in Subsection 3.3 For readability, we summarize notations that will be frequently used in the exposition below. Our method will require Alice to occasionally send continuous-valued vectors Tı [ra,) E R to each organization m each communication round. These residuals, to be elaborated in the next subsection, approximate the fastest direction of reducing the training loss in hindsight, namely sample version of 0L, (y1, F(x))/0F(zz given Alice's 's estimation F at a particular time (round). Upon the input of these residual vectors, the organization m will locally learn supervised function fm that maps from its feature space to the residual space. With slight abuse of notation, we also refer to fm as the learned model. To this end, the organization will perform the empirical risk minimization
fm = argmin E (1 ((x)) = argmin JEFm
fm (fr,,, (F),
JEF N
to obtain a locally trained model fm. Here, Fm and lm respectively denote the supervised functioon class and loss function of the organization We note that Lm are local regression loss functions for fitting the pseudo-residual "1 and may not necessarily be the same as L2 for fitting true labels. For example, L1 may be the cross-entropy loss for classification of label y. while f1MM could be the squared loss for regression the response r1- The above local training (optimization) is often performed using the stochastic gradient descent (SGD) algorithm. In our assisted learning context, Fi. and fm are proprietary local resources that cannot be shared across organizations.
3.3 The GAL Algorithm
We first introduce derivation of the GAL algorithm from functional gradient descent perspective. Then, we cast the algorithm into pseudocode and discuss each step. Consider the unrealistic case that Alice has all the data x needed for centralized supervised function a +> F(x). Recall that the goal of Alice is to minimize the population loss Ep.., L1 (y. F(x)) over data distribution Px,y- If

---

Algorithm GAL: Gradient Assisted Learning (from the perspective of the service receiver, Alice)
Input: M decentralized organizations, each holding data { J.V. (local) corresponding to N objects, the task label {3 initially held by the service receiver (Alice local), model class Fm (local), gradient assistance weights w (Alice local), assistance rate 7) (Alice local), overarching loss function L1 (Alice local), regression loss function fm to fit pscudo-residual (local), assistance rounds T.
Learning Stage:
Intialization:
Let t = 0, and initialize F (z) (v))
for assistance round from 1 to T do Compute pseudo-residual
(T (xx)
OF OFI-(())
Broadcast pseudo-residual rf other organizations
for organization m from to M in parallel do fm argmin EN m (ri, (Tm))
end
Gather predictions fm (Im), m I.... M, from all the organizations Optimize the gradient assistance weights
w argmin Par Ex l (ri. E.E wmJm (Im))
Line-search for the gradient assisted learning rate
ij' = argmin VER Ex L1 (1, Fx-((x) + E w In (Fm)
F F()) (z)
F (x) ftt EM wi I (Im)
end
Prediction Stage:
For each data observation x², of which xn, held by organization m: Gather predictions ft, (Im)), 1..... from each organization m, m 1....,A ......M M Predict with
F (xx)
F (zt) ΣL 1 EM
M th
Pa.y is known, starting with an initial guess (x)), Alice would have performed a gradient descent step in the form of
F + 0 OF Bpe. LI F(x)) 1===== -1 Ep.. OF L1 (v. F(x)) Ir-ro, (4)
where the equality holds under the standard regularity conditions of exchanging integration and differentiation. Note that the second term in (4) function on R.d. However, because Alice only has access to her own data T1, the expectation Ep.., cannot be realistically evaluated. Therefore. need approximate it with functions in a pre-specified function set. In other words, will à find f from Fas that 'best' approximates Ep... OF L1 (y, F(x)). We will show that this is actionable without requiring the organizations to share proprictary data, models, and objective functions.
Recall that Fm is the function set locally used by the organization m, and I' is a correspondingly observed portion of x. The function class that we propose to approximate the second term in is
FM = ( Σ
wm i (2m). Vfm E Fm, x Rdd w PM
(5)
where PM (w E R
> Em=1 Wm 1,, w. 2 0} denotes the probability simplex. The gradient assistance weights Wm 's are interpreted as the contributions of each organization at a particular greedy update step. The gradient assistance weights are constrained to sum to to ensure the function space is compact and the solutions exist.
M
We propose the following solution so that each organization can operate on its own local data, model, and objective function. Alice initializes with a startup model, denoted by Fx) F (21, 31), based only her local data and labels. Alice broadcasts rı (named 'pseudo residuals') to each organization
5

---

Uter
Ped
Sencc
E4 Booold
Aspadaa
Learinn Stage
(6) Predictin Bluge
Figure 3: Learning and Prediction Stages for Gradient Assisted Learning (GAL).
m, = 2, ................................ M, who will then fit a local model fm using r1 Each organization will then send the fitted values from fm to Alice, who will train suitable gradient assistance weights wm- Subsequently, Alice finds the 7 in 4 that minimizes her current empirical risk. The above procedure is iterated for a finite number of rounds until Alice obtains a satisfactory performance (c.g., on validation data). The validation will be based the same technique the prediction stage to be described below. This training stage is described under the 'learning stage' of Algorithm I Note that the pseudocode from the perspective of Alice, the service receiver. Each organization will only need to perform the empirical risk minimization using the label rf sent by Alice at each round t.
In the Prediction/Ifferece stage (given above in Algorithm [], other organizations send prediction results generated from their local models to Alice, who will calculate prediction result F (a:) that is implicitly operated on z, where T is the number of iteration steps.
The idea of approximating functional derivatives with regularized functions was historically used to develop the seminal work of gradient boosting (9 10]. The above method reduces to the standard gradient boosting algorithm when there is only one organization.
Organizations in our learning framework form a shared community of interest. Each service- providing organization can provide end-to-end assistance for organization without sharing anyone's proprietary data, models, and objective functions. In practice, the participating organizations may receive financial rewards from the
assist. Moreover, every organization in this framework can provide its own task and seek help from others. As a result, all organizations become mutually beneficial each other. We provide a realistic example in Figure to to demonstrate each step of Algorithm We elaborate on the learning and prediction procedures in the Appendix.
We also provide asymptotic convergence analysis for the GAL algorithm, where the goal is to minimize a loss f + L(f) over a function class through step-wise function aggregations. Because of the greedy nature of GAL, consider the function class to be the linear span of organization -speciffonns Fm. The following result states that the GAL can produce a solution that attains the infimum L(f). More technical details are included in the Appendix.
Theorem Assume that the loss (functional) f - L(J) is convex and differentiable on F, the function u r> LiF ug) has an upper-bounded second-order derivative PL + ugg//nu² for all f E span(Fi..... [........ FM.) and g UM U and the ranges of learning rates {at 1t1,2.... satisfy Ee a 0G, Ee of i fEspan(F [(f) as co, with convergence rate at the order of (01=/a1:)))))
Then, the CAL algorithm satisfies L(Ft)
Experimental Studies
Baselines Our experiments are performed with four baselines, including 'Interm' 'Late", 'Joint', 'Alone', and 'AL'. 'Interm' and 'Late" refer to intermediate and late data fusions ( respectively. The intermediate data fusion ( ('Interm') sums up the intermediate features (output before the last layer) from each separate feature extractor, and then the aggregated feature is passed into shared last layer to output the final prediction. The late data fusion ('Late') sums up the final prediction of each separate local model. Thus, 'Interm' works only for deep learning models such as CNN and L.M by averaging the hidden representation of each local model, while 'Latee also works for Linear models as it aggregates the output of each local model. 'Joint' is the case where all the

---

data are held by Alice and trained with the Gradient Boosting reduced from GAL. 'Alone' is the single-agent scenario, where only Alice's 's data are used for learning and prediction. 'AL' represents the performance of Assisted Learning [23]. GAL is expected to perform close to the centralized baselines, including 'Interm', 'Late', and "Joint' cases, while significantly outperforming the 'Alone' and 'AL' cases. The summary statistics of each dataset are elaborated in Table the Appendix. Details of learning hyper-parameter are included Table the Appendix. We conducted four random experiments for all datasets with different seeds, and the standard errors are shown in the brackets of all tables.
4.1 Model Autonomy
Recall that GAL allows each organization choose its own local model. We demonstrate the performance autonomous local models with UCI datasets downloadable from the scikit-learn
package (24), including Diabetes [25], Boston Housing [26). Blob (24), Iris [27]. Wine (28). Breast
Cancer [29]. and QSAR (30] datasets, where we randomly partition the features into 2, 4, or 8 subsets. For all the UCI datasets, train 80% of the available data and test the remaining.
Table I: Results of the UCI datasets (M 8) with Linear, GB, SVM and GB-SVM models. The Diabetes and Boston Housing (regression) are evaluated with Mean Absolute Deviation (MAD), and the rest (classification) are evaluated with Accuracy.
Dataset
Model
Diabetes(4)
BostonHousing(
Blob(t)
Wine(t)
BreustCancer(t)
QSAR(t)
Late
Linear
136.2(0.1)
8.0(0.0)
100.0(0.0)
100.0(0.0)
96,9(0.4)
76,9(0.8)
Joint
Linear
43.4(0.3)
3.0(0.0)
100.000.0)
(00.0(0.0)
98.9(0.4)
84.0(0.2)
Alone
Linear
59.7(9.2)
5.8(0.9)
41.3(10.8)
63.9(15.6)
92.5(3.4)
68.8(3.4)
AL
Lincar
51.5(4.6)
4.7(0.6)
97.5(2.5)
95.1(3.6)
97.7(1.1)
70.6(5.2)
GAL
Lincar
42.7(0.6)
3,2(0.2)
100.0(0.0)
96.5(3.0)
98.5(0.7)
82.5(0.8)
GAL
GB
56.5(2.8)
3.8(0.5)
96.3(2.2)
95,8(1.4)
96.1(1.0)
84.8(0.9)
GAL
SVM
46.6(1.4)
2.9(0.2)
96.3(4.1)
96.5(1.2)
99.1(1.1)
85.5(0.7)
GAL
GB-SVM
49.8(2.6)
3.4(0.8)
70.0(7.9)
95.8(1.4)
93.2(1.6)
82.9(1.5)
We experiment with Linear, Gradient Boosting (GB), and Support Vector Machine (SVM) for local models [m(() with the UCI datasets. We also demonstrate the performance the scenario (GB-SVM) where half of the organizations use GB and the other half uses SVM. The experimental results are shown in Tables [ indicates the smaller the better, while 1 indicates the larger the better. Our method significantly outperforms the baselines 'Alone' and 'AL.' The results also demonstrate that with GAL, an organization with little informative data and free choice of its local model (model autonomy) can leverage others' local data and models and even achieve near-oracle performance. We point out that although 'Interm,' 'Late,' and 'Joint' marginally outperform our method, they require training from centralized data. Our GAL algorithm replaces the true labels used in 'Interm,' 'Late,' and 'Joint' centralized cases with pseudo-residuee The results from both regression and classification datasets with various model settings lead to similar conclusions.
4.2 Deep Model Sharing
We demonstrate that our method is effective for deep models by using MNIST (11 and CIFARIO (32) image datasets, where we split each image into patches as depicted in Figure 6 We use Convolutional Neural Networks (CNN) for both datasets, and the model architecture can be found in Table of the Appendix. We visualize the performance of CIFARIO each assistance round in Figure (a-c). The number of assistance rounds needed to ap- proach the centralized performance small (e.g., often within ten). The experimental results are shown in Tables 2 GAL significantly outperforms the bottom line 'Alone' in all the settings. This is
Table 2: Results of the MNIST
and CIFARIO (M = 8) datasets with CNN model. The MNIST and CIFAR10 are evaluated with Accuracy. GAL-DMS represents the results with Deep Model Sharing. More results for M 2 and 4 are in the Appendix.
Dataset
MNIST(1)
CIFARIO(1)
Interm
98.8(0.1)
78.2(0.2)
expected since the first organization holds partial data and does not receive any assistance under 'Alone.' Interestingly, the per- formance of MNIST for M = 8 drops significantly under 'Alone' because the organization only holds the left upper image patch, which is usually completely dark, as shown in Figure 0
Late Joint
98.0(0.1)
74.4(0.3)
99.4(0.0)
80.1(0.2)
Alone AL
24.2(0.1)
46.3(0.3)
34.3(0.1)
51.1(0.)
GAL
96.3(0.6)
74.3(0.2)
GALoMS
96.3(0.5)
67.0(0.3)
7

---

Because local deep learning models such as CNN can consume extensive computation space, we propose Deep Model Sharing (DMS) allow sharing feature extractors of deep models across all iterations to save memory. In particular, we propose to jointly train the feature extractors with residuals from previous iterations as well as from the current iteration by adding an additional last prediction layer. The local deep model J (-) is composed of J ('..(fm.e(-)) where f' is the last output layer at assistance round 1 and fm,e(.) is the deep feature extractor shared across multiple assistance rounds. For each assistance round, local organizations fits pseudo-residuals across previous assistance rounds rf with fH argmin, (ms E.Fm Ex (ri:t. Smjo (mm.e((.m)) It worth mentioning that we do not expect such trade-off GALDMS to consistently outperform AL and GAL because single feature extractor may not well fit residuals across many iterations. The results in Table 2 show that sharing the feature extractor across multiple assistance rounds can still outperform the 'Alonc' case. Thus, DMS can provide trade-off between predictive performance and computation space. The detailed comparisons can be found in Table 14
4.3 Comparison with AL
As demonstrated in our extensive experiments, the proposed GAL outperforms AL in terms of predictive performance. In particular, AL converges not only worse but also slower than GAL. This due to the fact that AL uses a constant assisted learning rate and trains participating organizations in sequential manner. Moreover. sequentially training participating organizations also requires much more computation and communication overhead. We compare the computation and commu- nication complexity between AL and GAL under the constraint of the same communication cost as demonstrated in Table [4] Because AL sequentially trains each organization while GAL allows organizations to train locally in parallel, the computation time and communication round of AL is M x those GAL. The GAL with Deep Model Sharing (GALDMs) saves T'x computation space by sharing the feature extractor of deep models. In summary, GAL generalizes the problem scope, reduces the computation and communication complexity, and achieves significantly better results.
50
Gredent aroured marming 1906
joirt alone
Goudlent asietance wsigle
0.225
Accurayy
-0
60
70
AL
GAU// 2)
0.200
10
30
0.125
0.300
m
30
20
0.100
30
GALin -
0.075
10
5
9
10
10
Anoistancg gunds
(5)
Assistance rounds
Ascedanig courds
130
Joint
AL
Mone
GAUL 10
Goddcc 3555ted ko 0802
250
200
Joint
GALin 1:
Alore
Gordeen wrwiace weggen
0.
1.0
GALL 60
GALLi a
9.6
210
100
50-
0.0
5
9
10
4
6
10
8
10
Apodelance [e)
Asodetancc rourdds
(40
(0)
Figure 4: Results of the CIFARIO (a-c) (M = 8) and MIMICL (d-f) (M = 4) datasets. GAL significantly outperforms 'Alone' and 'AL', Our method also performs close to the centralized baselines. The gradient assisted learning rate diminishes to zero as the overarching loss converges. constant gradient assisted learning rate (17 = 1) converges much slower. The gradient assistance weights exhibits interpretability of the importance of organizations as the weights of the central image patches (m f2, 6, 7)) of CIFAR 10 dataset are larger than the boundary patches (m {1, 4, 8)) in the first few rounds. More results can be found in the Appendix.
4.4 Case Studies
The results in Table 3 demonstrate the utility of GAL in various practical applications. We illustrate the results across multiple assistance rounds of MIMICL in Figure X (d-f). GAL significantly outperforms 'Alone' and 'AL.' Our method also performs close to the centralized baselines.
8

---

Table 3: Results of case studies of 3D object recognition and medical time series forecasting.
Dataset
ModeINes40(1)
ShapeNes55(1)
MIMICL(4)
MIMIC((1)
Interm
75.3(18.2)
88.6(0.1) (
64.6(0.9)
0.90(0.0)
Late
86.6(0.2)
88.4(0.1)
71.4(0.2)
0.9((0.0)
Joint
46.3(1.4)
16.3(0.0)
91.1(0.7)
0.82(0.0)
Alone
76.4(1.1)
81.3(0.6)
106.1
0.78(0.0)
AL
77.3(2.8)
83.8(0.0)
119.3(0.3)
0.86(0.0)
GAL
83.010.2)
84.1(0.6)
91.9(2.3)
0.88(0.0)
GALass
83.2(0.3)
85.3(0.2)
97.7(2.9)
0.81(0.0)
340
1.
325
GALAN n 100 2-
0.23
CALI 1001 5
6.10
110
CALIn dw
CALi 0.03
10
Astiitanca
AS0003000
Assetares rounds
110
(C
(d)
Figure 5: Ablation study results CIFARIO (a-b) (M = 8) and MIMICL (c-d) (M = datasets. Plots (a,c) show that the GAL equipped with gradient assistance weight significantly outperforms the GAL with direct average under noise injections (A(0, o²). o {1, 5)) to the transmitted pseudo-residual to half of the organizations during learning and prediction. Plots (b,d) show the gradient assistance weight of noisy (in orange, 1) and noise-free organizations (in red).
dinre-dimmniionall object recognition We use the shape representation of 3D objects for recogni-
tion from collection of rendered views on 2D images. We generate M = 12 2D camera views of ModelNet40 (33) and ShapeNet55 (34) datasets following [35]. Cameras are treated as decentralized learners in experiments. We adopt the same CNN architecture used for MNIST and CIFAR10 datasets. is worth mentioning that 'Joint' of ModelNet40 and ShapeNet55 performs considerably worse than other baselines because 'Joint" uses single CNN feature extractor to process all images
from twelve angles (35]
Medical time series forecasting We use the in-hospital dataset MIMIC3 [36]. where the task aims to predict the Length-of (MIMICL) and Mortality rate (MIMICM) of the ICU stays of patients. We follow the benchmark work (37 to process raw data and split the features into four organizations, including 1) microbiology measurement, 2) demographic information, 3) body measurement, and 4) International Classification of Diseases (ICD). We use MAD to evaluate the result of MIMICL (regression) and the Area Under the Surve-Receiver Operating Characteristics (AUROC) to evaluate the result of MIMICM (imbalanced binary classification). Our backbone model (LSTM) is the same as the one used in the benchmark work (38). is worth noting that the data features among the four organizations aligned with time stamps which is natural identifier for time series forecasting.
4.5 Ablation Studies
Local objective functions Our method does not require local regression loss functions e,. to be shared with other organizations, but they are supposed be beneficial for assisting Alice. We conduct ablation studies on the choice of various local regression loss functions. In this study shown in Table 4 we use different regression functions E 9) ly - : where q E (1, 1.5, 2, 4). (eg) : lgg) allows half of the organizations use by while the other half to use 442- The results show that the classification task generally works better with q > 1.
Table 4: Ablation study on the local objective function (M = 8). More results are in the Appendix.
Dataset
Diabetes(J)
BostonHousing(L)
Blob(1)
Wine(t)
BreastCancer(1)
QSAR(1)
MNIST(()
CIFARIO(")
Alone f
59.7(9.2)
5.8(0.9)
41.3(0.0)
63.9(0.0)
92.5(0.4)
68.8(0.2)
24.2(0.1)
46.3(0.3)
42.7(0.6)
3.2(0.2)
42.5(12.5)
95.1(4.1)
97.4(1.1)
64.3(3.6)
90.5(1.9)
27.8(1.7)
43.4(1.))
29(0..)
100.0(0.0)
95.8(4.2)
97.4(0.6)
80.2(1.3)
94.5(0.1)
70.2(0.))
E2
44.((1.))
3.0(0.1)
100.0(0.0)
96.5(3.0)
98.5(0.7)
82,5(0.8)
96.3(0.6)
74.3(0.2)
E
45.8(1.3)
3.2(0.2)
97.5(4.3)
98.6(1.4)
99.1(0.9)
81.3(0.7)
98.1(0.1)
73.2(0.3)
(2).f2)
43.3(1.5)
3.2(0.))
100.0(0.0)
96.5(3.0)
96.7(0.7)
80.0(1.))
94.6(1.)
65.000.)

---

Privacy enhancement Our learning framework does not require organizations to share local data, models, and objective functions. One potential limitation of our approach is that assisting organiza- tions may infer Alice's information based on the shared pseudo-residuals. Therefore, we suggest to further enhancing privacy by adopting the framework of Differential Privacy (DP) (39) or Interval
Privacy (IP) [40]. We use the Laplace mechanism with a = 1 for DP and set the number of intervals of IP to be 1. We add a moderate amount of noise to the resudulresiduals in hindsight. In Tables we demonstrate that privacy-enhance
GAL can still outperform the 'Alone" case.
Table 5: Ablation study on the privacy enhancement (maximal M). GALDP and GALp represent privacy-enhanced by DP and IP, respectively.
Diriser 11
Weeriti
QUARIT
MINIST
CIRARIIT:
StepeNee5:013
MIMICE;1)
MIMDCM:
None
2)
5
639115.0
02.
$1.30.0
52.311.0
51.30008)
50.51135
56.80.7
0.5km.
GALp
$1.500.5
4.00.0
100000.00
05.8
96 10.50
96, 4..5
69.30.0)
59800.71
58025
95.54.99
0.5%0.0
Gradient assisted learning rate We illustrate the gradient assisted learning rate of CIFARIO and MIMICL datasets at each assistance round in Figure/4tb.e We perform a line search for the gradient assisted learning rate with the Limited-Memory BFGS optimizer, which improves the convergence rates compared with SGD and Adam. We conduct ablation study using a constant gradient assisted learning rate (17 = 1). As shown in Figure (a,d), the constant gradient assisted learning rate leads to convergence much slower than the line-search method. Fast convergence is desirable since the computation and communication cost increases with the number of assistance rounds. To determine the maximal number of assistance rounds T for the service receiver, until the gradient assisted learning rate becomes small. When the gradient assistance rate is low, as shown in Figure 4 (e), the overarching loss converges to zero. In this light, an organization may stop receiving assisted learning when the gradient assisted learning rate is below a threshold.
the GAL procedure
Gradient assistance weights We show the gradient assistance weights of CIFARIO 10 and MIMICL datasets at each assistance round in Figure (c..). The results of MNIST and CIFAR 10 show that the gradient assistance weights exhibit interpretability of the importance of organizations because the image patches with dominant contributions are m = (2, 3, 6, 7) (colored in red). These image patches correspond to the center of the original image, which matches weights converge to uniform as the residuals of later iterations are small. We also conduct an ablation study of the gradient assistance weights in Figure by adding noises (Gaussian with zero mean and o² variance, a E (1, 5)) to the output of a randomly chosen half of the clients during learning and prediction. Adding noise simulates realistic scenarios where some assisting organizations may be noninformative or inject noise. We summarize the ablation study results in T The results show that the GAL with gradient assistance weights more robust than the GAL with direct average. Table 6: Ablation study (maximal M) of gradient assistance weights by adding noises to the predicted outputs from half of the organizations.
intuition appealingly. The
Norio
Wi
QSAR(T)
MNISTET)
CIFABIITT
MacztNeita
ShspeNet5()
MINIC141)
MIMICNI
40/011.0
4-10-21
46.318-31
23.21330
15.133.4
4104111
13.341..0
61.60030
(L34/24)
$8.94.46
(86.1/9.0
12:91255
$1.36
61 910-)
11.80-3)
21-00155 5)
36/60-19
121.5000
0.53040
5 Conclusion
We proposed Gradient Assisted Learning, decentralized learning method for multiple organizations to collaborate without sharing data, models, and objective functions. The proposed method can significantly outperform the local learning baselines and achieve near-oracle performance as if data were centralized on various datasets. All participants form a shared community of interest by autonomously building their own model and iteratively fitting the gradients of the overarching objective function. We also demonstrate asymptotic convergence analysis and practical case studies of GAL. Moreover, this is achieved without any constraints on the models selected by the collaborating organizations.
Acknowledgments
The work of Enmao Diao and Vahid Tarokh was supported by the Office of Naval Research (ONR) under grant number 100014-18-1-2244. The work of Jie Ding was supported by the National Science Foundation (NSF) under grant number ECCS-2038603.
10

---

References
[1] Rodrigo Roman, Jianying Zhou, and Javier Lopez. On the features and challenges of security and privacy in distributed internet of things. Comput. Netw, 57(10):2266-22 2279, 2013.
[2] John T Farrar, Andrea B Troxel, Kevin Haynes, Ian Gilron, Robert D Kerns, Nathaniel P Katz, Bob Rappaport, Michael C Rowbotham, Ann M Tiemey, Dennis C Turk, al. Effect of variability in the 7-day baseline pain diary on the assay sensitivity of neuropathic pain randomized clinical trials: acttion study. Pain@, 155(8):6622-163) 2014.
[3] Bernard Lo. Sharing clinical trial data: maximizing benefits, minimizing risk. JAMA, 313(8):793-794, 2015.
[4] Lin Zhu, Dafeng Qiu, Daji Ergu, Cai Ying, and Kuiyi Liu. A study on predicting loan default based on the random forest algorithm. Procedia Computer Science, 162:503-513, 2019. 2019
[5] Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. Mlaas: Machine learning as service. In Proc. ICMLA, pages 896-902. IEEE, 2015.
[6] Xinran Wang. Yu Xiang, Jun Gao, and Jie Ding. Information laundering for model privacy. Proc. ICLR, 2021.
[7] Bahador Khaleghi, Alaa Khamis, Fakhreddine O Karray, and Saiedeh N Razavi. Multisensor data fusion: A review of the state-of-the-art. Information fusion, 14(1):28-44, 2013.
[8] Dana Lahat, Tülay Adali, and Christian Jutten. Multimodal data fusion: overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9): 144--4477, 2015.
[9] Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean. Boosting algorithms as gradient descent in function space. 1999.
[10] Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Ann. Stat., pages 1189-1232, 2001.
[11] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proc. CCS, pages
1310-1321, 2015.
[12] Jakub Konecny, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 2016.
[13] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicationn-effiieenn learning of deep networks from decentralized data. In Proc. AISTATS, pages 1273-1282, 2017.
[14] Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication effi- cient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2021.
[15] Enmao Diao, Jie Ding, and Vahid Tarokh. SemiFL: Communication efficient semi-supervised federated learning with unlabeled clients. arXiv preprint arXiv:2 004.01 2021.
[16] Jie Ding, Eric Tramel, Anit Kumar Sahu, Shuang Wu, Salman Avestimehr, and Tao Zhang. Federated learning challenges and opportunities: An outlook. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8752-8756. IEEE, 2022.
[17] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learn- ing for health: Distributed deep learning without sharing raw patient data. arXiv preprint 1tfi::181..00S. 2018
(18) Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. in Proc. TIST, 10(2):12, 2019.
[19] Yang Liu, Yan Kang, Xinwei Zhang, Liping Li. Yong Cheng, Tianjian Chen, Mingyi Hong, and Qiang Yang. A communication efficient vertical federated learning framework. arXiv preprint 1912: / / 187, 2019.
[20] Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication- efficient algorithm for federated learning. In Proc. ICML, pages 3973-3983, 2020.
[21] Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, and Seyit Camtepe. Splitfed: When federated learning meets split learning, arXiv preprint arXiv:2004. 2020.
11

---

[22] Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. Vafl: method of vertical asynchronous federated learning, arXiv preprint arXiv:2007. 06081, 2020.
[23] Xun Xian, Xinran Wang, Jie Ding, and Reza Ghanadan. Assisted learning: A framework for multi-organization learning. Advances in Neural Information Processing Systems, 33, 2020.
[24]
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-288 2830, 2011.
[25] Bradley Efron, Trevor Hastie, lain Johnstone, Robert Tibshirani, et al. Least angle regression. Ann. Stat., 32(2):007-499, 2004.
[26] David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. J. Environ. Econ, Manag., 5(1):81-102, 1978.
[27] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 172)1179-188, 1936.
[28] Stefan Aeberhard, Danny Coomans, and Olivier De Vel. Comparative analysis of statistical pattern recognition methods in high dimensional settings. Pattern Recognition, 27(8): 1065- 1077, 1994.
[29] W Nick Street, William H Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization, volume 1905, pages 861-870 International Society for Optics and Photonics, 1993.
[30] Kamel Mansouri, Tine Ringsted, Davide Ballabio, Roberto Todeschini, and Viviana Consonni. Quantitative structure-activigy relationship models for ready biodegradability of chemicals. J. Chem. Inf. Model., 53(4)8677-87,, 2013.
[31] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2778-2324, 1998.
[32] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[33] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[34] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An information-Rica 3D Model Repository. arXiv preprint 112.00251. 2015.
[35] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945-953,77 2015.
[36] Alistair EW Johnson, Tom J Pollard, Lu Shen, Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Sci. Data, 3:160035, 2016.
[37] Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. J. Biomed. Inform., 13:112-134, 2018
[38] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Sci. Data, 6(1):1-18, 2019.
[39] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211-40) 2014.
[40] Jie Ding and Bangjun Ding. Interval privacy: A privacy-preserving framework for data collec- tion. IEEE Transactions on Signal Processing, 2022.
[41] Christian Szegedy, Wojciech 7aremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
[42] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 1rXiv:1412.6572, 2014.
12

---

[43] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Anan-
thram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
[44] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39-57. IEEE, 2017.
[45] Seyed-Mohser Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Univer- sal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
[46] Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference Machine Learning, pages 6083-6093.
PMLR, 2020.
[47] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprini 1eXiv:/708,06755 2017.
[48] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprini arXiv: 2017.
[49] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks.
arXiv preprint arXiv: 1912.0:912.0220 192.00771 2019..
[50] Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor em- bedding in convolutional neural network models via invisible perturbation. arXiv preprint a80i 10307, 2018.
[51] Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14443-14452, 2020.
[52] Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint urXiv:2009.022101 2020.
[53] Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight perturbations inject neural backdoors. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2029-2032, 2020.
[54] Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. Advances in Neural Information Processing Systems, 34, 2021.
[55] Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 11966-11999 11966, 2021.
[56] Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. arXiv preprint arXiv:2007.0877. 2020.
[57] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv: 6389, 2012.
[58] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[59] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nota-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP), pages 19-35. IEEE, 2018
[60] Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger. and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 202/ ACM SIGSAC Conference on Computer and Communications Security, pages 3104-3122 2021.
[61] Maurice Weber, Xiaojun Xu, Bojan Karlavs, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. arXiv preprint arXiv:2003.0 009044 2020.
[62] Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoungshick Kim. Backdoor attacks and countermeasures on deep learning: A comprchensive review. arXiv preprint 2007::00077.07000 2020.
[63] Florian Tramer, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In Proc. USENIX Security, pages 601-618, 2016.
13

---

[64]
Yi Shi. Yalin Sagduyu, and Alexander Grushin. How to steal machine learning classifier with deep learning. In Proc. HST. pages 1-5. IEEE, 2017.
[65] Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan. Exploring connections between active learning and model extraction. In Proc. USENIX Security, pages 1309-1326, 2020.
[66] Xun Xian, Xinran Wang. Mingyi Hong, Jie Ding, and Reza Ghanadan. Imitation privacy. arXiv preprint arXiv:2009.00444 2020.
[67] Xinran Wang. Yu Xiang, Jun Gao, and Jie Ding. Information laundering for model privacy, In International Conference on Learning Representations, 2021.
[68] Xun Xian, Mingyi Hong, and Jie Ding. A framework for understanding model extraction attack and defense. arXiv preprint arXiv:2206. 1480, 2022.
[69] Tong Zhang and Bin Yu. Boosting with early stopping: Convergence and consistency. The Annals of Statistics, 13344:1538-1177 2005.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.2 for computation
space and Section B for theoretical analysis.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section B
(b) Did you include complete proofss of all theoretical results? [Yes] See Section B
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? (Yes] We provide source codes in the supplementary material. We use publicly available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section C Appendix.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] The numerical standard error are shown in brackets of tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] One Nvidia 1080TI enough for one experiment run.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the publicly available datasets use.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We use publicly available datasets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use publicly available datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "vpxH1Lw8NSg", "reviewer": "Reviewer_XaCz", "summary": "This paper studies collaborative learning between organizations without sharing data, models, and even objective functions. The authors propose a Gradient Assisted Learning (GAL) method that collaboratively optimize the aggregation of local loss functions, while each worker iteratively fits the gradients of the overarching objective function. The authors prove that the proposed GAL can achieve the performance close to centralized learning when all data, models, and objective functions are fully disclosed. The empirical results support this argument.", "questions": "Please response the cons in the rebuttal.", "limitations": "Limitations are duely discussed, while no potential negative societal impact is identified.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ This paper addresses a real and timely problem that collaboratively optimize the aggregation of local loss functions of different organizations, while each worker iteratively fits the gradients of the overarching objective functions.\n+ The paper is well-written and easy to follow.\n\nCons:\n- I find the setting is a bit unrealistic/confusing and needs further clarification. Is there any restrictions on the discrepancy between the data distributions of different organizations? In some extreme cases, would it be possible that the gradients from other devices cannot assist or have bad effects?\n- The experiments are not comprehensive enough. Does any other existing method can be applied here. I understand that these methods are not designed to this setting and they may be rigorously reasonable, but I find this would be helpful to discuss them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "f0BxgrLP3PK", "reviewer": "Reviewer_HvzM", "summary": "Gradient Assisted Learning (GAL), a decentralized collaborative supervised learning methodology, is proposed. GAL works to assist a particular organization (dubbed \"Alice\") with their learning problem by utilizing other organization's learning capabilities but without sharing any sensitive information such as models or raw data between any two organizations. This is accomplished by iteratively having each \"helper\" organization learn a model of the (pseudo-)residual using their own data and sending the model's prediction back to Alice. At each iteration, Alice updates the overall model to be learned by incorporating these predictions and optimizing their associated weightings. At test-time, Alice queries the helper organizations for their predictions and feeds those predictions through the learned weightings to arrive at a prediction. A theorem is given that states that, under mild technical conditions, the loss of the GAL model converges (as the number of \"assistance iterations\" approaches infinity) to the optimal loss amongst all models in the span of the organizations' model function classes. Experiments on a variety of datasets shows that GAL outperforms prior state-of-the-art decentralized methods, with performance approaching that of centralized learning paradigms.", "questions": "Concerns/Suggestions:\n\nCS1. Please explicitly tell the reader what the arrows in the tables represent.\n\nCS2. The authors mention the \"Interm\" method in relation to Tables 1, 8, and 9, but this method does not appear in these tables.\n\nCS3. In the proof of the theorem, when you multiplied both sides of (12) by $|\\mu_{f^*}^g-\\mu_{F^{t-1}}^g|$ and then summed up to arrive at (13), it appears to the reviewer that you forgot to multiply the $a_t^2$ term---shouldn't (13) have a factor of $\\lVert\\mu_{f^*}-\\mu_{F^{t-1}}\\rVert_1$ attached to the $a_t^2$ term? This may cause issues with the convergence of this term to zero as $t \\to \\infty$; you may need a condition requiring $a_t^2$ to go to zero faster than the partial sums of $a_t$ goes to $\\infty$.\n\nCS4. The authors state that \"Prior work on AL is limited to mean squared loss regression with only two organizations using a sequential exchange of information.\" This doesn't appear to be entirely true---the original AL methodology appears applicable to regression tasks with $m\\in\\mathbb{N}$ organizations using arbitrary models and loss functions (not just mean squared loss), so long as the loss functions are minimized at the origin (e.g., they are norms). What is true is that the theoretical guarantee for converging to the oracle performance given in the AL paper is restricted to all $m$ organizations using linear regression models. It so happens that the pseudo-residual GAL uses, i.e., the functional gradient step, reduces to the conventional residual in the regression setting when Alice's loss function is equal to the squared error loss. With respect to this limitation, GAL generalizes past the regression task setting (allowing for classification settings) by sharing these pseudo-residuals (gradient values) instead of the residuals used in AL. Please fix these overstatements.\n\nCS5. The reviewer is concerned with some of the claims made in Section 4.3 \"Comparison with AL.\" In particular, the authors argue that GAL's enhanced performance over AL is in part due to the parallelizability (with respect to the organizations) of GAL, whereas AL is trained sequentially. However, AL appears to be immediately parallelizable as well; the `for` loop over organizations $j$ in Procedure 1 of the AL paper [1] does not depend on organization $j' \\ne j$, so one may simply perform the AL model fitting for organization $j$ in parallel from the model fitting for the other organizations. With this in mind, GAL is not necessarily $M\\times$ faster than AL as claimed in the paper, so the impact of the authors' Table 12 is limited to the contributions of \"deep model sharing,\" which the authors claim is not always expected to help the performance of GAL.\n\nCS6. Overall, the reviewer found the contributions to be overstated with respect to the earlier paper [1].\n\n\nQuestions:\n\nQ1. Can you give an actual rate of convergence in your theorem instead of just an asymptotic result?\n\n\n[1] Xian, Xun, et al. \"Assisted learning: A framework for multi-organization learning.\" Neurips 2020.", "limitations": "The authors do a good job at describing their limitations, namely, that their theoretical convergence result is not sufficiently strong to describe why GAL converges after only a few iterations of \"assistance,\" and that it may be possible to infer sensitive information from the shared pseudo-residuals. The authors demonstrate that upon adding moderate amounts of noise to the pseudo-residuals to take care of the latter limitation, the performance of GAL does not significantly deteriorate, and that performance is still better than that without collaboration. However, it would be interesting to discuss (and potentially test) the issue of adversarial sensitivity; what happens if one of the organizations is malicious or subject to an adversarial attack, either in their data or in the predictions that they feed back to Alice? Relative to traditional adversarial machine learning settings (wherein the adversary is typically assumed access to just one mode of attack at a time, e.g., the input data at test time), the proposed decentralized learning paradigm potentially offers an adversary more avenues for attack, e.g., at a particular organization's input data and at the various predictions fed to Alice.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1. Quality. The authors formulate their methodology into a technically sound and rigorous mathematical and algorithmic framework. The experiments are thorough and demonstrate the strengths of the proposed method in a wide range of applications.\n\nS2. Clarity. The paper is well-motivated and easy to read.\n\n\nWeaknesses:\n\nW1. Originality. The proposed methodology is extremely similar to the prior \"assisted learning\" (AL) work [1], providing what seems to be only incremental extensions to the technical aspects of the AL framework. See below for more details regarding this concern.\n\n\nNeutral:\n\nN1. Significance. Since the proposed method (GAL) applies to more general problems (e.g., classification) than AL and is shown to outperform AL in the considered benchmarks, the reviewer finds that the paper advances the practical strength and performance of the literature's decentralized collaborative learning methods. However, the reviewer believes that the approaches taken by the authors to extend AL to GAL (i.e., replacing regression residuals with classification/regression pseudo-residuals, and optimizing the model weights and learning rates) are not significant conceptual/theoretical advancements.\n\n\n[1] Xian, Xun, et al. \"Assisted learning: A framework for multi-organization learning.\" Neurips 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zm7wMNLNqbf", "reviewer": "Reviewer_mrR4", "summary": "This paper proposes the gradient assistive learning (GAL) algorithm for a multi-organization collaborative learning setting. The proposed algorithm can be considered as a distributed version of the gradient boosting algorithm. The authors provide convergence analysis and experimental results to show the benefit of the proposed algorithm.", "questions": "What would be a more realistic setting to apply this method in the image / language modality?", "limitations": "Authors mentioned that they \"do not foresee any negative societal impacts\". I agree with this.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper studies an interesting distributed learning scenario which I think is relatively less explored in the community. The proposed algorithm is a reasonable extension of prior work on assistive learning. This paper also builds a connection between the GAL algorithm and the classic gradient boosting algorithm. The paper is relatively well-written and easy to follow. Authors provide results on a variety of datasets. I also appreciate that the authors provided detailed descriptions of the experiments in the appendix, as well as the results in the privacy enhancement setting. I did not check the proof details but the theoretical result looked sound.\nOverall, I think this is a good paper.\n\nWeaknesses:\n\nFor the MNIST and CIFAR experiments, authors assume that the images are split into patches, and different patches are stored in different organizations. I don't think this is a realistic experimental setup, and it's unclear why different organizations only store a patch of a natural image. I would like to see a more realistic scenario where this method can be applied in deep learning models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6yjtYPIiYse", "reviewer": "Reviewer_3PDZ", "summary": "This paper proposes an algorithm for multi-agent learning. By sharing gradient information but not data among agents, this algorithm can achieve collaborative learning while keeping data privacy. This paper also shows this algorithm can converge in limit, and provides supporting numerical experiments.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This algorithm solve the feature-distributed optimization problem, which is valuable and practical.\n\n2. The algorithm shares gradient information rather than data, which preserves the privacy.\n\n3. This paper proves the algorithm converges in limit.\n\n4. The numerical experiments are comprehensive and convincing.\n\nWeaknesses:\n\n1. The theoretical analysis may be able to be improved. The current proof only shows convergence in limit, maybe can obtain an explicit convergence rate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept