Assisted Learning for Organizations with Limited Imbalanced Data
Authors: Cheng Chen
Jiaying Zhou
Jie Ding
Yi Zhou
Bhavya Kailkhura
Venue: ICLR 2022 Submitted (2022)
Keywords: Assisted learning
Deep learning
Reinforcement learning
Optimization
Heterogeneous learner
Imbalanced data

Abstract: We develop an assisted learning framework for assisting organization-level learners to improve their learning performance with limited and imbalanced data. In particular, learners at the organization level usually have sufficient computation resource, but are subject to stringent collaboration policy and information privacy. Their limited imbalanced data often cause biased inference and sub-optimal decision-making. In our assisted learning framework, an organizational learner purchases assistance service from a service provider and aims to enhance its model performance within a few assistance rounds. We develop effective stochastic training algorithms for assisted deep learning and assisted reinforcement learning. Different from existing distributed algorithms that need to frequently transmit gradients or models, our framework allows the learner to only occasionally share information with the service provider, and still achieve a near-oracle model as if all the data were centralized.

Under review as a conference paper at ICLR 2022
ASSISTED LEARNING FOR ORGANIZATIONS WITH LIMITED IMBALANCED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
We develop an assisted learning framework for assisting organization-leve learners to improve their learning performance with limited and imbalanced data. In partic- ular, learners at the organization level usually have sufficient computation resource, but are subject to stringent data sharing and collaboration policies. Their limited imbalanced data often cause biased inference and sub-optimal decision-makinn In our assisted learning framework, service from a service provider and aims to enhance its model performance within few assistance rounds. We develop effective stochastic training algorithms for assisted deep learning and assisted reinforcement learning. Different from exist- ing distributed algorithms that need frequently transmit gradients models, our framework allows the learner to only occasionally share information with the service provider, and still achieve near-oracle model as if all the data were centralized.
organizational learner purchases assistance
INTRODUCTION
Modern distributed learning frameworks such as federated learning (Shokri & Shmatikov 2015 Konecny et al. 2016 McMahan et al., 2017) aim improve the learning performance for large number of learners that have limited data and computation/commuiictt
resources. These learning frameworks are well suited for cloud systems and IoT systems (Ray) 2016 Gomathi et al., 2018 that
manage numerous smart devices through wireless communication.
Over the past decade, many organizations, e.g., government agencies, hospitals, schools, and com- panies, have integrated machine learning models into their work pipeline to facilitate data analysis and decision making. For example, according to a recent survey (Financesonline) 202T), 49% of companies worldwide are exploring or planning to use machine learning, 51% of organizations claim to early adopters of machine learning, and the estimated productivity improvement from the learning models is 40%. However, the performance of their machine learning models critically depends on the quality of the data, which typically is of a limited population and is biased toward certain distributions. Unfortunately, the existing learning frameworks cannot help big organizations improve their learning performance due to the following major restrictions.
Unlike smart devices in the conventional federated learning, organizational learners typically cooperate with a single external service provider under a rigorous contract. Moreover, the service provider is often presumed to have more data with better quality than the organization.
Conventional distributed learners achieve the performance goal by frequently exchanging informa- tion with other learners. comparison, each learning round for organizational learners is costly, as they need to pay the provider for assistance and need to exchange a large amount of information with the provider. Hence, organizational learners desire to achieve a significant performance improvement within limited assistance rounds.
Therefore, there is an emerging need to develop a modern machine learning framework for organiza- tional learners that can significantly improve the model performance by purchasing limited assistance services from external providers without data sharing. This constitutes the goal of this work.
In this work, develop assisted learning framework in the 'horizontal-splittin the learner and the service provider possess different datasets that are utilized for training a common model. In context, the leamer''s 's data is assumed to be limited and imbalanced, while the provider's
setting, where

---

Under review as a conference paper at ICLR 2022
data is supposed to be big and complements the learner' data. Our learning framework nicely suits the organizational learners' characteristics: they have a very limited budget for purchasing external assistance services, yet they exchange a large amount of side information with the provider per assistance round to maximize the performance gain. This is opposite to federated learning, where smart devices are equipped with only limited communication budget but can endlessly learn through interacting with the cloud. We summarize our contributions as follows.
1.1 OUR CONTRIBUTIONS
We identify the need for developing an assisted learning framework for facilitating the deployment of general machine learning in large organizations. This learning framework addresses the unique challenges as explained previously.
We first develop an assisted deep learning framework for organizational learners with limited and imbalanced data, and propose stochastic training algorithm named AssistSGD. Specifically, every assistance round of AssistSGI consists of two phases. In the first phase, the learner performs local SGD training for multiple iterations and sends the generated trajectory of models together with their corresponding local loss values to the service provider. In the second phase, the provider utilizes the learner's 's information to evaluate the global loss of the received models, and uses the best model with the smallest global loss as an initialization. Then, the provider performs local SGD training for multiple iterations and sends the generated trajectory of models together with their corresponding local loss values to the learner. Finally, the learner utilizes the provider's information to evaluate the global loss of the received models, and outputs the best model with the smallest global loss. Under mild technical assumptions, we formally prove that AssistSGD with full batch gradient updates is guaranteed find critical point of the global loss function general nonconvex optimization.
We further generalize the framework to enable assisted reinforcement learning, and develop a policy gradient training algorithm named AssistPG, which has the same training logic as that of AssistSGD,
Through extensive experiments with deep learning and reinforcement learning, we demonstrate that the learner can achieve a near-oracle performance with AssistSGD and AssistPG as if all the data were centralized. In particular, as the learner data's 's level of imbalance increases, AssistSGD can help the learner achieve higher performance gain. Moreover, data are never exchanged in the assisted learning process for both participants.
1.2 RELATED WORK
Assisted learning. Earlier work on assisted learning (Xian et al. 2020) considers organizations that collect different features from the same cohort. This is in contrast with our context where organizations hold the same features but imbalanced data distributions or environments. Also, our method applies to general deep learning and reinforcement learning tasks, which are beyond the previously studied regression task. Consequently, our algorithm designs and application scenarios significantly different from the prior work.
Distributed optimization. In conventional distributed optimization, the data is evenly distributed among workers, which collaboratively solve a large-scale problem by exchanging local information (gradients, models.) via either decentralized networks (Xie al. 2016 Lian et al. 2017 2018) or centralized networks (Ho et al. 2013 Li et al., 2014] Richtarik & Takave| [016: Zhou et al.
2016; 2018). As a comparison, our AssistSGD only requires few transmission rounds between the
learner and provider. This is particularly appealing for organizational learners, who can employ a sophisticated optimization process locally while restricting the rounds of assistance.
Federated learning. Federated learning is an emerging distributed learning framework (Shokri & Shmatikov 2013 Koneeny et al. 2016; McMahan al. 2017 Zhao et al.) 2018 Li et al. 20200] Diao et al. 2021). which aims to learn a global model using the average of local models trained by
numerous smart devices with heterogeneous data. The existing federated learning algorithms require frequent transmissions of local model parameters. This is different from our solution designed for the organizational learning scenarios, where each learner is an organization that often has unconstrained communication and computation resources, but is restricted to interact with other external service providers. Our solution aims to help the learner improve learning performance within ten rounds, while federated learning needs many more rounds.
2

---

Under review as a conference paper at ICLR 2022
2 ASSISTED DEEP LEARNING
In this section, we introduce the assisted deep learning framework. Throughout the paper, L learner who seeks assistance, and denotes service provider who provides assistance L.
denotes
2.1 PROBLEM FORMULATION
We consider the case where the learner L aims to train machine learning model 0 that performs well on its own dataset D(I.) and generalizes well to unseen data. In general, L can train machine learning model by solving the empirical risk minimization problem mingge [(; D(L)), where ʃ(; D(L) is the loss D(L) and @ is the parameter space. Standard statistical learning theories show that the obtained model can generalize well to intact test samples under suitable constraints of model parsimoniousness (Ding et al., 20188). However, when the user's data D(t.) contains a limited number of samples that are highly imbalanced, the learned model will suffer from overfitting or deteriorated generalization capability to the unseen test data.
To overcome data deficiency, the learner L intends to connect with external service provider P (e.g., commercialized data company), who possesses data D(P) that are sufficient or complementary to the learner's data D(L), Ideally, the user I, would improve the model by solving the following auggeneddeeted problem, where D(L.P) = D(L) D(P) denotes the centralized data.
( min f(0; (():)(L.P)), where ((I.P) (L)) (D()))
8E0
We note that f(0; D(L.)) f(0, D()) + f(0, D(P). If DPF) is generated from a distribution that is close to the underlying data distribution. then is expected that ((L.P) will achieve significantly improved performance on unseen data. However, it is unrealistic to centralize the data since the interactions between the learner L and the provider P are often restricted by various regulations. Some representative regulations that formally define the assisted learning framework are listed below.
Assisted Learning Protocols
1. No data sharing: Neither the learner I. nor the provider will share data with each other.
2. Limited assistance: The learner I. has a limited budget for purchasing assistance service. The learner desires to maximize the performance gain with only few assistance rounds.
3. Unlimited communication bandwidth: In each assistance round, the learner and the provider can exchange unlimited information. For example, the learner (resp. provider) send employee (resp. technician) deliver large-capacity hard drive to the other.
The above assisted learning framework is different from the existing learning frameworks. For example, in federated learning, many devices collaboratively train global model via a large number of learning rounds with limited communication bandwidth. In comparison, the organizational learner in assisted learning can only query few rounds of assistance from the provider, but can exchange unlimited information with it.
Hence, need develop training algorithm that can substantially improve the learner's model quality using limited interactions with the provider for assisted learning. Next, we present an assisted stochastic gradient descent (AssistSGD) algorithm for this purpose.
2.2 ASSISTSGD FOR ASSISTED DEEP LEARNING
We propose AssistSGD in Algorithm for assisted deep learning. The learning process consists of R rounds, each consisting of the following interactions between the learner I. and the provider P.
(1) First, the learner L initiates local learning process. It initializes a model 0(-) and applies SGD with lcarning rate 1) to update it for T iterations using the local dataset (LL), Then, the leamer evaluates the local loss f(; D(L)) in a subset T of the iterations = 0, 1. Lastly, the learner sends this subset of models and their corresponding local loss to the provider P.
3

---

Under review as a conference paper at ICLR 2022
(2) Upon receiving the information from the learner L, the provider P first evaluates the global loss D(LD))))) of the received set of models (0 I. 7 and picks the best one (denoted by 0 ization. Note that the global loss can be evaluated because the local loss " D()),, ( E T} are provided by the learner I., and the provider P just needs to evaluate the local loss D ; the provider applies SGD with learn- ing rate 7) to update the model for T" iterations the local dataset D(P), Then, the provider evaluates the local loss [(; D(P) in a subset T' the erations / = 0, 1,.."' - 1, and sends the subset of models and their corre- sponding local loss to the learner L.
Algorithm AssistSGD
Input: Initialization model 0 learning rate 7), assistance rounds R, local iterations T,
for assistance rounds 1, ....., R do
Learner L:
) for initial-
Initialize 0g 00-1
Local SGD training to generate (0!4) IT-1
Send , ( ;(U)) to provider P.
f(0,')
Provider P
Initialize 0 = arg min
DE (of" S(0; PI).
T}. After that,
Local SGD training generate {0(²) 1r"-1
Send s(0 ((P)) ((P)) to learner L.
Learner L:
Output 0" = arg min
DE (o!") JETT [(0;
end
Output: The best model in {0T),- 1-=1-
(3) Once the learner L receives feedback from the provider P, it evaluates the global loss f(; P(I.P) of the received set of models (B!"), , E T') and picks the best model as the output model of this assistance round.
The above algorithm works for general deep learning tasks. It does not require data sharing between the learner and the provider. Moreover, in each learning round, the learner and the provider exchange a small number of their local training models. As we show later the experimental studies, it suffices to sample the iterations in T,T' at low frequency. Such an assisted learning process is very different from, for example, the federated learning process. Particularly, in each round of federated learning, all learners perform small number of local SGD updates and send their last output models to the cloud due to limited communication bandwidths. Consequently, the global model needs large number of learning rounds to achieve desirable performance.
Discussions.
2.3 CONVERGENCE ANALYSIS
In this subsection, we show that AssistSGD provably converges to stationary point in smooth nonconvex optimization. For simplicity, we consider the deterministic setting, where the local training of AssistSGD uses full gradient updates. We also make the following reasonable assumptions. Assumption 1. We assume that the assisted learning problem in equation / satisfies the the following conditions.
The global loss f(0; DIL.P)) has L-Lipschitz gradients. Moreover, inf, (0; D(LP)) > -0;;
2. There exists a G > O such that max (O; all generated by AssistSGD. This holds when the generated model trajectory is bounded.
((P) IIV (0; D(L.P))I)) < for
Note that in each assistance round both the provider and the learner pick the best model via the arg min operation to initialize and finalize their local training. This guarantees that AssistSGD continuously makes optimization progress. Specifically, we let @(l.)r g(P).r denote learner L's and provider P's initialization models, respectively, in the round r. Recall that is the output model of learner L. (see Algorithm [T. Then, the two arg min operations guarantee that the following proposition holds.
Proposition The sequence of global loss (f(0"; (D)P:))))))) achieved by AssistSGD monotonically decreases to a finite limit, namely, in any round r,
((0",D(L.))) S ,(0.)),, < s(o(2), D(I.P)) ((L.P)) = -((0-), D(U.P)).
Next, we prove that the output model or asymptotically converges stationary point.
4

---

Under review as a conference paper at ICLR 2022
Theorem 1. Let Assumption 7 hold and run AssistSGD with full gradient updates for R assistance + + as R - oc.
rounds. With = C -0.),, ,, we have that minogrsR-1 |
[V.L((()))
Therefore, with a proper choice of the learning rate 7, assisted learning is guaranteed to find stationary point in general nonconvex optimization. We note that it is generally hard to establish a tight convergence complexity bound for AssistSGD due to the uncertainty of the arg min operations. Still, our experiments show that it can often achieve the performance of centralized SGD training.
3 ASSISTED REINFORCEMENT LEARNING
In this section, we extend our assisted learning framework to Reinforcement Learning (RI.) scenarios enhance the model generalizability. We first introduce some basic setup of RI..
Markoy Decision Process (MDP). We consider a standard finite-horizon MDP that is denoted by tuple M (8, A, P, T, T. Po; T", where S is the state space, A corresponds the action space, P: S x A x S + [0, 1] denotes the underlying state transition kernel that drives the new state given the previous state and action, denotes the initial state distribution, and T is the episode length. Given a policy parameterized with 0. the goal of RL, also known as on-policy learning, is to learn an optimal policy parameter 8 that maximizes the expected accumulated reward, namely 0 = arg maxg J(0) == 7 where the expectation is taken with respect to the finite-length episode.
A I the reward function, π S
is the policy, Po
3.1 PROBLEM FORMULATION
We assume that an RL learner L has collected a small amount of Markovian data D(L) by interacting with certain environment. It wants train policy that generalizes well to other similar environ- ments. However, the data and environment that the learner L can access are limited. In assisted reinforcement learning, the learner L aims enhance its policy's generalizability unseen environ- ments by querying assistance from service provider P. For example, autonomous-drivee companies typically own limited data that are insufficient for training good autonomous driving models that perform well in heterogeneous environments, and they can purchase assistance service from big companies (who own massive data) improve the model performance and generalizability.
startup
Formally, we assume that there is an underlying distribution of transition kernel that models the variability of the environment. Specifically, denote Eş as an environment with the transition kernel P& parameterized by 3, which follows an underlying distribution g. Let (0) denote the expected accumulated reward collected from the environment E3 following the policy Ty. The learner L L's ultimate goal is to learn good policy that applies to the underlying distribution of environment, namely, max, Bg~ [J)( In practice, the learner only has training data collected from a limited number of environment instances, say B(L) (.8(T) .....
al.) On the other hand, the service provider may have rich experience interacting with a more diverse set of environments, say ((P) =18(1) .....,my ( Consequently, the learner aims solve the following assisted RL problem.
max Jga. (8) == Σ Jo(O)+
Σ (t)
BEA(L)
8E
which is similar the formulation of assisted deep learning. In the following subsection, will develop a policy gradient (PG)-type algorithm for solving the assisted RL problem in equation [
3.2 ASSISTPG FOR ASSISTED REINFORCEMENT LEARNING
Policy gradient (PG) is classic RL. algorithm for policy optimization. The PG algorithm estimates the policy gradient (() via the policy gradient theorem, and applies it to update the policy. Specifically, given one episode with length 7' that is collected under the current policy ma, the corresponding policy gradient takes the following form, where R(G) -i= -0-1,, is the discounted accumulated
5

---

Under review as a conference paper at ICLR 2022
reward over this episode. In practice, mini batch of episodes are used to estimate the policy gradient.
T
VJ(0) R(())
Σ
v log (s(")
we present Assisted Policy Gradient (AssistPG)-a policy gradient-type algorithm for solving the assisted RL problem in equation 2 The main logic of the AssistPG algorithm the same as that of the AssistSGD for assisted deep learning.
In Algorithm 2
Algorithm 2 AssistPG
Input: Initialization model 80, learning rate 7), assistance rounds R, local itera-
tions T.
for assistance rounds ..... do
Learner L:
Initialize o²4) 0
Local PG training to generate 10!- Jtmo
Send ( Esegus Jn(o!")) heT to provider P.
Provider P:
Initialize 0f arg (e(o heT (0)
Local PG training to generate (0(²)
$t=0
Send (0(²) Eoegn Jo(o(P) her to learner L.
Learner L:
Output or = arg max
BE her Joan (0).
end
Output:
The best model )
EXPERIMENTS
In this section, we first visualize AssistSGD training to help understand the mechanism of assisted learning. Then, we provide extensive experiments of deep learning and reinforcement learning to demonstrate the effectiveness of the proposed assisted learning algorithms.
Axditted solutions
csluterr
anditedd
(0..5))
Aaw-addandiaal soluton (700%)
Ausmacarr
Decitio Boarder
4.1 VISUALIZATION
OF ASSISTSGD TRAINING
We visualize the learning process As- sistSGD through a simple logistic regression task. Consider two classes of data samples: Class A data contains 50 points drawn from M[[11, 1], 12) and Class B data con- tains 50 points drawn from ((1 -1 1.5²
Figure 1: Visualization of AssistSGD: (a) the learner's classifiers after being assisted by the provider different rounds, and (b) oracle classi- fier obtained by using centralized data. The test accuracies are shown the parentheses.
where A and denote Gaussian distribution and identity matrix, respectively. Suppose that a learner L observes 90% class A samples and 10% class B samples. Another service provider P observes similar number data samples consisting of 10% class A samples and 90% class B samples. The learning process of AssisSGD is illustrated in Figure (Left), and we also show the oracle solution trained by SGD with centralized data in Figure (Right).
It can be that without any assistance, the learner only achieve 70% accuracy and the corresponding classifier performs poorly on the samples in class B. In comparison, after a single round of assistance, the classification performance is significantly improved to 91.9% accuracy. After

---

Under review as a conference paper at ICLR 2022
three rounds of assistance, the corresponding classifier is relatively close to the oracle classifier, which achieves an accuracy of 82.5%. Hence, can be seen that AssistSGD has the potential to achieve near-oracle performance. We also present the visualization of another regression example in Section A.2 the Appendix.
4.2 ASSISTED DEEP LEARNING EXPERIMENTS
We test the performance of AssistSGD by comparing with three baselines: standard SGD (using centralized data D)L.P),, Learner-SGD (using only the learner's data D(L),, and the FedAvg algo- rithm (McMahan et a 2017) for federated learning. We implement these algorithms to train an AlexNet (with learning rate 0.01 and batch size 256) and a ResNet-18 (with learning rate 0.1 and batch size 256) on the CIFAR-10 dataset (Krizhevsky 2009
CIFAR-I0, AlexNet, p-1/9, 70
CIFAR-10,
2.5
staa (llantzee
2.5
staa (Mhniaze Pniffrg (handaet
80
de Pulling (hantne)
it
Test Acrnnay
8
Acconeyy
-00
Triin Teea
1.5
Trin Lona
1.5
Handio Lemns-5GD
10
Lourer stc (uuddree
0.5
Fallog BB (adda))
de dedkkny (lun)i))))
Number of Bounds
Number of Rounds
Nambeer Rounds
Number of Rounds
CIFAR-10, ResNet-18, p=119, p- 0.1
CIFAR-10, ResNet-18, p-1/1, 0.1
Belba (Buciin))
Accunayy
Stain Lowe
Tee Accrssayy
28 85
3GD (Beatiec)
0.5
Fellon (Bacin)))
Number
Rounds
Number
Figure 2: Comparison of AssistSGD, SGD, Leamer--SGe -SG and FedAvg with balanced learner's data using AlexNet (top row) and R8 (bottom row).
CIFAR-10, AlexNet, p-1.9, p-1.9. m
CIFAR-10, p-11N1.
C SORP (Raecoo)) PelAvee (Bankind)
80
SoD toue leal Lamen SCD Aabi500
Toot Accurncy
Thet Accuraty
1.5
3-
40
da Deskky (ibuttto
C (lannino)
Train
40
0.5
20
F. Amibo5Ci
Number Rormis
Number
Number Rounds
Numbec Rounds
CIFAR-10, ResNet-18, 100.
CIFAR-10, ResNet-18,
2.5
80
30
Jecckng (ilaninn))
de RetAvg (Bascline) ie Aea
Accussey
Thes Accuary
Lnns
Thmil Loos
LannesSD CennsaCD-Raddta Follng (aaniee)
40
5220 de Bel (Baarier) AwrirSei
23
18
0.5
Burandl
Numberr
Figure 3: Comparison of AssistSGD, SGD, Learner-SGD and FedAvg with imbalanced learner's data using AlexNet (top row) and R8 (bottom row).
For AssistSGD, we distribute the entire training set of CIFAR-10 10 (50k samples) to the learner and provider according to two parameters: 1) data size ratio = / and 2) data imbalance ratio 7/. (0, 1) that specifies the imbalance level of the learner's data. Specifically, first randomly
7

---

Under review as a conference paper at ICLR 2022
assign sample class as the primary class of the learner's data. Then, Yz (D(()) number of samples sampled from the primary class, and the rest (1 --))=- |D(L.] number of samples are drawn from the remaining classes uniformly at random. The provider's 's data are sampled from all classes uniformly random and are balanced We also fix the number of assistance rounds to be 10. The total number of local SGD iterations in each assistance round is fixed to be 2000. We assign the SGD iteration budget to the learner and provider in proportion to their data samples' sizes. Both the learner and provider record their local training models and local loss values for every / = 50 SGD iterations, which is the sampling period. For FedAvg, we treat the learner and provider as two federated learning agents, and they inherit the same local data and local SGD iteration budgets from AssistSGD.
We first compare these algorithms with balanced learner's data 7t, 0.1 and varying data size ratios p == 1/9, 1/1. We plot the training loss (on centralized data D(l.,)) and the test accuracy (on the 10k test data) against the number of assistance rounds in Figure 2 (top row AlexNet, bottom row ResNet-18). Here, one assistance round on the x-axis is interpreted
2000 SGD iterations for SGD, LeDmerr-DD and FedAvg. The training loss of Learnrr-SGD is not reported as it is trained on D(L) only. It can be seen that AssistSGI achieves a comparable performance to that of SGD with centralized data. In particular, when p p= 1/9 and the learner has limited data, the test performance of AssistSGD is significantly better than Learner-SGD, demonstrating the effectiveness of querying assistance from the service provider. When p = 1/1 and the learner has more data, AssistSGD still achieves a near-oracle performance. In particular, it converges much faster and achieves a slightly better test performance than FedAvg.
We further test and compare these algorithms with imbalanced learner's s data Y = 1 and p = 1/9, 1/1. The results are shown in Figure 3 It can be seen that when p = 1/9 and the learner has limited data, AssistSGD achieves comparable performance to that of SGD, and slightly outperforms FedAvg. Moreover, when p = 1/1, AssistSGD converges slower than SGD due to the large amount of highly imbalanced data D(L), Nevertheless, still achieves a comparable test performance to that of SGD. On the other hand, AssistSGD significantly outperforms FedAvg, demonstrating the robustness of assisted learning to data heterogeneity. Comparing the results in Figure 3 with those in Figure 2 conclude that AssistSGD improves the test performance more (compare to Learner-SGD and FedAvg) when the learner's 's data are more imbalanced.
Due to space limitation, we present other CIFAR- 10 training results of both network models under the parameters p = 1/3 and ?L = 0.1, 1 and the corresponding SVHN training results in Section |A.3 of the Appendix, where can make the same conclusions for both datasets. We also explore the effect of sampling period (Section A.3.2) on the performance of AssistSGD in the Appendix.
4.3 ASSISTED REINFORCEMENT LEARNING EXPERIMENTS
We demonstrate the effectiveness of AssistPG via solving two reinforcement learning problems: the CartPole (Barto et al. 1983) and the LunarLander provided by the OpenAl Gym library (Brockman ct al. 2016). In the CartPole problem, a controller aims to stabilize a pole attached to a cart by applying left right force the cart (see the first figure in Figure and we show that AssistPG can help the controller stabilize the pole with different pole lengths. For the aunarl_ande problem, a lander initializes its landing from top left of the sky and aims to land on a landing pad by controlling its engine (see the first figure in Figure 5). We show that AssistPG can help land the lander with different engine powers.
We assume that the learner and the provider can query episode data by interacting with diverse environments. Specifically. for the CartPole problem, we parameterize the environment using the pole length. Both the learner and the provider train their control policies by playing 5 Cartpole games with the pole length randomly generated from Uniform(4. 5) (for the learner) and Uniform(0. 1) (for the provider). For the LunarLander problem, we parameterize the environment using the engine power. Both the learner and provider train their control policies by playing 10 LunarLander games with the engine power randomly generated from Uniform(10, 15) (for the learner) and Uniform(35, 40) (for the provider). Moreover, we consider two sets of testing environments that are uniform ("Test I'') and non-uniform ("Test II"),, respectively. For the CartPole problem. Test I environments randomly gener- ate the pole length from Uniform(0. (0, 5), and Test environments randomly generate the pole length from Beta(1, (1 5) with probability 0.2 and Uniform(0, 5) with probability 0.8. For the I Lunarl anderr
8

---

Under review as a conference paper at ICLR 2022
1,00
1000
3000
+
ASSetPG
AssetRS
Foskig (Bazaline) (Baseline)
Foskig (Basclinc)
Tran Dicoout Reword
800
800
(Baseline)
Test Ltwoward
Leamer-PG (Baseline)
Teat Rowwadd
Loamer-Pa (Baseline)
600
600
40
400
400
AsxiPP
right
20
PS (Bacelins)
Tastug (Daseinn)
200
200
LeamereG (Baseline)
10
Hounds
Rourda de
Rounds
Figure 4: Comparison of AssistPG, PG, Learner-PP PG and FedAvg in the CartPole game.
100
100
Trais Discouant Neward
left
right
-000
-100
200
down
landing pad
Anste7C
AssaPPG
AssatPG
-200
Pedlug (Dtawline)
Fediog (Baseline)
-400
Festug (Baselin))
PG (Baseline)
PS (Baseline)
Learnen PG (Base Inc)
Leamer-FG FG (Bacelre)
Leamer (Bazeline)
300
500
600
Figure 5: Comparison of AssistPG, PG, Learner-PG and FedAvg in the LunarLander game.
problem, Test I environments randomly generate the engine power from Uniform 40), and Test II environments randomly generate the engine power as 30r 10, where r Beta(5, 1).
We test AssistPG on both RL problems and compare its performance with three baselines: the standard PG (using centralized episode data), the Learner-PG (using only learner's episode data), and the FedAvg algorithm that uses policy gradient updates. All these algorithms are implemented with learning rate x 10-3 and episode batch size 32. We model policy using three-layer feed- forward neural network with and 32 hidden neurons for CartPole and LunarLander, respectively. Moreover, for AssistPG, we fix the total number of assistance rounds to be and 5 for CartPole and LunarLander,, respectively. The total number of local PG iterations in each assistance round is fixed to be 20 for both problems. We also set the sampling period to be four, namely, the learner and the provider record their local model and discounted training reward for every four local PG iterations. Figures f and 5 plot the discounted training rewards (collected in local environment only), Test I cumulative rewards, and Test II cumulative rewards against the assistance round obtained by all these algorithms, for solving the CartPole and round is interpreted as local PG iterations for algorithms other than AssistPG.
problems, respectively. Here, one assistant
indicattes that AssistPG outperforms Learner-PG and Fed-Avg, when the testing environments include diverse lengths of poles. Also, AssistPG can achieve comparable performance that the PG with centralized data. Moreover, Figure [ indicates that AssistPG can swiftly adapt to scenarios out their comfort zone (namely the training environments) only few rounds. These experiments demonstrate that our assisted learning framework can help the learner significantly improve the quality of the policy for handling complex RL problems. Extensions more details of these reinforcement learning experiments well video demonstrations are included in Section A.4 of the Appendix.
5 CONCLUSION
This work develops a new learning framework for assisting organizational learners to improve their learning performance with limited imbalanced data. In particular, the proposed AssistSGD and AssistPG allow the provider to assist the learner's training process and significantly improve its model quality within only few assistance rounds. We demonstrate the effectiveness of both assisted learning algorithms through experimental studies. In the future, we expect that this learning framework can be integrated with other learning frameworks such as meta-learning and multi-task learning. A limitation of this study is that only considers a pair of learner and provider. An interesting future direction is to emulate the current assisted learning framework to allow multiple learners service providers.
The Appendix document contains the proof of the technical result, more experimental details, and video demonstrations of the reinforcement learning performance.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846 846, 1983.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint (000:06000.0/444 2016.
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. Proc. ICIR, 2021.
Jie Ding, Vahid Tarokh, and Yuhong Yang. Model selection techniques an overview. IEEE Signal Process. Mag., 35(6):16-34, 2018.
Financesonline. Market share data analysis. Bechine--earminggsttttiis
https://ELaannesoolinneecc
2021. Accessed: 2021-01-18.
RM Gomathi, G Hari Satya Krishna, E Brumancia, and Y Mistica Dhas. A survey iot technologies, evolution and architecture. Proc. ICCCSP, pp. 1-5. IEEE, 2018.
Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B. Gibbons, Garth A Gibson, Greg Ganger, and Eric Xing. More effective distributed ML via stale synchronous parallel parameter server. In Proc. NeurlPS, pp. 1223-1231. 2013.
Jakub Konecny, H Brendan McMahan, Felix X Yu, Peter Richtárík, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610005492, 2016.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In Proc. OSDI, pp. 583-598, 2014.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In Proc. ICLR, 2020.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu, Can decentralized algorithms outperform centralized algorithms? gradient descent. In Proc. NeurIPS, 2017.
study for decentralized parallel stochastic
Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In Proc. ICML, volume 80, pp. -043-3052, 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effcce learning of deep networks from decentralized data. In Proc. AISTATS, pp. 1273-1282. PMLR, 2017.
Partha Pratim Ray. A survey of iot cloud platforms. Future Computing and Informatics Journal, 1 (1-2):35 46, 2016.
P. Richtarik and M. Takave. Distributed Coordinate Descent Method for Learning with Big Data. J. Mach. Learn. Res., 2016.
Reza Shokri and Vitaly Shmatikov. Privacy-preserviee deep learning. In Proc. CCS, pp. 1310-1321. ACM, 2015.
Xun Xian, Xinran Wang. Jie Ding, and Reza Ghanadan. Assisted learning: A framework for multi-organization learning. Proc. NeurlPS, 2020.
Pengtao Xie, Jin Kyu Kim, Yi Zhou, Qirong Ho, Abhimanu Kumar, Yaoliang Yu, and Eric Xing. fighter-communication distributed machine learning via sufficient factor broadcasting. In Proc. UAI, pp. 795-804, 2016.
10

---

Under review as a conference paper at ICLR 2022
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. ArXiv:1806.0055: 2018.
Y. Zhou, Y.L. Yu, W. Dai, Y.B. Liang, and E.P. Xing. On convergence of model parallel proximal gradient algorithm for stale synchronous parallel system. In Proc. AISTATS, 2016.
Yi Zhou, Yingbin Liang, Yaoliang Yu, Wei Dai, and Eric P. Xing. Distributed proximal gradient algorithm for partially asynchronous computer clusters. J. Mach. Learn. Res., 19(19):133,, 2018.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOF OF THEOREM 1
For brevity, throughout the proof, we denote the loss functions ((: f(; ((: as ft. fr. pL, respectively.
We consider any assistance round r, and first study the local training of the learner L. Recall that the learner L initializes with the output model obtained from the previous round, namely oft).-- - - the local training, learner performs T local gradient descent steps and generates the trajectory (o!!).) Y Then, provider P picks the best model from this trajectory that achieves the minimum global loss, and we denote this model as 8( for certain {0, ...,.... T). It is clear that (L.,((())))))) (0!t))) for all t. By smoothness of the global loss and the gradient descent update rule, we have that
f0. s f..((( + 10t
of) V.LLP(
L
110
2
Lap²
= -Lr(0 + (-nV ( () +
| L (1
2
Since f S f k+1 the above inequality further implies that
Ln (V VLP S 2 (hh00 - LG
(3)
Next, we consider the local training of the provider P. Recall that the provider P will initialize ((),r_ In the local training, the provider P
with the best model sent by the learner, namely o/ k performs T" local gradient descent steps and generates the trajectory (o(P)"TT According to the smoothness of the global loss, we have that
< fi.p(((P')) (r
(0(p)rr 1. Vi. -
10pp/rr -
2
Lai
S1.p(0b
V
(0
2
ViiP ()))))
2
Ln?
= -LLP(0( (IVV )1 (VLL0
(LL((
V (
*
2
IGRn22
IG
SL.,((( - (L 012
(4)
2
2
where the last inequality utilizes equation and the boundedness of the gradient. Denote (P)) the best model from this trajectory that achieves the minimum global loss. The above Inequality (4 and Proposition 1 further imply that
fL fL
(for some k' E (O....,T'))
5 (1. 1/ V VI (( + LG?2
1, 11 1) 7/l SL.
fi //l (i. _ -
V/l, (1).)r) / 11) (VV (1)) ||| LG
3 h1.((0 (1
27 LTG + 2
f1,((--11) - - fL.( 3n LTG2
L
(5)
12

---

Under review as a conference paper at ICLR 2022
where the inequality (i) uses the fact that
2n/ Vi. V... "II[VVi..P000 "1I.0000"""IIVVS
s 2n/ (0 - V.i.p(00
s 2 (.). 1
= 21 VL
S 2 LTG
Telescoping the incquality in Iquationnovver I,.... and rearranging them, we obtain that R-
(SLP()) fL.P(0%) - info fL.P(0)
nR +3nLTG..
R
Choosing
(1,000))
inf, (0)
finally obtain that
3RITG
R
12LTG
min [VLLP((')) )|² 0SPER-1
V (VLL.((0))
(fi,((8) - inf fL)P(
Consequently, minog-gn-1 P 0 as R + co. This completes the proof. We note the above complexity bound may be tight due the two arg min operations. We conjecture that the assisted gradient descent can achieve the same order of convergence rate as the gradient descent algorithm in nonconvex optimization.
the learning rate is small, the Inequality 3 shows that the gradient V (L (.. should be well aligned with the gradient of the global loss (e Intuitively, this is because (L)) is the best model chosen from the training trajectory (o(h)rr }r that achieves the minimum global loss, and therefore the subsequent gradient update (f must be badly correlated with V (L. that the global loss actually increases after the next gradient descent iteration.
Remark:
A.2 VISUALIZATION OF ASSISTSGD TRAINING: A REGRESSION EXAMPLE
We apply the AssistSGD to solve a regression problem with simulated data = 1-1, 1).
[1, [1.5| and hyperparameters T = 10, 17 0.9" for both the learner and the provider. We run the algorithm for R
assistance rounds. Figure o shows the learning trajectory of #" for 0, 1,... 9. It can be seen that at the beginning, the learner L's learning trajectory moves toward oracle solution since the directions of two local optima are roughly the same; then, oscillates in between two opposite directions and converges to the oracle solution.
L's initial d' trajectory L's local opt
0.25
0.00
Global opt
0.25
-0.50
-125
x
-150
Figure 6: Learning trajectory of AssistSGD in regression.
13

---

Under review as a conference paper at ICLR 2022
A.3
ADDITIONAL EXPERIMENTS OF ASSISTED DEEP LEARNING
A.3.1 EFFECT OF LEARNER'S 's DATASIZE AND IMBALANCE LEVEL
CIFAR-10 Dataset
CIFAR-10, AlexNet, p-1.,, 701
CIFAR-10, AlexNet, 1-3,3
2.5
80
2.5
@-SiD (itenae)) (Ruilln))
BGD toua Ena) AvdiA00D
Thst Jccunce
Tos Accussy
60
losn
Tiiie Lowe
3GD 90D (Radlix))
500 Baccoo 5GD (aaa
Re-kiiki (Ras lualin))
0.5
0.5-
0
0
Nunben Rounda
Rounds
Nunber
Namber
Figure 7: Comparison of AssistSGD, SGD, Learner-Son SGD and FedAvg with = 1/3 using AlexNet.
CIFAR-10, ResNet-18, pr 113, 0.
CIFAR-I0, ResNet-18, p-1/3, 7c-
Ea-Saa (laiiis)) Ledkng (tlaetto
Low
Accusacy
Train Low
Toa Ascusayy
D) 3GD (oueled) (ltecii)) P-i (Tharlle)) 49 Ausistii
1o-Scm [uadias LearendDD (ttetac) AwcadDD
0.5
0.5
Number
Figure 8: Comparison of AssistSGD, SGD, Learner-SGD and FedAvg with p = 1/3 using ResNet-18.
In this subsection, we present more CIFAR-10 experimental results training AlexNet (Figure 7 and ResNet-18 (Figure 8 under the data size ratio & 1/3 and different levels of data imbalance. We test and compare our AssistSGI with SGD, Learner-SGD, and FedAvg with balanced (7,. = 0.1)
and imbalanced (71. learner's data. From both figures, draw the same conclusions as those made in Section
SVHN Dataset
SVHN, AlexNet,,=119 7.=0.1
SVHN AexxNet,p-1111 2c-11
ebbiGD (Bentaa))
Fepooril Maniii Jeckey (Badax
(Bantine)
ngh 1500
Accursey
Text Accusacy
Traio Loos
1.5
3.
85
40
R0D (aannaa
32D (iaar boo)
0.5
LeaeunnAzz (Ranted)
0.5
1000
06
Number Hounds
Rounds
Number Rounds
Number
SVHN, ResNet- 101
SVHN, ResNet-18, 18, pm 1/1, 70
120
Baddact Fethes (Banlee)
Le SCED 18 Biuntee (Bawkoo))
Tox Acsurare
Aucunacc
1.5
80
1.5
50
Than
Train
10
SUL Jaancaal Leaare 3GD Danciie
40
36D Puandacl
0.3
20
0.5
20
Lowan 5KID [laodoo]
Bouuds
Bounds
Figure 9: Comparison of AssistSGD, SGD, Learner-SGD and FedAvg on SVHN with balanced
leamer''s s data using AlexNet (top row) and ResNet-18 (bottom row).
14

---

Under review as a conference paper at ICLR 2022
SVHN, AlexNet, p-119, 7c-1
SVHN, plexNet, 1x
100
G-RD (Sow Palkin (Raulla))
о OSdi
FolAke Suarlbae) wit th-leaddii
80
de Jeckky (llunin)) B Amse
Thais Loss
Tess Jocuncey
1.5
60
1.
Thais
[Badsel (Suclaa)) Belhve (laoliee)
40
0.5
Number Bounds
Nember of Rarante
Numbrr p.1...pp11111 1-111,1177
Namber Rerandde
SVHN, R8sNet-18, p- 11
SVHN, ResNet-18,
2.5
120
2.3
100-
StAL (Ractte)
SELL (hacted) ADRID
50
-00
Thon Lowe
Teah Accumey
Truin Toae
10
60-
Louree SGD (tauaiie))
0
10
3 4
Nambez
Number Bounds
Number of Rounds
Number of Rounda
Figure 10: Comparison of AssistSGD, SGD, Learner-SGD and FedAvg on SVHN with imbalanced learner's data using AlexNet (top row) and ResNet- (bottom row).
In this subsection, we present the SVHN experimental results with balanced learner's data 7/2 0.1 (Figure ()) and imbalanced learner's s data " = 1 (Figure 10 corresponding to the CIFAR-10 10 results in Section 4.2 From both figures, we can draw the same conclusions as those made in Section 4.2 which proves that our proposed AssistSGD can work well on wide range of dataset types.
A.3.2
EFFECT OF SAMPLING PERIOD
CIFAR-10, AlexNet, AlexNet,,=11,, p2-01
CIFAR-I0, ResNet-18, p-199 7--0.1
2.5
80
2.5
Fe-50D coumtre)
Auseetiit (1rtat-2)
The Aucuussy
g
Trae lees
AoviiSDD wish Posadel
TTo Aucusso
85
0.5
Amtabii Pedee-23
0.5
(Deetted)
with Preec-ON
Ae with Pareb-or
Numben Rounda
Numbez
Namber
Figure 11: Comparison for AlexNet and ResNet-18 with balanced learner's data (74. = 0.1) under different sampling periods.
CIFAR-10, AlexNet, 1.9
CIFAR-10, ResNet-18, 1-/ y050.5
2.5
80
2.5
AwaldD Porisddd
AwondoD with Perid-d0
te he aSdD with Peivlede
Text Accuracy
--hssesii with Peviodide
Tol. Accuracy g
Trin Looe
Train Loee
2.5
8
6.-
Andescm with Persdia
0.5
Awradtii ont Ported 58
Aa--aaaa witk Dotad--0
0
0
Number Bounds
Number Rounds
Nunber
Rorand
Nomber Rounds
Figure 12: Comparison for AlexNet and ResNet-18 with imbalanced learner's data (7z = 0.5) under different sampling periods.
15

---

Under review as a conference paper at ICLR 2022
CIFAR-10, 1-11,
CIFAR-10, ResNet-18, 1=19, 76
G-90 (lmae AmaiSom vall Pecxamn
Avves80DD with Posdlo3)
Jccumacy
Phoin Lewe
Tel Accuncyy
3.5-
2.5
25 85
1.
0.5
6.5-
AweaODD weah Prindes)) Priselad)
AMAI5DD wAS Pesseiei
Number
Figure 13: Comparison for AlexNet and ResNet-18 with imbalanced learner's data (yz 1) under different sampling periods.
We explore whether increasing the sampling frequency of the model (AlexNet and ResNet-18) and loss value can improve the performance of AssistSGD. Under data size ratio p 1/9, we test and compare AssistSGD and SGD with balanced and imbalanced learner's data, i.c., 7 0.1, 0.5, 1. 12 and
We set the sampling period to be 20 and 50. The comparison results are shown in Figures
13 It can be seen that using low sampling frequency for AssistSGD already achieves the baseline performance of SGD, It implies that AssistSGD does not require much information exchange between the learner and provider. This helps save computation resources and reduce information leakage.
A.4 VISUALIZATIONS OF THE REINFORCEMENT LEARNING GAMES
In this section, visualize the landing trace of the LunarLander trained by AssistPG and Leaner-PG different test environments.
Specifically, we consider fixed map and set the engine power of the lander to be 10, 20, 30. and 40, respectively. each setting, we train the lander using both AssistPG and Learner-PG for rounds. After each round of training, we let the lander play episode using the trained model and From these successfully
plot the corresponding landing trace. These traces are plotted in Figures 14 figures, it be that the lander with engine power 20-40 trained by AssistPG land to the landpad after rounds of assisted learning. As comparison, the lander trained by Learner-PG cannot even land after 5 rounds of training. This demonstrates the advantage of AssistPG. On the other hand, when the lander has small engine power 10, it is challenging for both algorithms land the lander properly, as the engine cannot provide sufficient acceleration.
15
16
Round 1
Round
Round 3
Round
Round
Figure 14: Comparison of landing traces of LunarLander with engine power 10 trained by AssistPG and Learner-PG.
LAST WA/10
LANTWNIN
LASTWNIN
Round
Round
Figure 15: Comparison of landing traces LunarLander with engine power 20 trained by AssistPG and Learner-PG.
16

---

Under review as a conference paper at ICLR 2022
Figure 16: Comparison of landing traces Lunarl Lander with engine power 30 trained by AssistPG and Learner-PG.
JANTMNIG
SARTINRG
MANTINRG
Round
Round 2
Round
Round
Round 5
Figure 17: Comparison of landing traces of LunarLander with engine power 40 trained by AssistPG and Learner-PG.
Moreover, after rounds of training (using both AssistPG and Learner-PG), we the lander in both the test environment I I ("Test I"" and II ("Test II"), and plot the landing traces in Figures [8 and 19 respectively. Here, for each test, we consider fixed map and randomly generate 10 different engine powers from Uniform(11 00 40) (for Test I) and 30 Beta(5, 1) + 10 (for Test II).
From both figures, can be seen that the lander trained by the AssisPG lands more smoothly in all test environments under diverse engine powers than that trained by the Learner-PG.
Lerrarto
Learne
ALSIFI
Figure 18: Comparison of landing traces of LunarLander with engine power Uniform (10, 40) trained by AssistPG and Learner-Po
Learnie
LeFT
Learnsn
Learree
LAST 101TWI1-1
Leamie
Figure 19: Comparison landing traces of Lunarl ander with engine power ~ Beta(5, 1) + 10 trained by AssistPG and Learner-PG.
The video version for the CartPole and LunarLander games can be accessed from hwtpboxxcom/sh..e
the anonymous link kttpps:/vwe //www In the CartPole game, four videos record the performance of AssistPG and Learner-PG against the first five rounds with pole length equaling 1, 2, 3, and 4, respectively. Another two videos record 10 plays in the test environment and II, respectively. In all the plays, both AssistPG and Learner-PG the model trained from the fifth
17

---

Under review as a conference paper at ICLR 2022
round. In the LunarLander game, four videos record the performance of AssistPG and Learner-Po PG against the first five rounds with engine power equaling 10, 20, 30, and 40, respectively. Another two videos record 10 plays in the test environment and II, respectively. In all the plays, both AssistPG and Learner-PG use the model trained from the fifth round. The videos show that with the assistance from the provider, the user can quickly generalize its model to more diverse environments.
18


Reviews:
[{"review_id": "jxjdKpgUtN", "reviewer": "Reviewer_D8UY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- I think there may be a scenario assumed in the paper, but I'm curious to see if there are any specific scenarios in which the service provider already has enough data of the distribution that the learner wants to learn. Can you provide a more specific example? If there is an organization and service provider that will use the proposed framework in the current world, it would be good to give an example.\n\nIn experiments:  \n- It would be good to compare the performance when learning only with the data the provider has. I wonder whether the experimental settings are where learning is good enough only with the data the provider has or not.\n- I understood that standard SGD (using centralized data) baseline train a model using centralized data.  Could you explain why the values change for each round in the graphs? What does \"round\" mean in this training? \n- Could you elaborate on how you did federated learning in the baseline? I wonder how the action for each round was defined in federated learning and whether the comparison was made fair.\n", "summary_of_the_paper": "This paper proposed a new assisted learning framework in which the service provider aims at helping the organization train a model with limited and imbalanced data.", "summary_of_the_review": "The paper proposed a simple and straightforward assisted learning framework for a learner with limited and imbalanced data. It seems that the paper can be improved by presenting specific examples of scenarios that require the proposed framework and supplementing some parts in the experimental part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4b8XTdZOzD", "reviewer": "Reviewer_Bk5z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The problem setting is novel and may be relevant for real-life deployment of ML.\n2. The theoretical analysis is rigorous.\n3. The empirical evaluations test several different tasks and models.\n\nWeaknesses:\n1. Regarding the setup: Even though no data is exchanged, exchanging the models may leak data. Unbounded communication is unrealistic.\n2. Regarding algorithm 1: How are the models in $\\tau, \\tau'$ selected? What's the relationship between the number of models selected and the convergence rate or accuracy?\n3. Theorem 1 is an asymptotic result for $r \\rightarrow \\infty$, but the goal of the paper is to minimize the number of rounds, so concrete bounds on $R$ are desired, i.e. $R$ should appear in the bound.\n4. No error bars in section 4.2.", "summary_of_the_paper": "This paper studies a novel problem setup where a learner has unbalanced data, and a service provider has complementary or sufficient data, and the learner needs to improve accuracy in as few rounds of communication as possible, where the communication in each round is unbounded. An algorithm AssistSGD is proposed and shown to converge to a stationary point. Experiments show that AssistSGD has better performance than baselines and is close to centralized SGD on CIFAR 10 and reinforcement learning tasks,", "summary_of_the_review": "The paper considers a novel problem setting and has a promising direction. The theoretical part can be enhanced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1cUiUkOqa7J", "reviewer": "Reviewer_cvfa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below please find a few questions/ concerns on the paper, which I hope the authors can clarify/address:\n* At a motivational level, privacy is an important consideration that motivates the problem, but discussion on data privacy of the proposed protocol is missing. When the learner sends the model parameters along with the local losses to the service provider, there is no guarantee that data privacy is still preserved. Can the provider infer the learner’s data distribution from the exchanged information? \n\n* In the 2nd step of the AssistSGD and AssistPG algorithms, the service provider outputs the best model among the ones received from the learner with the minimal global loss; in the 3rd step of the AssistSGD and AssistPG algorithms, the learner outputs the best model among the ones received from the service provider with the minimal *global* loss. How can the learner evaluate the global loss D^{(L,P)} without access to D^{(P)} (i.e. with no data sharing as specified in the assisted learning protocols)? Is there an assumption that both the learner and the provider share a validation dataset based on which they select the models  θ^{(P)}_0 and θ^r?\n\n* The consistency result is rather weak considering that a key objective in the desiderata is to achieve limited assistance.\n\n* For the assisted deep learning experiments, the service provider is set to have access to the (unbiased) underlying data distribution (i.e. data sampled uniformly at random from each class of CIFAR-10 and SVHN); this is a rather limited scenario as the service provider should already (intuitively) return a model with decent global loss. \n\n* The paper claims that the proposed assisted learning framework is developed to facilitate the deployment of general machine learning in large organizations. However, the empirical sections do not cover any realistic use cases under the assisted learning protocol in the proposed context. \n", "summary_of_the_paper": "This paper investigates a novel learning scenario, where the learner has limited access to the global data distribution and can share learned model parameters with a so-called service provider through multiple (but limited) rounds of interactions. The motivation is interesting and seemingly useful for the scenarios described in the introduction. The authors propose an intuitive assisted learning framework applicable to both (deep) supervised learning and reinforcement learning, and experiments show that the proposed algorithms achieve comparable performance with learning from centralized data. However, there are a few concerns/ questions in the problem setup and the proposed assisted learning protocol, please see the detailed review below.", "summary_of_the_review": "The paper investigates an interesting learning protocol, proposes an intuitive assisted learning framework, but lacks sufficient theoretical justification and empirical support for the significance of the solution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a novel assisted learning scenario which would likely be useful for organizational level learners (i.e. learners with sufficient computational resources but limited and imbalance data). The paper is generally well presented, but there are shared concerns amongst the reviewers in the significance of technical contributions: (1) Due to the asymptotic nature of the consistency results, the technical strength is not strongly supported with the existing theoretical analysis. (2) Although the problem setup is novel and seems interesting, the practical significance of the results is not well supported without a concrete real-world application. (3) There are a few clarity issues raised in the reviews, which suggest that the paper could benefit from a major revision to address the above concerns.
Decision: Reject