Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates a novel learning scenario, where the learner has limited access to the global data distribution and can share learned model parameters with a so-called service provider through multiple (but limited) rounds of interactions. The motivation is interesting and seemingly useful for the scenarios described in the introduction. The authors propose an intuitive assisted learning framework applicable to both (deep) supervised learning and reinforcement learning, and experiments show that the proposed algorithms achieve comparable performance with learning from centralized data. However, there are a few concerns/ questions in the problem setup and the proposed assisted learning protocol, please see the detailed review below.
main_review: Below please find a few questions/ concerns on the paper, which I hope the authors can clarify/address:
* At a motivational level, privacy is an important consideration that motivates the problem, but discussion on data privacy of the proposed protocol is missing. When the learner sends the model parameters along with the local losses to the service provider, there is no guarantee that data privacy is still preserved. Can the provider infer the learner’s data distribution from the exchanged information? 

* In the 2nd step of the AssistSGD and AssistPG algorithms, the service provider outputs the best model among the ones received from the learner with the minimal global loss; in the 3rd step of the AssistSGD and AssistPG algorithms, the learner outputs the best model among the ones received from the service provider with the minimal *global* loss. How can the learner evaluate the global loss D^{(L,P)} without access to D^{(P)} (i.e. with no data sharing as specified in the assisted learning protocols)? Is there an assumption that both the learner and the provider share a validation dataset based on which they select the models  θ^{(P)}_0 and θ^r?

* The consistency result is rather weak considering that a key objective in the desiderata is to achieve limited assistance.

* For the assisted deep learning experiments, the service provider is set to have access to the (unbiased) underlying data distribution (i.e. data sampled uniformly at random from each class of CIFAR-10 and SVHN); this is a rather limited scenario as the service provider should already (intuitively) return a model with decent global loss. 

* The paper claims that the proposed assisted learning framework is developed to facilitate the deployment of general machine learning in large organizations. However, the empirical sections do not cover any realistic use cases under the assisted learning protocol in the proposed context. 

summary_of_the_review: The paper investigates an interesting learning protocol, proposes an intuitive assisted learning framework, but lacks sufficient theoretical justification and empirical support for the significance of the solution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies a novel problem setup where a learner has unbalanced data, and a service provider has complementary or sufficient data, and the learner needs to improve accuracy in as few rounds of communication as possible, where the communication in each round is unbounded. An algorithm AssistSGD is proposed and shown to converge to a stationary point. Experiments show that AssistSGD has better performance than baselines and is close to centralized SGD on CIFAR 10 and reinforcement learning tasks,
main_review: Strengths:
1. The problem setting is novel and may be relevant for real-life deployment of ML.
2. The theoretical analysis is rigorous.
3. The empirical evaluations test several different tasks and models.

Weaknesses:
1. Regarding the setup: Even though no data is exchanged, exchanging the models may leak data. Unbounded communication is unrealistic.
2. Regarding algorithm 1: How are the models in $\tau, \tau'$ selected? What's the relationship between the number of models selected and the convergence rate or accuracy?
3. Theorem 1 is an asymptotic result for $r \rightarrow \infty$, but the goal of the paper is to minimize the number of rounds, so concrete bounds on $R$ are desired, i.e. $R$ should appear in the bound.
4. No error bars in section 4.2.
summary_of_the_review: The paper considers a novel problem setting and has a promising direction. The theoretical part can be enhanced.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new assisted learning framework in which the service provider aims at helping the organization train a model with limited and imbalanced data.
main_review: - I think there may be a scenario assumed in the paper, but I'm curious to see if there are any specific scenarios in which the service provider already has enough data of the distribution that the learner wants to learn. Can you provide a more specific example? If there is an organization and service provider that will use the proposed framework in the current world, it would be good to give an example.

In experiments:  
- It would be good to compare the performance when learning only with the data the provider has. I wonder whether the experimental settings are where learning is good enough only with the data the provider has or not.
- I understood that standard SGD (using centralized data) baseline train a model using centralized data.  Could you explain why the values change for each round in the graphs? What does "round" mean in this training? 
- Could you elaborate on how you did federated learning in the baseline? I wonder how the action for each round was defined in federated learning and whether the comparison was made fair.

summary_of_the_review: The paper proposed a simple and straightforward assisted learning framework for a learner with limited and imbalanced data. It seems that the paper can be improved by presenting specific examples of scenarios that require the proposed framework and supplementing some parts in the experimental part.
review_title: null

Meta-review: The paper proposed a novel assisted learning scenario which would likely be useful for organizational level learners (i.e. learners with sufficient computational resources but limited and imbalance data). The paper is generally well presented, but there are shared concerns amongst the reviewers in the significance of technical contributions: (1) Due to the asymptotic nature of the consistency results, the technical strength is not strongly supported with the existing theoretical analysis. (2) Although the problem setup is novel and seems interesting, the practical significance of the results is not well supported without a concrete real-world application. (3) There are a few clarity issues raised in the reviews, which suggest that the paper could benefit from a major revision to address the above concerns.
Decision: Reject