Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Background disclaimer: I work in RL research for quite an amount of time, but I do not know much about the domain of distributed systems. For this reason, I may not know the details of technical terms, and I might not be the best person to review this work (when compared with the literature in this field). Nevertheless, below I try to give my evaluation based on reading the paper. 

====================
In this work, the authors applied value-based reinforcement learning to learn an optimal policy for global parameter tuning in the parameter server (PS) that trains machine learning models in a distributed way using  stochastic gradient descent. Example parameters include SGD hyper-parameters (such as learning rate) and system-level parameters. Immediate cost is to minimize the training time of the SGD algorithm, and i believe the states are the server/worker parameters. From the RL perspective, the algorithm used here is a standard DQN with discrete actions (choices of parameters). But in general I am puzzled why the action space is discrete instead of continuous, if the actions are the hyper-parameters. State transition wise, I am not sure if the states follow an action-dependent MDP transition, and therefore at this point I am not sure if DQN is the best algorithm for this applications (versus bandits/combinatorial bandits).  While it is impressive to see that RL beats many of the SOTA baselines for parameter tuning, I also find that instead of using data in the real system to do RL training, the paper proposes generating "simulation" data by training a separate DNN. I wonder how the performance would differ if the RL policy is trained on the batch real data.


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes to use deep RL to learn a policy for communication in the parameter-server setup of distributed training. From the perspective, the problem formulation is a nice contribution. 

While it is a reasonable idea and the initial results are promising, the lack of an evaluation on a real cluster, or for training more computationally-demanding models, is limiting. I fully appreciate the need to perform experiments in a controlled environment, such as the ones reported in the paper. These are useful to validate the idea and explore its potential limitations. However, to truly validate such an idea completely it is also necessary to implement it and run it "in the wild" on an actual distributed system. From my experience, although performing such experiments is certainly more involved and challenging, there can also be significant differences in the outcomes when one goes to such an implementation. Normally these are due to discrepancies between the assumed/simulated model, and real system behavior.

Is it clear that deep RL is needed for this application, as opposed to more traditional RL approaches (either tabular, with suitably quantized actions, or a simpler form of function approximation? And to ask in the other direction, did you consider using a more complex policy architecture, e.g., involving an LSTM or other recurrent unit?


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies how to improve the synchronization policy for parameter server based distributed SGD algorithm. Existing work focus on either synchronous or asynchronous policy, which often results in straggler or staleness problems. Recent research proposes different ways to make the policy design fully adaptive and autonomous. This paper proposes a reinforcement learning (RL) approach and shows promising results to reduce the total training time on various scenario. 

A major challenge is to design the state and action spaces in the reinforcement learning setting. It requires the design can be generalized to different scenario, while ensuing efficient policy learning. Compared to existing policies such as BSP, ASP and SSP, RL has an advantage to adapt to non-stationary situations over the training process. Compared to other existing approaches, RL could be fully data-driven. 

The paper formalizes an RL problem by minimizing the total training time to reach a given validation accuracy. To minimize the number of actions, only 3 actions coming from BSP, ASP and SSP are used. The state space is designed to capture similar loss curves, and to be independent of the number of workers (as much as possible). A policy network is used make decisions after trained with exiting methods. 

Numerical results validate that the RL policy improves the training time compared to BSP, ASP and SSP. Different number of works and models, dataset are also tested to show the RL policy is generalizable to unseen scenario. Although all the results are simulated in a controlled environment, Figure 4 gives a very interesting illustration showing the advantage of using the RL policy. I still have detailed comments (see below), but I find the paper well written, and the author(s) has obtained promising results.

Detailed comments:
-	The validation accuracy 88% on MINST seems pretty low to me to stop the algorithm, in particular when training multiple layer neural networks. What would happen if the accuracy is increased, can the RL approach still find a good policy? What about the validation accuracy on CIFAR-10?
-	I still have some concern of the computation time obtain the RL state per step. In particular, the time cost to compute the loss L on different weights w. How do you address this issue? Is L computed on the validation set? What is its size? This parameter seems to me highly sensitive when the policy is used for different dataset, in particular the dataset vary. It would be better to have more discussions in the paper or appendix. 
-	What is the final test accuracy on the trained models using different policies? This allows us to see whether the approach has not over-fitted to the training/validation set. 

Some typo:
Page 2 line 2 AP -> ASP
Page 5, last line 4: converge -> convergence

review_title: Official Blind Review #3

Meta-review: The authors consider a parameter-server setup where the learner acts a server communicating updated weights to workers and receiving gradient updates from them. A major question then relates in the synchronisation of the gradient updates, for which couple of *fixed* heuristics exists that trade-off accuracy of updates (BSP) for speed (ASP) or even combine the two allowing workers to be at most k steps out-of-sync. Instead, the authors propose to learn a synchronisation policy using RL. The authors perform results on a simulated and real environment. Overall, the RL-based method seems to provide some improvement over the fixed protocols, however the margin between the fixed and the RL get smaller in the real clusters. This is actually the main concern raised by the reviewers as well (especially R2) -- the paper in its initial submission did not include the real cluster results, rather these were added at the rebuttal. I find this to be an interesting real-world application of RL and I think it provides an alternative environment for testing RL algorithms beyond simulated environments.   As such, I’m recommending acceptance. However, I do ask the authors to be upfront with the real cluster results and move them in the main paper.

Decision: Accept (Poster)