Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present VER, a method that could both tackle the challenge in synchronous RL that the policy updates need to wait for all steps done in all environments (straggler effect), and the challenge in asynchronous RL that we need to explicitly tackle stale samples when do on-policy learning. Specifically, VER is an environment-dependent variable step size training method that uses variable step numbers according to environments. Evaluation experiments are done sufficiently.
strengths_and_weaknesses: Strengths
---------------------------------
This paper discusses an important problem in the RL community, i.e., what techniques should we use in the context that we usually vectorized a batch of environments in simulations to speed up the training time. Overall, the reviewer enjoys reading the author's analysis of problems in AsyncOnRL and SyncOnRL. And the motivation for variable step size rollout makes sense to me. Experiments are sufficient, and efforts are obvious.

Weakness
---------------------------------
- (1) The authors fail to describe how exactly the step length is decided. As a comparison, the baseline DD-PPO [1] paper clearly describes the whole training solution.
 - (2) Although several tasks are evaluated to test the efficiency of VER, it's hard for the reviewer to determine the claimed performance, for example, "in Habitat 2.0 VER is 150% faster on 1 GPU and 200% faster on 8 GPUs". At least VER's performance gain is not obvious in Fig. 4.

Ref:
-----------------
- [1] Wijmans E, Kadian A, Morcos A, et al. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames[J]. arXiv preprint arXiv:1911.00357, 2019.
questions: Please check the 2 points stated above in the weakness section.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an approach for bridging the gap between asynchronous and synchronous onpolicy RL to leverage the best of both worlds, preserving the throughput of the former and the  sample efficiency of the latter. They achieve this by parallely simulating N environments for different number of timesteps, thus avoiding being bottlenecked by the slowest environment. The authors evaluate their approach on habitat environments (including the more complex rearrange tasks from habitat 2.0), and empirically validate their claims. 

strengths_and_weaknesses: Significance 

-> Performance in terms of sample efficiency and training speed are critical for RL agents. This paper provides an intuitively simple approach (i.e having a budget of samples for all environments each time before the policy update, as opposed to a fixed budget for each one of the environments) that improves on both metrics. This idea can be broadly adopted by the community due to its simplicity, and thus has potential to expedite iteration cycles for all RL researchers, pushing the field forward. Further the analysis performed by the authors comparing the sample efficiency and throughput to that of prior methods (pure async or pure sync) is quite thorough and exhaustive (including a comparison to a version of their method with all micro-optimizations, except the main idea - i.e having a fixed budget for all environments). 

-> The authors include an interesting result that with access to skills not directly required for a task, the agent is able to leverage these to perform the task better given more data (Specifically, given access to navigation skills, the agent is able to perform pickplace better, even though navigation isn’t necessary for this task). It would be interesting to see if this emergent behavior can also be seen by running previous approaches (eg DD-PPO), even given more data. If prior methods require a lot more data to discover similar interesting emergent behavior, this would make the argument for adopting the proposed approach even stronger.

-> An interesting aspect of the approach is that it naturally induces a curriculum. Since easier environments are faster to simulate, samples from these are seen more often earlier on, and this helps learning. The authors mention this in passing, but I think there can be further analysis here, and perhaps a design change to specifically create a curriculum between easier / harder environments (difficulty might sometimes not vary exactly  with simulation speed, and I wonder if there are some trade-offs here that can be analyzed).  


Quality                                                                                  
-> The empirical evaluation in terms of throughput and sample efficiency are the most critical components in establishing the author’s claims, and this is done quite exhaustively, even taking account variability in the number of GPUs available for training. Evaluation is included on simpler established navigation tasks from habitat1, and also on more complex longer horizon rearrangement tasks from habitat2 (which include multiple pick-place steps). The grasp action in these environments requires contact with the object and the robot to issue a command, as opposed to the ‘magic grasp’ commonly used in habitat environments (where the object is automatically grasped when close enough to the arm). This makes the control problem significantly harder, and brings the approach closer to real world settings (though that gap is still quite large, but addressing that is outside the scope of this paper). 


Originality 

-> The idea proposed is quite simple. The focus of the paper is not on algorithmic novelty but rather on a highly performant RL system, which will be quite useful. The related work adequately covers related work in the field, including relevant asynchronous and synchronous approaches. 


Clarity 

-> The paper is clearly written and motivated, and easy to follow. In order for this work to have an impact on the community the code must be released (since the focus of this work is a better RL system, and there are some micro-optimizations included that might be critical). The code is included in the supplementary, so I think the authors intend to release it. 


questions: Questions - 

-> Does the emergent navigation skill use described in section 6.2 also happen when using prior methods like DD-PPO, SampleFactory etc, even given more data (up to an order of magnitude more)? 

-> Further analysis on the induced curriculum due to the relationship between simulation speed and difficulty of environments would be interesting. Does the proposed approach suffer in cases where the difficult environments are harder to simulate? How can this be mitigated?

limitations: Limitations are adequately discussed
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a scaling technique for reinforcement learning (RL) to speed up sampling and learning in on-policy RL methods. The paper claims to blend the benefits of SyncOnRL and AsyncOnRL to increase both throughput and sample-efficiency. The main approach is to enable multiple environments to collect a variable number of steps simultaneously without synchronization. The paper performs experiments on several simulation scenarios and analyzes the performance of several baselines.
strengths_and_weaknesses: + The paper tries to address an important problem in RL to speed up sampling and learning.

+ The paper introduces an engineering scaling method that aims that takes advantages of SyncOnRL and AsyncOnRL. 
questions: + The theoretical novelty and contribution of the paper are not clear. Why the proposed technical can theoretically improve both throughput and sample-efficiency?

+ The speed increase of VER on 8 GPU is only 70% faster than 1 GPU. Is the problem big enough to use the power of 8 GPU? Or is the algorithm for the 8 GPUs well parallelized? If 8 GPUs are not fully utilized, the motivation of using the proposed method on multiple GPUs seems not essential.

+ What is the difference between VER and AsyncOnRL? Theoretically/mathematically (not only quantitatively), why does VER perform better than AsyncOnRL?

+ The organization of the paper needs improvement. Experiment results are scattered throughout the paper, from introduction to the four sections of experiments. Some experiment setups also appear in the approach section. This increases the difficulty of understanding the theoretical novelty of the paper.

+ More details need to be provided to explain several concepts including TP-SRL and the architecture in Line 226.

limitations: No potential negative societal impact is perceived in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on the problems pertaining to SyncOnRL and AsyncOnRL and proposes a solution that combines the advantages of these methods. SyncOnRL suffers from the straggler effect, where the slowest worker decides the overall speed of execution, and the system capability is not utilized properly. AsyncOnRL solves this by taking action in an environment as soon as it is ready. Although this leads to efficient system utilization, it leads to reduced sample efficiency. 
The proposed method, Variable Experience Rollout aims to mitigate these issues. The paper points out that collecting an equal number of time steps T per environment is easier to implement but is not a necessary requirement for reinforcement learning. VER collects a variable number of steps across the different environments. This overcomes staggler effect while at the same time is more sample efficient than AysnOnRL. 
strengths_and_weaknesses: Strengths : 
* The paper works on an exciting and vital problem of accelerating on-policy RL algorithms while maintaining sample efficiency. 
* The idea is simple and original based on an ingenious observation about the training of the Reinforcement learning algorithm. It is a relatively novel idea. 
* The amount of time required to train the RL algorithms when compared to the fastest SyncOnRL is much much less while maintaining the same sample efficiency. Making these algorithms more accessible to the general audience. 
* The experimental details are comprehensive about the system capabilities, which is essential, especially for a paper that is mainly about utilizing system capabilities. 
* Clear writing and neat plots. 

Weaknesses
* Although the idea is exciting and novel, the presentation can be improved by including the following plot
    * it would be constructive to see one plot where : 
       * x-axis: Policy training time (in hours or any other suitable time unit) 
       * y-axis: Success Rate 
       * Baselines: Fastest AsyncOnRL, Fastest SyncOnRL, VER
    * It is not clear whether VER leads to improvement in training time (ie the time required for the policy to converge to a high success rate). 

* Emergent skills 
    * Although it is an interesting result, it is not very clear how the results support the main idea of the paper. In my opinion, the analysis of these skills should be part of the appendix rather than the main paper. 
    



questions: Regarding Emergent Skills
* Would these skills not emerge when trained with AsyncOnRL on SyncOnRL?
* Is there something specific to VER that leads to the emergence of these skills? Or any method trained for a sufficient amount of time can lead to the emergence of these skills?

limitations: Limitations are not discussed properly.
review_title: null

Meta-review: metareview: The paper proposes a novel method that takes the best of both worlds: synchronous and asynchronous on-policy RL methods.
The rebuttal nicely addressed the concerns of most reviewers.
Why the method makes sense and has benefits is rather straightforward and intuitive (which is a good thing!). The paper is clearly an experimental paper / systems paper with very extensive evaluations that show significant practical benefits. The method potentially has large practical impact, which is an important contribution to the community (and a valid - though less common - NeurIPS paper format). Hence I disagree with reviewer 45yB about requiring a theoretical novelty/contribution.


*** not visible to authors as posted after discussion phase ***

Agree to raise the score

NeurIPS 2022 Conference Paper1458 Reviewer ep2X

18 Aug 2022

Excellent work, impressive results!

I will raise my score.

confidence: Less certain
Decision: Accept