Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces EnvPool an open-source software for parallel environment execution. EnvPool aims to speed up the execution time of existing reinforcement learning (RL) benchmarks, which can be a bottleneck for performance.  EnvPool can be run in synchronous or asynchronous mode and run faster than similar software. EnvPool is implemented in C++ and custom environment can be implemented in C++.
strengths: EnvPool is the first parallel environment execution engine which allows for asynchronous environment execution. This allows EnvPool to significantly increase the speed of environment execution in parallel.  Due to this parallel environment execution, training times for RL algorithms can be reduced, increasing the pace at which research progresses.


weaknesses: EnvPool does not seem to be a dataset or a benchmark, but rather software to improve the efficiency of working with benchmarks. The NeurIPS datasets and benchmarks track is for datasets/benchmarks and a forum for discussion on how to improve the development of datasets/benchmarks (see https://neurips.cc/Conferences/2022/CallForDatasetsBenchmarks).  Although I have no doubt about the utility of EnvPool, I am unsure whether EnvPool meets these criteria.

correctness: The paper does not provide enough experimental details to fully understand the experiments that were run, nor to gauge the correctness of these experiments.  Furthermore, uncertainty of results is not discussed in detail.

For experiment 1 on pure environment simulation (section 4.1), the number of runs performed is not clear, and so I assume only a single run of the experiment was performed. Were multiple runs performed? If not, why was only one run used? Why not report mean performance improvement with some measure of uncertainty over many trials? Perhaps this would show the benefits of EnvPool are better or worse from what is shown here.

In experiment 1, the best performance of EnvPool (async) and Sample-Factory are reported, tuned over multiple `num_envs` given a constant `batch_size`.  Why was this choice made? 
Why was a consistent `num_envs` not used for all execution engines in this experiment? Setting `num_env` for some execution engines while tuning it for others seems to bias the results in favour of those for which this hyperparameter was tuned.
Wouldn't using a constant `num_env` for all algorithms be a fairer comparison?
Perhaps I am not understanding the experimental details correctly. Can this experimental decision be qualified?

For experiment 2 on end-to-end agent training (section 4.2), multiple experimental details needed to gauge correctness are also lacking. How many runs were used to generate these results? What are shaded regions describing? How were PPO's hyperparameters tuned?  From the text, it sounds like hyperparameters were tuned to increase EnvPool's throughput, why was this choice made? Wouldn't this choice give the baseline execution engines a disadvantage?

The purpose of experiment 2 is to show that EnvPool can train algorithms faster by an increase in environment throughput. Unfortunately, this experiment may give EnvPool an advantage over the baselines execution engines. When running PPO on EnvPool, the paper clarifies that there will be a reduction of the usage of stale data (the definition of which is unclear).  This does not occur with the baseline engines. Perhaps the experiment would be better if this confounding factor was removed. Maybe the increase in performance (equivalently, the decrease in required training time) is not due to EnvPool but rather due to the data used by PPO. Furthermore, on line 256, what does it mean to reuse data with 10 epochs? Does this mean PPO is updated with the same data for 10 updates?

A number of experiments in the appendix use different underlying environments for each execution engine (MuJoCo v4 vs v2) and then compare the results. Why was this choice made? Wouldn't it be a more fair evaluation if both execution engines were run on the same environment? 
clarity: The paper is fairly well written and understandable, the only portion of the paper that was unclear was lines 205 - 214. What are these bullet points referencing? I have assumed that they refer to the choices made in experiment 1, but this is not clarified in the paper. Also, the paper refers to "workers" without defining what these are. I assume they are "CPUs".
relation to prior work: The paper does a satisfactory job in relating to prior work.

additional feedback: _Small suggestions that did not impact the score of the paper:_
- Figure legends would be improved if filenames were replaced by English phrases.
- L63-64: This sentence does not make sense to me. Perhaps you mean "EnvPool increases environmental throughput without affecting the learning algorithm"?
- L66: "in the previous literature" -> "in the literature"
- L115: Replacing "return" with "returned value" would make this sentence more clear, since "return" has a specific meaning in RL.
- Figure 2 could probably be placed side-by-side to reduce the space it takes up.
- L254: "we can search alternative..." -> "we can search an alternative..."
- L255: "PPO by default use..." -> "PPO by default uses..."
- L256: "re-use" -> "re-uses"
review_title: Useful Software with Questionable Experiments

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: - The authors present `envpool`, a C++ tool with python bindings to parallelize RL environments.
- The tool is able to wrap arbitrary environments that do not have to be parallelizable using matrix operations.
- The environments can be stepped in a synchronous way and also asynchronously to speed up the evaluation even further.
- The documentation is well developed and readable.
strengths: - Parallelizing arbitrary RL environments has the potential to speed up research in many directions and enable researchers to try more of their ideas in a shorter span of time.
- Focussing on the single machine use-case improves the situation for the usual RL researcher without a large cluster at hand.
- The API is simple to use and can be integrated easily into existing RL training setups.
- The most relevant environments are implemented and can be used off the shelf.
- The environment parallelization can also be optimized via XLA which is especially useful for jax-based agents.
weaknesses: - Adding an environment requires the C++ build setup and several steps, which is not as easy as the authors make it sound in the paper. However, there is sufficient documentation to guide the researchers through the process.
- The XLA optimization feature is not evaluated but would be really interesting as well for the paper.
correctness: - The paper's presentation and evaluation of the tool is sound and highlights the characteristics of it well.
clarity: - The paper is well written and the structure makes it easy to follow.
relation to prior work: - Other approaches to parallelize environments as well as their shortcomings are described in enough detail.
additional feedback: - The authors should note, what the shaded area in e.g. figure 4 is.
review_title: The paper presents a helpful tool to accelerate research in RL, though its usefulness is limited a bit by the complex way of integrating new custom environments.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The EnvPool system proposed in this paper achieves improved throughput in some cases by programming the environment in C++. 
strengths: The benchmark can be suitable for different RL environments, and the issue of paralleling RL environments is addressed well.
weaknesses: 1. The authors claim in Section 1 that " we focus on tackling a common bottleneck in the RL training system in this paper: parallel environment execution. To the best of our knowledge, it is often the slowest part of the whole system but has received little  attention in the previous literature. ". Consequently, I think the in-depth analysis of the impact of environment execution throughput on overall training speed should be supplemented.

2. In Section 1, the author claims that EnvPool is 14.9/19.2 times faster than the “Python implementation [3]” , but do not state how the Python implementation utilizes hardware devices. Reference [3] is only the simulation environment OpenAI gym, not a reinforcement learning training framework.

3. Compared with the related work of using GPU for environment parallel acceleration mentioned in Section 2, the speed of this research utilizing CPU for environment parallel acceleration should be much slower. The proposed system also requires low-level editing of the environment (converting it to C++ code), which is also limited in scope.

4. In Section 3 describing the specific methods, the proposed parallelization acceleration methods are basically generally adopted, and lack novelty and contribution.

5. In the experiments in Section IV, experimental comparisons with state-of-the-art methods using GPU acceleration are lacking.

6. Based on Figure 3 and Table 1 in Section 4.1, the sample throughput of Sample Factory is roughly comparable to that of the proposed EnvPool system. Howerer, why is Sample Factory not included in the overall training comparison in Section 4.2? Why did the authors switch to the RAY system that was reported to be inefficient for comparison? In my experience, doubling the experience sampling speed may not result in a significant improvement in overall training speed.
correctness: YES, but mostly correct.
clarity: YES.
relation to prior work: YES.
additional feedback: More environments may need to be integrated, such as grid worlds.
review_title: A RL system with highly parallel reinforcement learning environments.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This system aims to reduce the time to collect transitions for on-policy RL methods, like PPO. With the rapid training enabled by EnvPool, researchers can prototype new ideas faster, thus accelerating the progress of RL research. Currently, most paralleling rollout environment wrappers suffer from inefficient Python thread execution. EnvPool solves this problem by building the paralleling system with C++, which can access the computation resources in a more straightforward manner than Python. The experiment shows that EnvPool can greatly reduce the training time, compared to Ray and  Sample Factory, especially in async mode. The code is released, and tutorials and instructions are well documented.


strengths: The rollout efficiency is a huge concern for RL researchers, especially those trying RL on complex control tasks, *i.e.* Robotics. Due to the inefficient simulation and frequent communication between GPU and CPU, it usually takes days or even weeks to tune parameters and verify ideas. Although recent works such as IsaacGym achieve parallel GPU simulation to address inefficient simulation and eliminate communication costs, there are still a large number of environments/simulators that are built to run on CPU. Therefore, it is **valuable** to develop tools, like EnvPool for boosting the rollout efficiency of CPU-based simulation.

This project is highly developed with a well-constructed codebase and comprehensive documents. Currently, it already supports many commonly used RL environments, like Mujoco and Atari. The authors conduct experiments in these environments. It turns out that EnvPool outperforms other paralleling schemes by a large margin.



weaknesses: The weakest part of EnvPool is that it is difficult to be applied to custom environments/simulators. Although EnvPool shows astonishing performance on Gym and Mujoco tasks, for researchers using these environments, the rollout efficiency is not a sticky problem. As I mentioned before, there are a lot of custom environments that require powerful paralleling schemes. Therefore, it is more important to make the combination between EnvPool and custom environments easy than to officially support common environments. 

Actually, I would like to try it on our in-house simulator which is built on the Bullet Physics engine, and report some results. However, the process to add a custom environment is too complex, so I give up. Also, I found that it currently only supports the Linux platform. Is there any plan to make it available on other platforms, like Windows? There are some cross-discipline researchers whose simulator is built on Windows. Therefore, I suggest Windows support, at least, should be added.

Overall, I really like this valuable work. If the authors could describe their future developing plan or make some commitments to address the issues (especially the compatibility issue) in the revision, I think it deserves a higher score.
correctness: The project is solid and the experiments are convincing. 
clarity: It is easy to read.
relation to prior work: The survey is comprehensive.
additional feedback: N/A
review_title: A system for paralleling RL environments by using a C++-level thread pool.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes *EnvPool*, the C++-based framework to accelerate OpenAI Gym-style RL environment execution with efficient parallelization. EnvPool has both Synchronous and Asynchronous communication to the thread, and especially Asynchronous EnvPool significantly speeds up FPS of RL training in any kind of environment (MuJoCo/Atari, etc) on any computers/servers (Laptop, Workstation, DGX-A100). For example, PPO on EnvPool can train MuJoCo Walker2d within 10 minutes and Ant within 5 minutes, which seems faster than the existing parallelization frameworks. Furthermore, in contrast to recent hardware-accelerated simulators (Brax, IsaacGym), EnvPool seems to depend on GPU/TPUs less and can support existing environments widely used in previous RL research.
strengths: - EnvPool tries to solve the execution-time bottleneck in RL research. This seems to be an important contribution to the community, since the researcher can run the try-and-error cycle faster.
- EnvPool can train Ant in 5 min, which seems comparable to hardware-accelerated simulators, such as Brax or IsaacGym.
- Although Brax and IsaacGym have their own eco-system of the environments, EnvPool supports existing environments widely used in previous RL research, which is beneficial for benchmarking.
weaknesses: - This framework is written in C++, not in Python, a standard/widely-used ML community programming language. This may limit the actual developers/users from RL research communities (e.g. Brax is all written in Python library: Jax. IsaacGym supports rich Python bindings.).
- If we want to use self-made environments, we must design additional environment wrappers in C++ per novel environments. This might be a bit costly, compared to the existing parallel engine (Subprocess, Sample Factory, etc), which can be used in common across environments.
- As mentioned in Section 5, it might be unclear whether EnvPool is superior to the existing large-scale RL works (i.e. highly-distributed IMPALA, Seed RL)
correctness: - The comparison and benchmark of execution time/performance seem well-designed and fair.
clarity: - This paper is well-structured and easy to follow.
- It may be better to fix the caption of Figure 6 and Figure 7. Does blue *rl_games (tuned)* also use EnvPool, right?
relation to prior work: - This paper compares the proposed EnvPool with a variety of existing parallelization engines (e.g. Ray, Subprocess, Sample Factory) on a variety of environments (Atari, MuJoCo) and RL frameworks (baselines, rl_games, clean rl, acme).
- As mentioned in `Summary And Contributions`, EnvPool seems to depend less on high-end GPU/TPUs and can support existing environments widely used in previous works, compared to recent massively-parallelized hardware-accelerated simulators, such as Brax or IsaacGym.
additional feedback: I feel the development of EnvPool seems an important contribution to the community. One concern is sustainability since the users should write a C++ wrapper, not an ML-friendly programing language like Python. This may limit the number of users/developers.
review_title: Review: Speeding up existing environments is good contribution to the RL community

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: EnvPool speeds up execution of Reinforcement Learning environments by running them in parallel. EnvPool uses C++ threading for environment execution but provides a standard python interface for learning at the batch level. EnvPool improves upon other implementations by allowing either synchronous or asynchronous batching. The authors demonstrate speedups in both raw environment frame rates and algorithmic training time without loss of sample efficiency.
strengths: The authors have demonstrated significant speedups in some of the most common environments used for reinforcement learning: Atari and MuJoCo. Completing training on these environments can be a significant barrier to research: researchers may need to run on many different environments over many different seeds. Increasing the speed of training may make RL research accessible to more researchers and improve the field as a whole.

The generic interface provided by EnvPool makes adoption simple for those already using environments supported by EnvPool. For new environments, EnvPool provides significant documentation on developing compatibility.
weaknesses: Because some engineering effort is required to adapt new environments to EnvPool, it may not be useful to all researchers in the field. In addition, it is unclear how much improvement EnvPool will provide to other types of environments, in particular for more GPU intensive environments.

Many RL algorithms are bound by training compute rather than sampling time, and in this case EnvPool also may not provide significant benefit.
correctness: Yes, the claims appear correct. The evaluation methods use multiple trials and clearly state their limitations.
clarity: Yes, the paper is clearly organized and well written.
relation to prior work: Yes, comparisons to prior work such as SampleFactory and Ray are discussed in sufficient detail.
additional feedback: None
review_title: EnvPool addresses a significant bottleneck for RL

Meta-review: metareview: I would like to thank the authors for their submission and for addressing the reviewers' comments. I also want to thank the reviewers for their helpful feedback.

The paper introduces EnvPool, an open-source system that parallelizes the execution of reinforcement learning workloads on a single node. The code is well documented and is supported on GitHub. Some reviewers were able to replicate the results of the pure environment benchmarking using documented code. This work is very important, especially due to improving the rollout efficiency, which is a pain point for RL researchers that often spend days waiting for their RL training to finish. The authors do a good job of addressing the majority of the reviewer's concerns and incorporating these changes in the final draft.

Main Pros of EnvPool:
* Has support for some of the widely used simulation systems like Atari/MuJoCo and can be used off the shelf. 
* Speeds up the execution of RL benchmarks, which speeds up the research cycle.
* Has a Python API, which is simple to use and can be integrated easily into existing RL training setups.

Main Cons of EnvPool:
* It only supports Single node execution. With bigger and more complex models, a large portion of RL workloads is running on multiple machines.
* Supporting a custom RL environment requires writing C++ code which is not user-friendly.
* In general, this work seems to be targeted at the case where the environment is extremely cheap to simulate. Such simulators are not very common outside of research. It would be great if the authors explain how (or if) this work can be applied to more complex environments that cannot be simulated. 

Considering the pros and cons above and the reviewers' insightful feedback, I recommend accepting this paper and presenting it as a poster in the program. Congratulations! 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept