Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper shows that batch simulation can accelerate reinforcement learning in 3D environments. Batch simulation accepts and executes large batches of simulation requests at the same time on one accelerator. The authors demonstrate that this technique can substantially speed up the processing and achieve ~100x speed up in convergence. They also propose minor-changes to DD-PPO to speed up the convergence even further. The authors also included the code which is always appreciated. 

I believe the problem that this paper is addressing is quite important. Long training time of RL agents is a big issue which makes evaluation of the current methods harder and, as authors mentioned in the introduction, unaccessible to many people due to limited number of people who have access to large clusters of computation.

However, I find the paper limited in both novelty and impact. The idea of "batch" simulation is not new and it has been shown before, not only for complex environments (check Sample Factory from ICML 2020 for an async version) but also for simple ones such as atari when ported to the GPU (check NVlabs CuLE from NeurIPS 2020). Also given the limited number of the environments that the paper covers and (probably) the complexity of adding new ones, it has limited impact. 

Overall, I find this to be a great technical paper. The authors achieved remarkable convergence speed for PointGoal navigation problem and I believe that upon releasing the code some researchers will benefit from it. However, I find it lacking in novelty and future impact as a scientific paper for a top machine learning conference. 
review_title: limited novelty and impact

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a high-performance reinforcement learning system for training agents in 3D simulated environments.
The system uses batching to significantly increases both simulation time and training time.
It also proposes tricks for large mini-batch policy optimization.

The experiments do evaluate several important aspects of the proposed method. However, I want to see how generalizable the system design is. The current paper only evaluates the system for one environment and one algorithm, which is a major weak point.
review_title: A Nice Enginnering Work

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work extends the Habitat simulator to perform large batch training. Although this is mainly an engineering contribution, the approach is useful for the community. However I have some questions:


1. You do not appear to specify the CPU memory requirements of this approach, I believe each Habitat environment instance requires a significant amount of memory, how many independant instances can be loaded in paralel with this approach and what is the memory requirement? What is the value of K that is discussed on the last paragraph of page 4?

2. How significant is the use the Lamb optimizer, can you include an ablation of this, do the other baselines use this optimizer?
3. In table 2 you do not show the performance of the Baseline++ network architecture
4. In table 2 some entries have standard error and some do not, if these results are now available please can they be included.
5. Would it be possible to update figure 3 with comparisions of the baseline methods.

review_title: A useful (engineering) contribution to the Deep RL community, that lacks some comparisons and details.

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a novel RL simulator design that executes large batches of simulation simultaneously. The core innovation lies in an efficient GPU-based batch rendering and an improved training procedure adapted for large mini-batch optimization. Compared to strong baselines, the method achieves a significant boost in efficiency with slightly reduced performance. 

Pros:
* The paper tackles a very critical speed bottleneck in practice for RL training in a simulation environment. 
* Very significant speed improvement
* The paper is well-written and presented

Cons:
* The paper misses some important ablations to showcase each technical component's contribution to the final speedup. 
* Evaluation is only conducted on PointNav task; many technical design choices seem to be tailored for such a task, e.g., very low-resolution images, etc. 
* It seems the choices of DNN architecture, policy optimization, and batch rendering are highly coupled. It's not clear to me if it is possible to use batch-renderer off-the-shelf to accelerate the majority of RL algorithms' training. 


I found some ablation is lacking, making it unclear how much each part contributes to the final acceleration / reduced performance. For instance, it is unclear to me how much the frame rate increases by changing the resolution from 256x256 to 64x64 and how much performance drop it brings. Further, how much acceleration of BPS is due to the simplified DNN architecture choice (shallow layers, SE blocks, SpaceToDepth, etc.)

In many RL tasks, it might be necessary to render high-resolution images. Thus it's essential to know if the proposed method is suitable for such a job. Could you provide an acceleration-vs-resolution curve of the proposed batch renderer?

It seems the simulation takes little time in breakdown compared against rendering and network inference/training. Is this framework suitable for heavy non-rendering simulation jobs, e.g., if actor simulation or physical dynamics simulation is involved? 

Could you justify whether this proposed BPS is also suitable for multi-agent RL training, in which multiple agents might share one environment?

The author should elaborate a bit more on why large-batch training harms sample efficiency. 

Could you discuss the batch rendering acceleration compared to other graphics techniques to improve rendering efficiency, such as level-of-details (LOD), mesh simplification, DLSS, etc? And if the proposed batch rendering could be used together with these techniques?

-------------------------------------------------------- Post Rebuttal --------------------------------------------------------------------------------------------

The author did a great job in terms of addressing most of my concerns and answer all the questions. I also like the updated paper. The update reflects most major concerns from the reviewers. Thus I would like to raise my score to 7 and would like to champion this paper. I do think the paper would potentially provide great value to the community not only due to its open-source effort but also as a general approach to improve simulator efficiency for RL, despite it's not "novel" methodology-wise.  



review_title: Large Batch Simulation for Deep Reinforcement Learning

final_score: rating: 7: Good paper, accept
review_content: The paper achieves a significant speed-up for the PointGoal Navigation task by,
- Batched simulation of  environments (N > #CPUs)
- Batched rendering of multiple environments with a single rendering request, effectively drawing the environments as one big frame
- Reusing scene assets between environments being rendered in a way that doesn't hurt sample diversity but allows more environments to be rendered at the same time
- Using multiple accelerators
- Shows that comparable results can be obtained faster

The paper is well-written and clear. Similar ideas have been tried out but the combination of the ideas is well done and with a large efficiency improvement. One downside is that the setup is not easily applicable in general.

Pros:
- Well written and clear.
- Large efficiency gain with the same amount of resources. Allowed faster research iteration.
- Analysis on runtime (4.5) is clear and insightful.

Cons:
- Not easily transferable to other environments.
- The paper would benefit from plots showing agent performance over time and plots that shows how sample efficiency hurts with scale and increased batch size. As well as stability across time.

Comments:
- In abstract, it is mentioned one GPU setup achieves 19k FPS and 8 GPU setup 72k FPS. These two numbers are with different types of GPUs which makes it look like the speed-up is only 4x going to 8 GPUs when it's really 6x (72000/12900) and possibly more if batch size per GPU is kept the same.
- Wrt. section 3.3, it's mentioned that an efficient network is used (Resnet + 2xLSTM) compared to a less efficient CNN + LSTM network and that it has lower capacity. I assume most of the capacity in the CNN+LSTM network is from initial layers with the large input and a FCN before the LSTM?  What effects does the changes have? e.g. does 256^2 -> 64^2 alone hurt in any way? I suspect the Resnet makes up for some of it?
review_title: Well written paper on improving resource efficiency by batched environment simulation and rendering

Meta-review: This paper proposes techniques to lower the barrier to run large scale simulations under resource (compute) constraints. The key idea is to do batch simulation and policy learning on 1 or more GPUs without sacrificing the fps rate for rendering (~20k fps on 1GPU). The proposed setup and methods are evaluated on the point navgiation tasks on two environments namely Gibson and Matterport3D. One of the key ideas for rendering is to render a big tile of images for separate instantiations of the environment in parallel. This gives big speeds up to rendering and policy optimization. 

${\bf Pros}$:
1. Large number of FPS with smaller compute budget. Large scale Deep  RL research has been difficult to democratize due to the need for big compute budgets. Although this paper is more heavy on the engineering side, I think it can greatly accelerate research and therefore seems like a good fit for the ICLR community to consider. 

2. The paper and proposed steps are clearly written and justified

${\bf Cons}$:
1. This method is limited to environments where the observation space follows a particular structure. This is perhaps the biggest limitation of this approach but the underlying assumptions are reasonable and quite a few realistic environments will fall into this category.  
 

During the rebuttal and discussion period, R2 raised concerns about ablations but was satisfied with author's response. R5 raised concerns about other prior work - CuLE (Dalton et al, NeurIPS 2020). However, this paper is concurrent work and does not tackle 3D simulation rendering as done in this paper. For these reasons I believe the paper does not have any big red flags or pending concerns. 
Decision: Accept (Poster)