Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors investigate the interplay between the training speed/performance of an offline RL method and the total number of environments used during training. A large number of RL papers default to online RL methods, specifically PPO, because of the relative ease of these methods, however, offline methods should be more sample efficient by virtue of reusing samples from the replay buffer. The authors implement an offline RL training scheme, PQL, to leverage recent advances in GPU simulation environments to accelerate the training process while achieving comparable results with online methods. PQL breaks up the data collection and training pipeline into 3 processes, an Actor, P-learner, and V-learner. Since the simulation environment is located on the GPU all 3 processes may be co-resident on a single GPU or distributed across multiple GPUs on a single node. The placement of processes on different GPUs and the ratio of data collection vs training speeds are studied empirically and optimal settings tend to vary, as expected, across environments. The authors also study empirically how the batch-size and replay-buffer sizes influence the training process as the number of processes changes.
strength_and_weaknesses: Strengths
--------------------
- The fundamental question being addressed by the paper is interesting. The training dynamics of RL algorithms using GPU-simulated environments may be dramatically different from those of CPU-simulated environments. The abundance of environments producing data at a rate much higher than previously studied coupled with lower communication overheads, because data is generated directly on the device, provide ample motivation to rethink training algorithms and hyperparameters typically used by members of the community. In short, is PPO the best off-the-shelf algorithm in light of the differences between training environments?
- The authors demonstrate faster convergence for their offline training method compared to PPO on several GPU-simulated environments.
- The influence of several hyperparameters, such as batch-size, replay-buffer size, the number of training environments, and the number of GPUs were studied as part of the empirical results.

Weaknesses
-------------------
- The paper is empirically heavy but theoretically lite. There's no motivation underlying the choices for exploration of the design space of the parallel training environment hyperparameters.
clarity,_quality,_novelty_and_reproducibility: The writing is easy to follow and clearly conveys the intended motivation and goals of the authors. The quality of the empirical data is good (though I found it a bit difficult to read some of the legends in Figures 3, 4, 7, 8, 9, 10). The novelty lies in the focus on training an off-policy algorithm with a large number of environments vs the typical setup with an on-policy algorithm and a relatively small number of environments (unless you have access to a large distributed system). I believe the results should be reproducible given access to the implementation.
summary_of_the_review: For researchers that do not have access to a large pool of computational resources the ability to take maximum advantage of the resources they do have on hand is imperative. As such, on-policy methods typically underutilize GPU resources because they cannot generate a sufficient amount of data to keep the GPU saturated. Though many works have tried to address this issue by increasing the number of environments this typically requires an increase in the number of CPUs as well. I think it is quite interesting to try taking full advantage of the GPU-centric training environments to better utilize the hardware and investigate the interplay between the number of environments and the training dynamics. This type of investigation seems to be inline with the spirit and direction of the Issac Gym GPU simulator project.

The improved training time to accuracy is visible in a number of the plots. In Section 2 the authors reference an improvement in regard to both the sample efficiency and training times of off-policy algorithms but when I think of sample efficiency I would like to see plots of the return vs the number of environment interactions. The authors noted in section 4.6 that the volume of data generated by the simulators is so high that data in the replay buffers are being replaced rapidly causing the method to become more on-policy. I assume this means the sample efficiency also resembles that of an online policy in some cases, is that correct? Does this training scenario detract from one of the selling points of offline algorithms to reduce the volume of required training data?

In the discussion section the authors mention not using many modifications that are typical of modern offline algorithms. Was it too difficult to add these features on top of the current implementation? I feel that omitting these features could be a mistake since top-performing offline algorithms use them so without those reference points it's more challenging to compare PQL performance against prior work.

A concern is that the primary contribution is the framework that implements PQL, however, this artifact mostly builds on existing work (Issac Gym, PyTorch, etc). This diminishes the novelty of the work and makes the results appear more informative rather than a novel contribution that rises to the level required for publication at the current venue.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to run thousands of simulations of a RL environment to speedup Q learning. Unlike previous efforts, that leverage distributed computing to run agents on large numbers of machine, they propose to leverage a single workstation running numerous tasks in parallel.
The paper contributes a novel framework, called PQL, that can train RL policies faster than previous efforts without requiring a large amount of compute resources.
strength_and_weaknesses: Scale is a major force behind the success of reinforcement learning. However, unlike numerous previous approaches that achieve scale by distributing the computation over large amounts of servers, PQL proposes to leverage a single workstation by multiplexing the simulation of thousands of environments along with a value function and a policy on a single GPU.

The approach appears to be similar to that of DDPG, and the implementation is based on an existing framework targeted for distributed training that was repurposed to run multiple simulations of the Isaac Gym environment on a single workstation, which is the main contribution of the paper. However, several points are unclear:
 * Where do the wall-time gains come from ? Is Isaac Gym more efficient when running multiple simulations in parallel, or is there something else driving the performance improvement ? Ideally, I'd like to be able to answer the question: what aspect(s) of PQL approach is inherently better and drives most of the performance improvements.
 * How applicable is this beyond Isaac Gym ? Would it help with other simulation based gyms ?
 * How would the PQL approach compare against other distributed RL implementations such as IMPALA/seed-RL if these implementations were also repurposed to target the Isaac Gym ?
clarity,_quality,_novelty_and_reproducibility: The empirical evaluation of PQL shows that it's able to learn faster than other approaches in 6 simulated environments. However, the presentation of the paper doesn't help understand how this result is achieved. Figure B.2 indicates that PQL is roughly as sample efficient as DDPG which is expected since they use similar off-policy approaches, but more sample efficient than PPO (on-policy), which is also expected. Figure 3 shows that PQL is faster than PPO and DDPG on the same hardware, so the implementation of PQL has to be able to do more work per second. How this is achieved is not explained. Is the GPU utilization increased and if so, how ? Does the framework better balance the workload between the CPU and the GPU ? Is one of the code bases that PQL leverages more efficient than the ones leveraged in the PPO and DDPG implementations used as baselines, thus reducing the training time ?

It's not clear what figure 5&6 demonstrate. If the relative ratios of total rollouts, updates to the Q function, and policy updates had been kept constant while the number of environments changes, and if the noise levels had been modified to avoid increasing or decreasing how much exploration is performed, we could have discovered the presence of absence of a communication bottleneck in the implementation. As done in the paper, there are too many unaccounted factors to be able to draw any conclusions.

The batch size experiment needs to be better explained. How is the batch size applied ? Is it applied solely on the Q and policy networks, or is it also applied to the actors ? It would have been better to show how the efficiency of the 3 components of the approach changes as batch size varies.

Some aspects of the paper, such as the beta factors in section 3.2, are described in more detail than necessary: it's enough to define them, any reader can infer what they do from there. On the other hand, some mechanisms need more explanations. For example, why is f_a defined as the number of rollouts per environment per unit of time instead of the total number of rollouts per unit of time ? The later could make the setting more robust to changes in the number of environments operating in parallel, and could have decreases the number of hyper-parameters. Similarly, in section 3.2, how are the values of a_u and a_l determined and how would the noise injection process change if the actions were discrete instead of continuous ? 

Please increase the font on the figures. The legends are very tiny, and I suspect impossible to read on a printed version of the paper.



summary_of_the_review: PQL is an interesting paper that demonstrates that a better implementation of the Q learning strategy can make it competitive against PPO from the point of view of training time. However, the author need to provide an intuition as to why their approach is inherently more performant. Furthermore, the evaluation needs more work to clearly demonstrate how the authors achieve superior performance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes paralleled Q-learning, a three-component framework to accelerate Q-learning. The design of the components enables concurrent execution, and experiments on Isaac Gym and vision-based benchmarks against other state-of-the-art algorithms such as PPO, DDPG and SAC show significant improvement as measured by cumulative return. 
strength_and_weaknesses: Strength:

1. The paper is written very clearly, with all the sections accurately describing what the algorithm is about. The relationship between the different components in the framework has been clearly demonstrated by graphs, and the intuition behind the components as well as choices of returns/RL structures is easy to follow.

2. The experiments are comprehensive, and provide sufficient details for reproducibility as well as ablation studies. In particular, the discussion on the ratio of beta values(indicating update frequency comparisons) is very informative.

Weaknesses:

1. The algorithm is named Parallel Q-learning. However, the key of the algorithm lies in decomposing DDPGs/SACs and making use of techniques of improving Q-learning to better estimate Q-functions in the learning process. It would be helpful to emphasize in the abstract and intro where the novelty lies since the title may be a bit misleading.

2. While the results of the experiments are largely self-contained and informative, it would be helpful to include the performances of PQL with SAC as the underlying algorithm. Since it is indicated in the paper that the PQL framework can be naturally extended to SACs, the inclusion of such ablation studies would answer the question how much the effectiveness of the PQL framework would depend on the choice of off-policy algorithms.    


clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, and the research question is interesting with high relevance. The novelty appears to be non-trivial, and the results are detailed enough for reproduction.
summary_of_the_review: I would rate this paper a 8, given its scope of contributions and good quality.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Motivated by the data inefficiency problem of off-policy algorithms, like PPO, this paper proposed a parallel version of the Q-learning algorithm. Specifically, this approach uses three parallel processes, namely actor, V-learner and P-learner, to perform a DDPG-like learning process. Empirical evaluations on six Isaac Gym tasks has demonstrated that the proposed approach has better learning efficiency than baselines like PPO, SAC, etc.
strength_and_weaknesses: Strength:
1. The evaluation of the proposed method is extensive and involves many different tasks.
2. The parallel simulation environments considered here are interesting and they might be important to greatly reduce the training costs for RL. 

Weaknesses:
1. The introduction of the three parallel processes is quite abrupt without adequate discussions about the motivation of such considerations. It's unclear why these design choices are inspired from existing methods like DDPG or other variants. Furthermore, what's the challenges to design a Q-learning algorithm for parallel environments? Why a direct extension of DDPG can not work?
2. It's not clear what's the key difference between parallel environments on a single workstation and parallel environments on a computer cluster. Does such difference really matters in the design of the algorithm? Why?
3. The discussion of the main techniques is too brief and can not convince readers the design rationales behind the proposed algorithm.
4. Another suggestion is to write the proposed method as an algorithm so that it would be more clear the algorithmic improvements made over existing approaches. Stressing the design considerations that make the proposed approach more efficient than others.
clarity,_quality,_novelty_and_reproducibility: The writing of this paper has a considerable space to be improved. The novelty of the proposed approach over existing methods requires further elaborations. The reproducibility is good.
summary_of_the_review: Despite an extensive amount experiments have been conducted, the writing of this paper requires a considerable revisions to convince the contributions of this paper and illustrate how each design consideration leads to performance improvement over existing methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This was a borderline paper. On one hand, reviewers appreciate the value of investigating the computational regime in RL where a GPU-based simulator is available, and results showing that a Q-learning based approach is competitive with PPO are interesting. On the other hand, reviewers were generally unsatisfied with the framing and presentation of the paper. While improvements made by the author response were appreciated, the main requests from reviewers were still for a more systematic approach to the systems considerations of the paper: what were the motivating design principles and theoretical basis for the implementation? More details about why the system is more efficient; What findings from these results are robust and transferable? A secondary concern was around the framing of the contribution, where reviewers found there to be less originality in the framework itself as compared to the implementation and systems considerations.

I'll also note that authors were unhappy with the level of engagement from the reviewers, and I agree it is unfortunate. However, all reviewers attended and participated in the in-person virtual discussion, and there was a relatively clear consensus amongst the reviewers coming out of the discussion.
justification_for_why_not_higher_score: Would like to see a more systematic approach around the design and evaluation of the system, including design principles, alternatives considered, crucial assumptions, and robustness of findings to violations of the assumptions.
Decision: Reject