Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper analyzes various design decisions made by PPO, a popular actor critic policy gradient method, and proposes various fixes to the existing algorithm/architecture to improve the performance. Specifically, they study the impact of the objective noise levels on the policy training and how it interacts with the various design decisions like the network architecture, return estimation, batchsize etc. They use the analysis to propose improvements to the algorithm and the architecture which results in improvements to the policy performance when tested on the Atari-5 benchmark.
strengths_and_weaknesses: **Strengths**
1. The paper provides simple, well motivated fixes to PPO/PPG that demonstrate improved performance on the Atari Benchmark!
2. The paper is very well written and easy to follow. I also really liked the motivating example in section 3.1 to demonstrate the impact of noise scale on the architecture.
3. I liked the careful ablations in sections 6.1 and 6.2 analyzing the impact of the specific hyperparameters.
**Weaknesses**
1. Although the paper mentions that they leave the continuous control tasks for future work, I would still have liked some analysis of it in this paper itself to better understand the implications on those settings as well since PPO is widely used there.
2. There have been various other papers in the field analyzing the various aspects of PPO over the past few years. I would have liked a more thorough review of the literature especially along those lines and see this work placed in the context of those papers.


questions: I think both the points in the Weaknesses sections are fixable and would appreciate the authors making amends along those directions.

**Minor Comments**
1. I think the following was a typo : Line 185 : policy -> value 
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a dual network architecture extension of PPO. It considers three modifications: a separate neural network for value function and policy, calibration of bias-variance trade-off, and constrained distillation. The method is evaluated on a new Atari-5 environment and compared with a similar baseline Phasic Policy Gradient (PPG).

strengths_and_weaknesses: ### Strength:
- Tackle the important problem of improving the policy gradient method, which has wide applicability
- Overall, well-written paper and easy to follow.
- Show some improvement over baseline PPG on the Atari-5 benchmark.

### Weakness (Details in Questions):
- Lack of empirical evidence that demonstrates DNA’s advantage.
- Missing experiment on Procgen benchmarks where the baseline PPG was evaluated.

questions: How is the combined score in Figure 6 achieved? The performance improvement is visible only in the “Phoenix” environment and overlaps in other environments. Any aggregated metrics, such as the probability of improvement proposed in [1], might give a better understanding. How many seed run was used to generate Figure 6? It is mentioned that “multiple seeds run” in line 175.

The baseline PPG was heavily tested on the Procgen benchmark [2] for sample efficiency. Thus a comparison of the Procgen benchmark seems critical to assess the significance of the proposed method. Therefore, I suggest adding the experiment on Procgen to verify how the proposed method performs on existing and well-established benchmarks in addition to the newly proposed Atari-5.

limitations: Yes. Limitations and potential societal impacts are discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper identifies that the noise level between learning the value function and the policy in reinforcement learning can have an order of magnitude difference. The paper thus proposes to learn these two tasks independently followed by a constrained distillation phase. The paper also argue that the policy gradient noise level can be reduced by using a lower variance return estimate. On the other hand, a lower bias estimate should be used to reduce the value learning noise level.

strengths_and_weaknesses: Strength

The paper demonstrates the issue of jointly learning two tasks with different noise level in a simple motivating example. The motivating example clearly illustrates destructive interference and provides a great starting point for the discussion in the remainder of the paper.

The paper introduces additional modification on top of PPO, motivated by the previously identified issues, that leads to significant performance Improvement.

Weakness

I would have liked to see a discussion on how the reader should interpret equation 4 and 5 in the paper. Also, How does the y-axis of Figure 2 relate to the left hand side of equations 4 and 5? This is not clearly explained in the paper if I am not mistaken. It would have been helpful if the authors include pseudocode on how the y-axis of Figure 2 is calculated.

Also, the noise scale as explained in equation 3 does not seem to take into account the directions of the true gradient. Is that true? If that is true then the quantity used to compute the noise scale seems a bit odd to me because shouldn't we take into account the direction of the true gradient and the direction of the noise vector as well? 

The paper also introduces a new benchmark. Even if an anonymous copy is included, I would have liked to see a self-contained description in the main paper on why results on the new benchmark are meaningful or more meaningful compared to the previous version of the arcade learning environment benchmark.

Can the authors also please elaborate on the discussion at the end of section 4.2 – why is it that because of bootstrapping, estimate used for V Target benefit from being low bias? 

I found the exposition in the experimental results section a bit unclear. If the paper is about noise level then why is it that the first experimental result presented has to do with the number of training epochs? 

questions: Can the authors also reconcile the results presented in the paper with this paper https://arxiv.org/abs/2101.05982? This paper seems to suggest that we should use a high update to data ratio, whereas the current paper seems to suggest that we should perform fewer updates. 

limitations: Yes
review_title: null

Meta-review: metareview: The reviewers found this to be a well-executed technical contribution, and all reviewers agree it meets the bar for acceptance.  While this paper does not seem to provide a breakthrough novel insight, it does contribute useful information for the field, and I believe sharing with the community is beneficial.  I recommend accepting this paper.
 

confidence: Certain
Decision: Accept