Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper claims that training policy and value networks separately for actor critic algorithms like PPO or PPG can lead to overfitting of the value network in procedural generalization tasks like Procgen. To get around the overfitting issue, it proposes to update the value network less often and with more data. It also proposes to add a self-supervised objective to further improve generalization performance, and observes modest gains on some games in Procgen. 
strengths_and_weaknesses: Strengths:
* The paper makes an interesting empirical observation of overfitting in separate update PPG agents that could harm their generalization performance. 
* The paper is very clearly written except for section 4.3 and gets the idea across clearly. Figure 1 is a great summary of the proposed modification with clear differences to PPG and PPO. 
* The fix for the overfitting issue is straightforward and easy to implement. 
* The experiments are performed on a well benchmarked environment suite (Procgen) and include informative ablations. There are also a bunch of valuable auxiliary experiments investigation value function learning.

Weaknesses: 
* The contribution of delayed value updates itself seems not quite significant, very specific to PPG and its unclear how well it would generalize to other algorithms. 
* The overfitting issue might just vanish as we scale the model and data size for these agents. So I fear the delayed value update contribution might end up being a "regularization game" for this specific PPG setup and end up getting "bitter lesson'ed" in the long run.
* Note: The above two are subjective opinions on significance of the contributions. 
* The improvement for DCPG over PPG seems marginal, especially when we look at individual performance curves in Figure 6 of the appendix. It seems like there are only 3-4 out of 16 games where the regularization has a clear and significant benefit.
* The contribution a self-supervised objective with learned dynamics has been explored multiple times in the literature (SPR (https://arxiv.org/abs/2007.05929), PBL (https://arxiv.org/abs/2004.14646), DeepMDP (https://arxiv.org/abs/1906.02736) etc. Moreover, it has been explored explicitly in Procgen as well in MuZero++ (https://arxiv.org/abs/2111.01587) and DRIML (https://arxiv.org/abs/2006.07217). The idea of using a discriminator for this task was proposed in CPC|Action, and evaluated in Procgen in DRIML and ablated in MuZero++. Using an inverse modelling objective along with the forward objective was also proposed in SGI (https://arxiv.org/abs/2106.04799). The paper unfortunately doesn't mention or appropriately position itself against these works.
* The reporting of results does not follow the recently proposed recommended practices as described in rliable (https://github.com/google-research/rliable), and thus might be prone to making incorrect conclusions. More details below. 
questions: * The state-of-the-art claim might not stack well against MuZero++ (https://arxiv.org/abs/2111.01587) given the gap b/w MuZero++ and PPG/PLR/etc. However, it looks like the authors were not simply aware of this work, and DDCPG being model-free is not directly comparable to MuZero++ anyways, so I don't hold any negative points for this one if its mentioned in the revisions. 
* It would be nice to see if and how does the performance changes as the model size and data size is increased. Perhaps you can try increasing the IMPALA network size systematically or using a ResNet, and similarly train on more amounts of data. 
* Generalized in RL is an overloaded term, and this paper studies a very specific form of generalization (i.e. observational or procedural generalization). There are other forms of generalization such as changing the task or reward functions. Make sure that this distinction is clear. 
* The self-supervised objective should be appropriately positioned with respect to prior work listed above. 
* It would be a good idea to use the rliable colab (https://arxiv.org/abs/2108.13264) to plot the results since it has been found that standard reporting of results can be misleading. Specifically, please include the IQM and probability of improvement plots. Also, following the recommendation its better to report Min-max rather than PPO normalized scores.
* The bolding of scores in Table 1 relies on mean scores it looks like, which can be very misleading. If two methods have overlapping mean and std devs, they should both be bolded or neither since variance can be high for these scores. 
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new approach for RL in procedurally generated environments with empirical evidence. At its core is the problem of the critic learning problem and potential overfitting. The proposed approach mainly builds on the Phasic Policy Gradient method by unifying it. The method is compared against natural baselines and other environment-relevant methods. The experiment section also shows several ablation studies.
strengths_and_weaknesses: The contribution of this paper is original and appears relevant to the community. It nicely builds on a battery of previous work, which has some overfitting limitations. The empirical analysis is quite furnished and gives very some interesting insights. However, the improvements provided by DCPG or DDCPG over the baselines appear to be less important than what is claimed by the authors. In particular, the statistical significance in Tables 1, 2, and 5 depicted by bold text and the empirical analysis in Section 5.2 cannot be conclusive. Statistically speaking when compared to the baselines, DCPG appears to improve on 2/18 tasks and DDCPG on 4/18. Indeed, unfortunately, the stds appear in many instances to be generally too significant to draw any statistical conclusion. We develop more points below. Regarding the quality, overall, the paper is also mostly well written and develops its ideas clearly.
questions: The above comment does not detract from the pertinence of the method, but I have a few comprehension questions or some comments about the paper, please see below:

1. Some relevant works tackling the accuracy of the critic and its impact on the agent's learning seem to be missing: [1] with an empirical analysis of the phenomena and [2] with the introduction of a critic trained only using the residual variance to speed up learning.
2. Could the authors elaborate on the motivation for adding the Forward/Inverse Dynamics Learning task? The paper's focus is mainly on value function overfitting and its regularization - if any, could the author help understand to what extent does using this auxiliary task help avoid overfitting?
3. Have the authors considered applying this Forward/Inverse Dynamics Learning task to PPG? It would be interesting to know if using this component also helps PPG and to what extent (maybe even more than DCPG).
4. Table 2 is quite interesting. Could the authors provide an intuition about why learning the inverse dynamics separately seems to be worse than learning it jointly with the forward dynamics with a single discriminator? Is the total number of parameters for the “dynamics learning module” kept the same? i.e. how many parameters does DCPG+FI have vs. DDCPG?


[1] Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2020). A closer look at deep policy gradients. In International Conference on Learning Representations.  
[2] Flet-Berliac, Y., Ouhamma, R., Maillard, O.-M., & Preux, P. (2021). Learning value functions in deep policy gradients using residual variance. In International Conference on Learning Representations.
limitations: The potential negative societal impact or the limitations of the work have not been addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, it is first shown that the value function is easier to overfit compared to the policy function.
Based on this observation, a new policy gradient algorithm called Delayed-Critic Policy Gradient (DCPG) is proposed, which reduces the overfitting of the value function by less frequent updates with more training data.
Moreover, a single discriminator is applied to learn both forward and inverse dynamics of environments, which helps to generate better representations and improve the performance of DCPG across different tasks.
strengths_and_weaknesses: This work presents two novel policy gradient algorithms --- DCPG and DDCPG.
Compared to PPG, DCPG has a more compact model structure with only one encoder, while achieving comparable or better performance on different tasks.
Moreover, by combining the discriminator that learns environmental dynamics, DDCPG boosts the performance of DCPG furthermore.
This paper is clearly written with a good flow.
The analysis experiments and the ablation studies are also helpful in understanding the effectiveness of DCPG and DDCPG.

There are several minor issues.
In Section 2.2, the clip operation is not explained though it might be obvious to readers familiar with PPO.
In Section 4.3, both the encoder $e_\theta(\cdot)$ and the dynamics head $f_\theta(\cdot, \cdot, \cdot)$ are parameterized by $\theta$. Is the encoder the same one as shown in Figure 1(c)? If so, how could the same $\theta$ be used to parameterize a different function, i.e. $f_\theta(\cdot, \cdot, \cdot)$? It is better to present the model structure of DDCPG as well to clarify.
questions: - In Figure 4, how are the true values computed? How to make sure that these computed values are true values during training?
- Do DCPG and DDCPG use a larger replay buffer than the baseline algorithms, such as PPO and PPG?
limitations: In Table 2, the error bars of some results overlap with each, making the claim in Section 5.4 less convincing.
review_title: null

Meta-review: metareview: All reviewers were in favor of acceptance, and after reading the paper myself I am in agreement. The empirical results were good and the experimental work quite comprehensive. The method is well explained and the writing is clear and easy to read. The only real detraction I saw, distinct from things already mentioned by reviewers, was that there are some statements that feel overly strong given the presented results (e.g. L161, L179). I was curious about sensitivity to hyper-parameters, specifically the settings around how long each phase lasts, etc, along the lines of what was done in the PPG paper. That said, I would generally down-play the importance of this as the proposed method is using the same hyper-parameters as for PPG and appears to have undergone minimal hyper-parameter tuning.

In all I think this makes a clear contribution to the field and should be accepted.



confidence: Certain
Decision: Accept