Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for reducing gradient interference between different components (Q, V, and policy) in deep RL with discrete actions. By carefully placing stop-gradient (sg) operators in the objective, the authors show (both theoretically and empirically) that the gradients between the Q function and the policy can be made parallel, so that there is no interference. The method, CASA, can be applied to value-based or policy-based algorithms using the same computations, and attains strong performance on Atari.
strength_and_weaknesses: Strengths:
* The paper addresses an issue that many ignore in deep RL – the relationship between the policy evaluation and policy improvement steps in policy iteration.
* The clever usage of sg in the objective leads to the desired result without any approximation or extra computationally intensive steps.
* CASA substantially improves the performance of the base algorithm (IMPALA) on the Atari benchmark.

Weaknesses:
* The method as presented only applies to discrete-action problems, owing to the softmax parameterization of the policy and the summation over actions when computing the advantage.
* The paper shows plots for many different gradient angles, but it’s not clear which we should care the most about. It would be helpful to have a better understanding of how the angles between the gradients correlate with performance.
clarity,_quality,_novelty_and_reproducibility: The writing of the paper is mostly clear, but could be improved in places.

In the Preliminary section, the objective $J$ is written incorrectly. If you’re taking the expectation over the discounted stationary distribution, the random variable inside the expectation should just be $r(s,a)$, not the discounted sum of rewards.

The CASA approach is novel, to my knowledge.

The appendix includes the relevant Python package versions and an exhaustive list of hyperparameter settings, so I believe it would be reproducible.
summary_of_the_review: I think the paper is a useful contribution, as it tackles a relatively underexplored problem with a clever and effective solution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the problem of solving the alleged inconsistencies between the policy evaluation and policy improvement steps of the “policy iteration” algorithm. The paper proposes a method called CASA as a solution which suggests some modifications on the gradient calculations of standard algorithms such as PPO.
strength_and_weaknesses: *Strengths:*

The formulation of the beta variable is interesting.

*Weaknesses:*
  
 * I cannot make sense of the studied problem at all. What does it mean to have a gradient conflict between two operational steps (policy evaluation and policy improvement) which have complementary contributions to the solution?
 * The paper states its aim as to “eliminate the inconsistency between the policy evaluation and policy improvement steps”. However, I do not understand how the proposed CASA method solves this problem. It only sets up a tunable continuum between these two steps. But one would also get the same using the \lambda-policy iteration formula, which is introductory textbook stuff.
 * I do not find the reported ablation study informative. The benefit of CASA can be more directly quantified if the paper compares PPO vs PPO+CASA and R2D2 vs R2D2+CASA directly in the Atari game use case. 
 * Overall the experiments section is rather slim. The paper does not compare against some strong candidates such as SAC and MBPO.

clarity,_quality,_novelty_and_reproducibility: The paper has a rather scattered presentation style. I found it hard to follow the storyline, disambiguate the studied problem, and a main hypothesis drawn as a solution to it. This also makes it hard to view the experiment results as empirical evidence to a scientific claim.
summary_of_the_review: The studied problem of the paper is not stated in a technically rigorous and unambiguous and intuitively clearly motivated way. The solution attempt also does not appear to follow a consistent deductive process. The reported experiment results have weak relevance to the main hypothesis.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This works studies the relation between policy evaluation and policy improvement in the context where the value-function(s) and the policy share the same parameterization, and suggests a way to consolidates the two type of updates. 
strength_and_weaknesses: The basic ideas underlying the paper are interesting and could potentially be a valid contribution. However there are several key issues that make the claims either difficult to understand (in the good case) or wrong/inaccurate (in the worse case). The evaluation is also not satisfying, relying on complex architectures and environments (see more below) to try and support what are fundamentally abstract theoretical claims.

I would recommend the authors to reconsider the evaluation experiments that are used to support the theoretical claims. Ultimately, even a much simpler tabular settings where everything is much more controlled can be a much better starting point for this kind of paper, even if not all claims or properties can be rigorously proven theoretically.
clarity,_quality,_novelty_and_reproducibility: The paper is hard to follow. There are technical details which are not explained well, or are misleading, throughout. Some can be easily fixed, but some are more fundamental.

**Major issues**
- In general, the paper claim to "CASA, Critic AS  an Actor, which theoretically guarantees that the evaluation gradient of the state-action values and the policy  gradient to be parallel". But the way this is achieved is a triviality, because the architecture forces the log-policy to be linear in the state-action values. In that sense, this is really a value-based method as there is no "independent" learning of a parameterized policy. Everything goes through the value (more precisely, advantage) function. And the main idea is to use a Softmax instead of a Max (greedy) for the action-selection rule.
  
- The definition of $\beta$  and its explanation (Section 3.1) are misleading (or at least unclear). The definitions of $\mathbf{I}$ and $\mathbf{E}$ included (and rightly so) an expectation over state-action pairs as induced by $\pi$. This expectation for some reason is missing from the definition of $\beta$. Note that with this expectation, it is **not** the case that "$Q^\pi-Q_\theta$  and $Q^\pi-V_\theta$  are two scalars" that can be pulled out of the expectation because in general they will be correlated with the gradients. In fact the entire PG method relies on this correlation, otherwise the expectation value of the score function by itself $\mathbb{E}\left[\nabla\log\pi\right]$ is simply $0$. Since this is not explained it's also unclear how $\beta$ is calculated empirically later, from full trajectories.
- The explanation that follows is also confusing, or at least I couldn't understand what is being claimed (The math says that $\mathbf{I}$ and $\mathbf{E}$ are parallel, but the text says they are perpendicular?)
-  Related to that, the question on "under what conditions $\nabla_\theta Q_\theta \propto \nabla_\theta\log\pi$  is misleading because $\nabla_\theta Q_\theta$ does not reflect policy evaluation at all, if anything it is more related to policy improvement: since $\theta$ (implicitly) parameterizes the policy, $\nabla_\theta Q_\theta$  (implicitly) means a change in the policy that will make the action values higher, i.e., improve the policy.

- The fact that all experiments are done with a recurrent network (a detail that is somewhat "hidden" in Section 4.1) is a very serious shortcoming for the evaluation. The entire theoretical attempt is built on top of memoryless/reactive value-function and policy estimations (as is standard for MDPs), but then the experiments is done with an RNN. This might be a source of correlations in the gradients, estimations, etc. throughout the trajectories which is completely out of scope of the more "theoretical" explanation.

**Minor issues**
*(or, really, not minor but more easily fixable)*

*Preliminary section:*
- The presented objective wrongly mixes the "trajectory" and "state-action occupancy measure" formulations (e.g. Sutton et al. 1999, Ghosh et al. 2020) for the RL objective. If $s\sim d^\pi$  then this already encapsulates the $\sum_t \gamma^t \mathbb{P}^\left(t\right)\left[s\right]$ averaging. Since the discounted occupancy measure is not really used later in the paper it seems preferable to simply write the objective in the "trajectory formulation" i.e., $J=\sum_t\left[\gamma^tr\left(s_t,a_T\right)\right]$ and explain how the sequence $s_t,a_t$ is being sampled by the policy-environment interaction. Moreover:
	- it is currently not defined that $a_t$ is sampled via $\pi\left(s_t,\cdot\right)$ .
	- The measure $d^\pi$ as defined is **NOT** the "stationary distribution induced by $\pi$" but rather the normalized discounted occupancy measure / visitation distribution. It becomes the stationary distribution only in the limit $\gamma=1$ (yielding the time-average criterion instead of the discounted criterion of performance).

- In the next paragraph, the definition of $V\left(s\right)$ and $Q\left(s,a\right)$ should be conditioned on $s_0=s$ and $s_0=s,a_0=a$ respectively, rather than on $s$ and $s,a$ which can be confusing to interpret.

- The same issue of mixing the "trajectory" and "state-action occupancy measure" repeats in the paragraph on Policy-Gradient methods (namely $\Psi$ has an explicit time-index inside it which is not really defined. Contrast this to how it is defined in Schulam et al. 2015, Eq.1).
- It is not clear why the concept of $\gamma$-justness, which is never used again in the paper, is introduced here, instead of a more simple explanation. This also confuses between the exact PG and an estimate of it based on sampled trajectories (the concept of  ""$\gamma$-justness" being relevant to the latter rather than the former case, while here the paper describes the exact gradient).
  
- The Equations defining the objective/optimization problems, gradients, and (in the next Section) $\beta$ should all be numbered. 

*Methodology section*:
- Why is the PG theorem cited *again*, only a few sentences after it was already presented, but using a different form than the one presented in the Preliminary section? This will only confuse readers.
  
- What does $\mathbb{E}_p$  represents (in another un-numbered equation, discussing the relation to dueling DQN)? what is $p$? 
- What is $\mu$ in Table 1?

- The text in Figure 3 (axes labels, labels) is **tiny** and becomes readable only in 350% zoom. 
  
- Figures 4 and 3: Several panels report $\cos$ values side by side but have (very) different y-axis scale. This is very misleading. All Figures depicting $\cos$ values should have a normalized scale between 0 to 1 (or -1 to 1 if relevant).
 
*Related work*:
- In general, the concept of $\beta$ might be related to the concept of a "compatible value function" (Sutton et al. 1999, see also Kakade 2001). It would be interesting if the proposed concept here can (maybe after some corrections/modifications) serves as a generalization of the original compatibility notion (which was more of a binary concept, yes or no). I would encourage the authors to relate and explain how and if their ideas relate to that, or at the very least mention it in the related work.
summary_of_the_review: The paper deals with an interesting question and some parts or aspects of it could be a valid contribution. However in its current form it has too many open gaps and issues that should be fixed before it is ready for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel approach to generalized policy iteration (GPI) exploiting the inherent connection between policy evaluation and policy improvement. It firstly shows that the policy evaluation and policy improvement steps are connected in the case of parameter sharing between the value function and the policy, it describes the benefit of this connection and proposes the method CASA to exploit this connection. CASA is tested on the Atari 200M benchmark where it achieves remarkable state-of-the-art results.
strength_and_weaknesses: Strenghts
-----------
- The paper studies the interesting problem of generalized policy iteration from the original point of view of bridging the gap between policy evaluation and policy improvement;
- Empirical results on Atari are impressive.

Weaknesses
--------------
- The quality of the writing is not satisfactory and makes the paper hardly readable and understandable;
- The proposed method CASA seems to rely on the fact that ALL parameters should be shared between value functions and policy. However, in the paper it is stated that only "most parameters" are shared. Doesn't this critically go against the assumptions of CASA? Please clarify;
- The figures are too small and hardly readable.
clarity,_quality,_novelty_and_reproducibility: The writing of the paper is quite suboptimal. Many sentences are not well formulated and make the paper not easily understandable. For example:
- "Since $Q^\pi − Q_\theta$ and $Q^π − V_θ$ are two scalars, when $\nabla_\theta Q_\theta \varpropto \nabla_\theta \log \pi_\theta$, the two sides of $\beta$ meet": instead of saying that the sides meet, considering rephrasing saying that the gradients of policy improvement and policy evaluation overlap;
- "As I and E are perpendicular to each side of β, they become parallel arrows on opposite directions.": how can I and E become arrows? This should be rephrased;
- "By equation 6, the directions of the gradients g are in common. We call it a path. The equation 6 shows that $\nabla_\theta LQ$ and $\nabla_\theta J$ walk along the same path g.": these sentences are redundant and should be merged in one.

The novelty of the proposed method is satisfactory.

The list of used hyperparameters is thoroughly reported for reproducibility, but the source code is missing.
summary_of_the_review: This paper studies the important problem of generalized policy iteration in reinforcement learning, and the proposed method is intriguing and able to achieve remarkable results in the challenging Atari 200M benchmark. However, I'm not convinced about the theoretical correctness of the method. It seems to me that conflict between policy evaluation and policy iteration is avoided only in the case where the parameters of value functions and policy are exactly the same. This is never the case in practice. As also stated in the paper, the parameters are mostly shared, except for individual heads. It is not clear to me whether the fact that MOST parameters are shared, is considered sufficiently good to meet the assumption of CASA; in that case, it would be nice to have a better discussion about it. Unfortunately, the low quality of the writing makes everything slightly harder to grasp.

It would be really nice if the authors could solve these doubts in the rebuttal, and I'd gladly increase my score in case the answer will be convincing enough.

Post-rebuttal feedback
------------------------------
I have read the revised paper and appreciate some improvements on the previous version. I am still doubtful about the approach, sharing similar concerns to those expressed by Reviewer aLPu, in particular
> In that sense, this is really a value-based method as there is no "independent" learning of a parameterized policy.

Thus, I raise my score for the improved version but I still consider this work under the threshold of acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the Generalized Policy Iteration approach when the same parameterization is shared between the policy and the value function. The authors propose a method to reduce the interference between the updates of the different components and test it on Atari games.
After reading each others' reviews and the authors' feedback, the reviewers acknowledge that the paper has merits (relevant topic, interesting approach, good empirical results), but they raise some concerns (the description of key concepts is not clear, the presentation is poor, the significance of the contribution is quite limited) that need to be solved before the paper can be accepted for publication.
We encourage the authors to consider the reviewers' suggestions while preparing a new version of their paper.

justification_for_why_not_higher_score: The paper needs to be improved from several points of view.
The presentation is poor and the significance of the contribution is quite limited given the assumptions considered.
Decision: Reject