Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper outlines a connection between the causality literature and the advantage function in reinforcement learning. The advantage function can be viewed as an estimate of the causal effect of an action on the expected return. As a second contribution, a theorem shows that the advantage function can be viewed as the solution of a variance-minimization problem subject to a certain constraint. This leads to a different objective for learning the advantage function without resorting to estimating the Q and V value functions. This approach is shown to outperform generalized advantage estimation (GAE) in a suite of standard benchmark tasks. 
strengths_and_weaknesses: The paper has two distinct contributions: the connection of advantage functions to causal effects and the new objective for estimating the advantage function. I find both of these to be interesting and of interest to the community.
Moreover, the paper is well-written and easy to follow, from the organization to the choice of notation.

The exposition of the benefits of the advantage function over Q-values was well-done. The intuitions was nicely explained and supported by the synthetic experiment and Fig. 2.
After checking some proofs, they seem to be correct and are adequately explained. I had no trouble following them. To me, it's a nice result that you can view the advantage function as a variance-minimizing state-action-dependent baseline (under some constraint). 
The experiments seem sound with reasonable choices as baselines and good experimental procedures. Sufficient details are presented in the appendix to understand the experiements in-depth. In particular, I thought the shuffling of actions in the synthetic experiment (as described in the appendix) was nice experimental design. The experiment varying the size of the network  also yielded some interesting results. 

Although I am not very familiar with the causality literature, the link between causal effects and advantage functions is intriguing to me. While this connection wasn't developed too thoroughly in this work, it does outline a direction that could be explored further in the future. 
Overall, I think this is a solid paper and haven't found any major issues, although I do have some questions which I've written in the next section.
questions: - Table 1 presents counts for how often an algorithm was better (DAE or GAE). Does this account for the confidence intervals? As a rule-of-thumb, if the confidence intervals overlap, it wouldn't be fair to say one was better than the other. This table could include a number for the the number of environments with approximately equal performance.

- The paper makes a connection between the advantage function and causal effects but, afterwards, doesn't make use of this link. Do you have any other thoughts on how this interpretation could lead to better RL algorithms?

- Concerning Fig. 2, do you have an explanation why the advantage function tends be more stable? Do you think it's related to Prop. 1 and actions have limited effects across time in these RL environments?
Also, have you looked at the plot of Fig. 2 for other environments? Do you still find the same thing?

- Lines 162-166: Is it true that in a multi-step setting, the advantage function would be more stable? It seems like delayed rewards are an important factor in explaining the difference between Q-values and advantage functions.

- Any thoughts on viewing the advantage-function as a variance-minimizing state-action-dependent baseline (with a constraint)? How would you compare that to the solution of the unconstrained version of the same optimization problem?


limitations: Yes, I don't see any major issues here.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes Direct Advantage Estimation, an algorithm which estimates
directly the advantage function instead of the traditional state(-action) value
estimation. They draw some links with a causal interpretation and perform
experiments on a synthetic environment as well as MiniAtari and ALE, comparing
themselves to PPO with GAE.

strengths_and_weaknesses: # Strengths
The method proposed is motivated theoretically, is simple to implement and seems
to show improved performance compared to the baseline in the experiments. The
paper is overall easy to read.

# Weaknesses
- The paper displays a collection of different results and interpretations but
  few seems actually necessary to explain the method. More precisely
    1. I find the connection to causality weak overall. In this light any
       centered variable can be thought of as an instance of a causal power, this does not necessarily mean there is any causal estimation behind.
    2. The content of section 3 is mostly motivated by the synthetic example.
       I find this problematic as I believe the synthetic example is a very
       contrived one. While one motivation for DAE is the credit assignment
       problem (l22-23) this environment is specifically designed so there is no
       credit assignment. 

- Baselines: while in practice DAE ends up being very close to a n-step return
  method, the only comparison point in the paper is with GAE. A minimum I would
  need to recommend this paper for acceptance would be a fair comparison with
  its 1. n-step return ablation (by removing the effect of A in eq 17 if I
  understood correctly).
  Then I think, to show the interest of directly estimating the advantage,
  another baseline could be to replace your directly learned A by
  2. Q - V where Q and V are learned like usual
  3. A = Q - V learned with a duelling architecture.
  1) would show if your estimator which incoporates the advatange is superior to
     the tradional TD one
    2) and 3) would be useful to show whether learning the advantage directly
       is useful. Indeed while you make a case for the advantage being a quantity of interest, I find there is little evidence in the paper for why learning it directly with your method is important compared to the methods 2) and 3).

Minor points:
- eq 4: maybe rather than the approximate symbol it would be more precise and
  exact to say that it is a sample?
- As you do not seem to be Vtrace or the Impala algorithm, I think using "IMPALA" just to mean the architecture can be misleading.
questions: - As you are sampling on a trajectory by trajectory basis instead of minibatch of
  frames (l 497): how does your algorithm perform in terms of wallclock time
  compared to PPO?
- Fig 4: Does DAE still have lower variance in a more general setting rather
  than one designed so that DAE can reach 0 variance easily? I am
  thinking a simple gridworld, the Q and V function can be computed in closed form.
- Do you use p for the discounted or undiscounted stationary distribution? It seems in eq 39-41 it would be the discounted if you are using an unbiased estimator of policy gradient, however it seems to be used as the undiscounted in the main text.
- To make sure I understand clearly, what exactly is \hat{A} in alg 1? Is it just the learned advantage or the n-step TD with advantage function?
limitations: Some limitations are addressed with Fig 3. However I think the authors need to make a stronger case for their method:
- First, a comparison with a more similar algorithm (n-step returns) would be better than GAE, as it would showcase that incorporating the advantage in the TD error is important
- Then they would need to show why estimating directly the advantage as they do is useful/important/interesting. This could be tested experimentally by comparing with other methods for estimating the advantage.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the problem of learning the advantage function directly (with no need for learning Q first). The authors coined the term $\pi$-normalized [a bit weird terminology] as any function whose expectation over $\pi$ is zero. Advantage function is an obvious example. Clearly, if the reward is shifted by any $\pi$-normalized function, the expected return remains unchanged. This property is then used to minimize the variance of return by making it disentangled from the policy. One main result (via Theorem 1) is that the optimal $\pi$-normalized function (in terms of minimizing the variance) to be subtracted from the reward is indeed the advantage function of the current policy for on-policy learning. This also gives rise to a learning paradigm with the loss function coming directly from the grounded minimization result in Theorem 1. This is then extended to N-step learning using the value function. Finally, promising experimental results are provided to confirm the method. 
strengths_and_weaknesses: Strengths:

- I liked the theoretical results. Direct learning of advantage function is novel and useful. Additionally, the minimization formulation from Theorem 1 is also interesting. 

- Disentanglement of learning from current policy is indeed interesting, as it is also evident from the experimental results. 

Weaknesses:

- The implementation of this paper requires several steps, which are somehow distributed in various parts of the paper. A clear algorithm block is missing. 

- The second part of the theory is not presented clearly, and is too condensed. 

- Minor: I do not understand the emphasis on some "causal" concepts. These are neither needed (or even used) nor are they any addition to the presented theory. 
questions: - L54: Your definition of reward is indeed *expected* reward (expectation taken oven next state $s’$).

- Paragraph of L150: I generally accept that A is lower variance than Q but using PPO to demonstrate this is not adequate. In particular, I would examine this with a value-based method that directly reinforces Q, instead of AC methods, which normally does not give rise to accurate value function. 

- The term $\pi$-normalized is somehow misleading, as normalization is a process of mapping to one, not zero. Consider a different term; perhaps something like  $\pi$-neutralized or $\pi$-nullified. 

- From proof of theorem 1 (L473), it looks like that non-zero policy is a *missing* assumption, $\pi(s,a)\neq 0$ for all $(s,a)$. This kind of assumption is very normal in policy gradient results. I do not consider it as a downside. 

- Using value-function for the N-step case is somehow defeating the purpose as the main premise is to disentangle advantage learning from value learning. Isn't it? 

- The last part of the theory (from L232 onwards) is not well-written and requires more explanation and expansion. I recommend removing several parts from the first 4 pages (which are simply reviewing textbook stuff) and expand on the last part of the theory with more scrutiny. In particular, $V_{target}$ is not clearly defined and its dependence on learning $V$ is not clear. The trajectory augmentation seems also to be required. From the text, it is very hard to come up with a clear algorithmic machinery. I strongly suggest to add a algorithm block and summarize the details. 

limitations: No limitation is explicitly mentioned, though from the future work discussion there are some comprehendible limitations: 

- Action space is discrete. 
- On-policy learning.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new method for estimating the advantage function of a policy that does not require first estimating the Q or value function. They demonstrate that on a variety of tasks, it works better than the "industry-standard" GAE.
strengths_and_weaknesses: (+) This paper was a joy to read. I appreciated the nuance of the discussion (e.g. the examples of where the advantage fn. was more or less policy dependent, the example of why TD-based methods can't work for advantage functions in general). The paper felt fundamental yet accessible, which is a hard balance to strike.

(+) The experiments were well-done and involved results averaged across many environments, indicating a significant gain in performance when switching from GAE to DAE.

(+) I appreciated the connection to causal effects and especially the Halpern citation :).

(-) In general, making decisions based on expected returns (rather than greedily) is a good idea. Some parts of the first few pages could be read as indicating otherwise. If I were you, I would focus on why advantages are good (cite PPO/Kakade&Langford) and delve into why (they are sort of an appropriate level of "local"), rather than discussing how Q functions depend on the policy at future steps (which is somewhat obvious and useful).
questions: (1) Much of the motivation in the intro of the paper centers around being able to directly estimate the advantage function without having to first fit a value function. However, it seems like from Thm. 2 and beyond, the authors actually end up fitting a value function. Why does your proposed method not suffer from the same issues that plague value-based methods that you point out prior (e.g. small policy changes leading to large distribution shifts and inaccurate value function estimates)?

(2) You mention in the appendix that for DAE, one needs to sample entire trajectories rather than mini-batches. Do you have a wall-clock time comparison between DAE and GAE?

(3) The concern about estimating the sum of advantages for normalization reminds me a lot of the difficulties estimating the partition function for maximum entropy / energy-based / exponential family models. I wonder if you could borrow from the rich literature there to solve your problem.

(4) The DAE-Wide results are particularly good. Could you provide some reasons as to why this might be the case?
limitations: The authors correctly acknowledge that their method is dependent on discrete action spaces for enforcing the normalization constraint and provide some potential solutions. I think the related work section could be improved as well (i.e. there are other advantage estimation methods outside of GAE).
review_title: null

Meta-review: metareview: Overall, the proposed approach is well-motivated, simple to implement and appears to work well empirically. The connection between the advantage function and causal concepts is appreciated by some of the reviewers, but a point of confusion and viewed as insufficiently motivated by others. I recommend acceptance for this paper, but for the causal connection, I want to ask the authors to either provide more in-dept explanation or to de-emphasize this connection.
confidence: Certain
Decision: Accept