Direct Advantage Estimation
Authors: Hsiao-Ru Pan
Nico Gürtler
Alexander Neitz
Bernhard Schölkopf
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
advantage function
actor-critic
causal effect

Abstract: The predominant approach in reinforcement learning is to assign credit to actions based on the expected return. However, we show that the return may depend on the policy in a way which could lead to excessive variance in value estimation and slow down learning. Instead, we show that the advantage function can be interpreted as causal effects and shares similar properties with causal representations. Based on this insight, we propose Direct Advantage Estimation (DAE), a novel method that can model the advantage function and estimate it directly from on-policy data while simultaneously minimizing the variance of the return without requiring the (action-)value function. We also relate our method to Temporal Difference methods by showing how value functions can be seamlessly integrated into DAE. The proposed method is easy to implement and can be readily adapted by modern actor-critic methods. We evaluate DAE empirically on three discrete control domains and show that it can outperform generalized advantage estimation (GAE), a strong baseline for advantage estimation, on a majority of the environments when applied to policy optimization.

Direct Advantage Estimation
Hsiao-Ru Pan Nico Gürtler' Alexander Neitz Bernhard Schölkopf'
1 Maa Planck Institute for Intelligent Systems, Tübingen DDeepMind
Abstract
The predominant approach in reinforcement learning is assign credit actions based on the expected return. However, we show that the return may depend on the policy in way which could lead to excessive variance in value estimation and slow down learning. Instead, we show that the advantage function can be interpreted as causal effects and shares similar properties with causal representations. Based on this insight, we propose Direct Advantage Estimation (DAE), a novel method that can model the advantage function and estimate it directly from on-policy data while simultancously minimizing the variance of the return without requiring the (action- value function. We also relate our method to Temporal Difference methods by showing how value functions can be seamlessly integrated into DAE. The proposed method is easy to implement and can be readily adapted by modern actor-critic methods. We evaluate DAE empirically on three discrete control domains and show that can outperform generalized advantage estimation (GAE), a strong baseline for advantage estimation, on majority of the environments when applied to policy optimization.
1 Introduction
Reinforcement learning (RL) methods aim to maximize cumulative rewards sequential decision making problems (Sutton and Barto 2018). Through interactions with the environment, agents learn to identify which actions lead to the highest return. One major difficulty of this problem is that, typically, vast amount of decision makings are involved in determining the rewards, making complex to identify which decisions are crucial to the outcomes. This also known as the credit assignment problem (Minsky 1961]. A straightforward approach is assign credit according to the expected return, which is aligned with the RL objective([Sutte) and Barto 2018). However, the expected return accumulates all the future rewards and does not directly reflect the effect of an action. Consider, for example, the environment illustrated in Figure [ Since the transitions at each state independent of the actions chosen, the effect of any action should be immediate. The expected return conditioned on any action, however, depends on the policy of all subsequent states. This property could be undesirable because the expected return would vary significantly whenever the policy changes during training, making it a difficult target to learn.
A natural question to ask is, then, how should we define the effect of action? In this work, take inspiration from the causality literature, where the notion of causal effects is defined. We show that the advantage function can be interpreted the causal effect of an action the expected return. Additionally, we show that it shares similar properties with causal representations, which are expected improve generalization and transfer learning (Schölkopf et al. 20021).
We prove that the advantage function minimizes the variance of the return under constraint, and propose Direct Advantage Estimation (DAB), method that model and estimate the advantage
*Correspondence to: hpan @ muchingen.mpg.de
Work done while at MPI-IS.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

r=1 1
r=1
u
u
u
81
82
d
T= 0
T= 0
r= 0
Figure I:
An example environment where the effect of an action on the return is only immediate. The agent starts in state and, at each step chooses either the action (up) (down) with immediate rewards of or O, respectively, before transitioning into the next state 8t+1-
function directly from sampled trajectories unlike previous methods that rely on learning either the value function or the Q-function beforehand. Furthermore, we show that DAE can be combined with value function approximator and be updated in a way similar to Temporal Difference methods [Sutton 1988]. We then illustrate how DAE can be integrated into modern deep actor-critic methods using the example of Proximal Policy Optimization (PPO) [Schulman et al. 2017]. Finally, we test our method empirically on three discrete domains including (1) a synthetic environment, (2) the MinAtar suite (Young and Tian 2019] and (3) the Arcade Learning Environment [Bellemare et al.
2013). and demonstrate that DAE outperforms Generalized Advantage Estimation (GAE) [Schulman
al. 2015b] most of them.
The main contributions of this paper are
We analyze the properties of the advantage function and show how they are linked to causality.
We develop DAE, a novel on-policy method for estimating the advantage function directly, and compare it empirically with GAE using PPO.
2 Background
We consider discounted Markov Decision Process (S,A, P,r,7) with finite state space S, finite action space A. transition probability P(s'|s, a), expected reward function r : x - R, and discount factor 7 E [0, 1). A policy is function
S x A - R with T[als) denoting the probability of action given state s. A trajectory T = (so; do, 81, (I....) is said to be sampled from policy if a, x--1st) and 8++1 P(-[st, dz). We denote the return of a trajectory by G(T) Ei20 7'r,, where r2 r(t,, dz). The goal of an RL agent is to find a policy n² that maximizes the expected return, i.e., x² = arg max, E, [G(r)]. Given a policy n. we define the value function by V*(()= Er [s(r)[sp=s], the action-value function by (s. Erer [G(r)]sy sy= ag Kg=a and the advantage function by A² (s, a) = (s, a) V* (s) [Sutton and Barto 2018 Baird 1993] For simplicity, we shall omit the superscript. These functions quantify how good a certain state (state-action pair) is, which can be used to improve the policy. In practice, however, they are usually unknown to the agent, and have to be learned from its interactions with the environment. Below, we briefly summarize how they are typically learned.
Monte Carlo (MC) method. By definition, an unbiased way to estimate V(s) is by sampling trajectories starting from state and averaging the sample returns Vss) Rs 1 " being conceptually simple and unbiased, MC methods are known to suffer from high variance, which can cause slow learning.
== G(T:). While
Temporal Difference (TD) learning. different approach based on bootstrapping is proposed by Sutton [988]. In this approach, we begin with an initial guess of the value function V and improve our estimates iteratively by sampling transitions (s, a, s') and updating our current estimates to match the bootstrapping targets via V(s) + (ss) + a(r + (s')) V()).. This is also known as the TD(0) method. Unlike MC methods, TD methods have lower variance at the cost of being biased due to their use of bootstrapping targets. Methods based on bootstrapping have been shown to converge faster than MC methods empirically and have been widely adapted in recent deep RL. methods (Mnih]
et al.
2015 2016 Schulman et al. 2017

---

One disadvantage of the TD(O) method is that it only propagates the reward one step at time, which can be slow when the rewards are sparse. Instead, one can utilize the rewards that are up to n-steps ahead and update value estimates by r. Mr++1 +... + +," ((**+n)) This allows the reward information to propagate faster to relevant state-action pairs. Additionally, this view unifies both methods in the sense that TD(O) methods correspond to the case when n while n - oo recovers MC methods. Choosing the parameter n can be seen as a bias and variance trade-off, where lower relies more bootstrapping which could be biased while higher relies more on the sampled trajectory which less biased but could lead to larger variance. In TD(A), these n-step estimations further combined to produce more robust estimate by averaging them exponentially with X Watkins 1989]
Suttonn 1988
TD errors and the advantage function. One way to estimate A(s, a) is to sample transition (s, a, r, s') and compute A(s, a) = Q(s, ((s) Rs -++ The term (s') is known as TD Error and was introduced in the TD Learning algorithm to update the value estimates. This method can also be combined with TD(A) to produce more accurate estimates of the advantage function (Schulman et al. [0155]]
Policy optimization. One important application of the advantage function is in policy optimization.
Williams [992], Suttonn et al. 2000) showed that, for parameterized policy To, the gradient of the
RL objective is given by
Vo E [G(T)] Prog
[G(r)]= Ed (s)
(((ss,)) b(s)) Voro(als),
RES
aEA
where E:s0''' and b(s) is an arbitrary function of the state known as the baseline function. The baseline function can be seen control variate, which can be used reduce the variance of the gradient estimator when chosen carefully [Greensmith et al. 2004). One common choice of ()) is the value function V. which results the advantage function.
In different approach, Kakade and Langford [2002] showed that the difference between the expected return of two policies is directly related to the advantage function, which paved the way for some of the most popular policy optimization methods (Schulman et al. 2015a 2017).
3 Revisiting the advantage function
We begin by examining the motivating example Figure I In this example, it is clear that the optimal decision depends solely on the immediate reward at each state, since the transitions are independent of the actions. However, if we calculate the Q-function, we get
1 1+
Ee Ex ((ulse) a 0+ Erse 7 Tulse) a =
("((),
which shows that the Q-values at each state depend on the policy evaluated at all subsequent states despite the actions having only immediate effects on the expected return. This property can be undesirable as the policy is typically being optimized and varied, which could lead to variations in the Q-values and make previous estimates unreliable. This is also known as the problem of distribution shift, where the data generating distribution differs from the distribution we are making predictions on. Similar problems have also been studied in offline RL, where distribution shift can cause Q Q-function estimates to be unreliable Levine et al. [2020] Kumar et al. [2020). Fujimoto et al. [2019]. One source of this problem is that the expected return by nature associates an action with all its subsequent rewards, whether they causally related not. Alternatively, we would like to focus the effect of an action. But how should we define it?
Questions like this are central to the field of causality and have been studied extensively (Holland 1986, Pearl [009]. We draw our inspiration from Splawa-Neymal et al.) [1990] and Rubin [1944]
in which the authors developed framework, now known as the Neyman- Rubin model, to estimate causal effects of treatments. defines the causal effect of treatment to be the difference between (1)
the outcome of a trial resulting from one treatment, and (2) what would have happened if we had chosen another treatment. The work focused mainly on the problem of counterfactuals, which we shall not delve into. Instead, we focus simply its definition of causal effects. In RL, concern
3

---

ourselves with the effect of each action on the expected return. However, the action space usually consists of more than two actions, so is ambiguous as which action we should compare to. One way to resolve this is to compare each action with what would have happened normally (Halpern and Hitchcock 2015). This way, we can define the causal effect of an action a on some quantity X at state $ by
s - [X]s]
(3)
If we choose X be the return, then
E [G([]]ss: d, a| -E = = V(s) A(s,a)
(4)
is precisely the advantage function, suggesting that it can be interpreted as the causal effect on the expected return. Readers with more exposure the causality literature may wonder why machinery such as intervention or counterfactual are not required to define such quantities. This is possible mainly because we focus on fully observable environments where confounders are not present, and we expect more sophisticated treatments would be required if one wishes to extend similar ideas to
more general problems (e.g., partially observable environments [Bareinboim et 2013
Kumor
et al. 2021).
If we calculate the advantage function for the example environment in Figure
get
(uls), a u <(lls) A (s. = (l)()) a
(5)
which now depends solely on the policy evaluated the state This is because, in this example, the future state distribution is independent of the actions, so the conditional expectation cancels out, leaving only the immediate term. More generally, we only need the expected return for the future state to be independent of the current action.
Proposition 1. Given t' > t such that IE [V(se)
)1 a.] = E [V(ee)
[8e], we have
#'
A(s,, a.) =
rx
8r, at
E
rk
St
(6)
k=
See Appendix A for proofs. This proposition shows that the advantage function is local, in the sense that the advantage of an action is only dependent on the policy of subsequent states to the point where the action stops being relevant to future rewards. This property strikingly similar the Sparse Mechanism Shift hypoth- esis (Schölkopf et al., [202]]. which states that small distribution changes should only lead to sparse or local changes if the representation is causal/disentangled.
Ols.a)
1.2
1.0
5
0.6
3.4-
62-
The condition of Proposition may be too strict hold in most cases. As a relaxation, hy- pothesize that, for a large class of problems of interest, the advantage function is more sta- ble under policy variations compared to the Q- function, making it an casier target to learn. This may partially explain performance improve- ments reported by Baird (193), WWag et al.) (016b). In Figure 2 we verify this empirically using the Breakout environment from the Mi- nAtar suite (Young and Tian 2019) by tracking variations Q and A throughout the training of a PPO agent. We can see that the advantage function remains more stable over the course of training compared to the Quuuctioon From a more theoretical point of view, this softer condition is satisfied when p( (st, ax) R ((sr(8)) for some t' > t. This can happen when, for example, the Markov chain induced by the policy approaches
Policy updates
Figure 2: simulations between each policy update computed with 3072 state-action pairs sampled throughout training. Lines and shadings represent the median and the interquartile range over all sampled state- action pairs. See Appendix C for details.
estimated using MC

---

stationary distribution; as another example, if the environment contains certain bottleneck states (e.g., doorways), that are traversed with high probability, then the state distribution after passing those states would depend weakly on the actions before passing them. On the other hand, if the environment "tree-like", where actions lead to completely different branches, then the advantage function would also suffer from instability.
The question we are now interested in is how to learn the advantage function. In previous work, estimating the advantage function typically relied on using TD errors. This method also faces the problem of relying on an estimate of the value function that suffers from strong policy dependency. Instead, we seek methods which could directly model the advantage function and estimate it from data. Since both Q and can be learned with bootstrapping via TD learning, one might wonder there is an equivalent for the advantage function. Here we give a counterexample showing that the advantage function cannot be learned the same way.
Consider the environment in Figure 3 In this ex- ample, the undiscounted advantage function for is also equal to Equation 5 while the advan- tages for both Su and Sa are zero because there is only a single action in their action spaces. Since the immediate rewards and the advantages for the next states are all zero for the state we conclude that not possible learn the advantages of s based solely on the immediate reward and the advantages of the next state as in TD learning. An intuitive explanation for this failure is that, unlike the value function, which contains information about all future rewards, the advantage function only models the causal effect of each action; therefore, if the next ac- tion is not causal to a distant reward, then the information about that reward is lost.
d
8d
7
Figure 3: A synthetic environment demonstrating the impossibility of learning A from bootstrapping. An agent begins in state s with actions and which transition the agent into states 8 and &a. respectively, with no immediate reward. In both Su and 8a, there is only one action with immediate rewards I and 0, respectively.
4 Direct Advantage Estimation
In this section, we present the main contribution of this paper. We develop a simple method model and estimate the advantage function directly from sampled trajectories.
Before we begin, we first observe that the advantage function satisfies the equality EaEA A"als)A" = O. Functions with this property, as it turns out, are useful for the de- velopment of our method, so we provide a formal definition.
Definition 1. Given a policy n, we say a function f S x A + R is n-centered if for all states S
x(als)f(s,a) =
As discussed in Section 2, MC and nistep methods often suffer from excessive variance due to their sampling nature. It is therefore desirable to find unbiased low-variance estimates of the return. One interesting property of r-centered functions is that, if we transform the reward by r rr A(s az) with any n-centered function A, then the expected return remains unchanged, that is E (G( E ([G'(T),, where GT(T) = Eizo Y'r/. This can be seen as a form of reward shaping [Ng et al. 1999
where the expected return is now invariant to the choice of Å for a fixed policy. Now, consider the variance of the transformed return G'(r): we have Var [G' = E [G'(T)2] - E [G(r)) In other words, minimizing its variance with respect Å is equivalent to minimizing the following quantity,
Σ r' (rr A(stn mii 1=
E
(8)
5

---

Minimizing the variance of the return this way has the additional benefit of simultaneuusly obtaining the advantage function without having to learn the (action- faction-Jvalue function first, as we show in our main theorem.
Theorem 1. Given a policy x and time step 2 O, we say (s, a) E A or S is reachable within t if there exists 0 I t' I such that p( -8, ar=a) > 0 or p(s > 0, respectively, Let
F& be the set of all r-centered functions, A. Ast az), and
A = arg min E AgF.
(G6)-
G(T)) Σ o
Σ
Ae
t'=0
then for all (s, a) that are reachable within 6, we have (s, a) = A (s,
See Appendix A for proofs. Note that the first term inside the expectation is equal to the return, while the summation in the second term is only up to step t. We observe two interesting properties of the advantage function from this theorem. (I) Disentanglement: If (s, is reachable within t, then Å" (s, a) A (s, a) remains invariant when we apply the theorem to any i > suggesting that additional Á.''s do not interfere with existing ones. (2) Additivity: One intuitive way to understand the term Σe 7" Ar to view it as the total effect from the sequence of actions, which simply the sum of individual effects Ap. A similar property was also pointed out by Kakade and Langford [2002), where they showed how the consecutive sum of the advantage function related to the expected return.
This theorem also suggests a natural way to estimate the advantage function directly from sampled trajectories T1. T2...... TN by minimizing the following constrained loss function
)
2
T(O) N Σ
Σ
r2 Ao (81, (11,
In practice, the constraint can be forced upon a general function approximator fe. such as a neural network, by choosing Ag(s,a) = fo(s, -) - fo(ss a). In our method, neither V nor Q is required priori. Instead, the advantage function is directly modeled and estimated from data, and hence we name our method Direct Advantage Estimation (DAE). Unlike learning V or Q where one aims to find mapping between states (state-action pairs) and its expected return, DAE tries to regress the sum of A(s, a) along the trajectories towards the associated returns. This can be interpreted learning distribute the return rather than predict it.
One limitation of this method, similar to the MC methods, is that we have to sample trajectories until they terminate. To overcome this, we show that the value function can be seamlessly integrated into our method, which enables us to perform updates similarly to the n-step bootstrapping method.
Theorem 2. Following the notation in Theorem I be the value function be learned, and
let Viarget (s) be the bootstrapping target and (s)
27
L(Å, V) = E
Σ 1=0
+ ('r ( + Vearge - ((0))
(10)
If V. arg miny L(À, V) and p(so=s) o, then for any Å FG, we have
(*)s)= EE
Σ
r'r + 7" " Viarget (sn) 80=s
(11)
r=
Additionally, let (s) = --"" and (s) = En_ wz(s). If Å = arg min L(Å, V) and (s, a) is reachable within tot 1, then for any V. S.S- R. we have
w. (s)
Ár a)
rr + + rn + 7 Viarget (sn) s1=s, a:=a]
Wn-1(3)
t=0
E
r + --- rn-1 + +7" "-"""""Vaee (S)) 31=5] (12)

---

Note that in Equation 12 the minimizer A depends only on the bootstrapping target Vtarget but not the estimated value function V, and vice versa. One might notice that Equation 11 is actually the multi-step update for the value function. This suggests that we can update V iteratively by letting Viarget = VE-1 and (s) (s) IE
= 7'r + (V-=-1(8)) so=b] which replicates multi-step TD learning. Similarly, Equation 12 can be understood as a multi-step estimation for the advantage function with bootstrapping. At first sight, this may seem to suffer from the same problem of spurious policy dependence due to the use of value functions (see discussion in Section 3). We argue, however, that this can be mitigated by using long backup horizons (i.e. n >> 1) and the
assumption that actions have local effects (i.e. p(5t Sec a.) 23 p(Sy (8)) for some t' > t). If the backup horizon is large enough such that ((8t=n|82,0, (4)) RJ p(sttt// (8)).. then Equation 12 suggests that the dependence on Vunget can be reduced since [B[Vurger ($c+n) (e4tm](s0 [8, 0z] - [angee (
R 0.
In practice, we can sample n-step trajectories and use Equation T0 the loss function to estimate both the advantage function and the value function simultancously. However, naively applying the theorem this way can cause slow learning because the value function is learned only for the first state in each trajectory and distant state-action pairs are heavily discounted. To mitigate this, we further utilize each trajectory by treating each sub- ura-trrjeetory policy. For example, let (su. ag..... .n)) be an n-step trajectory. Then we treat (8s, ai...... , 8n) for all i {o..... ..... - ]} also as a trajectory sampled from the same policy. This can be seen repeated applications of Theorem [ which leads us to the following loss function
trajectory sampled from the same
27
(TV ( Ao() + (sn) vo(ss))
(A E
(13)
In practice we may use single function approximator with parameters 0 to model both Á and V simultaneously as done by Wang et al. [2016b), which reduces the loss function to (0).
Experiments
We evaluate our method empirically by comparing the performance of policy optimization between DAE and GAE on three different sets of discrete control tasks. (1) A synthetic environment based on Figure T where the true advantage function and the expected return are known, (2) the MinAtar Young and Tian 2019] suite, set of environments inspired by Atari games with similar dynamics but simpler observation space, and (3) the Arcade Learning Environment (ALE) [Bellemare et al. 2013). set of tasks with diverse mechanisms and high-dimensional observation spaces.
5.1 Synthetic environment
We consider finite variant of the environment shown in Figure with state space S {ss, 191,... $128 }. and train agents using a simple actor-critic algorithm where in each iteration we (1) sample trajectories, (2) estimate the advantages of the sampled state-action pairs, and (3) perform single policy gradient step weighted by the estimated advantages. For both methods, we use the same network architectures and hyperparameters to train the agents, see Appendix for more details. We assess the quality of the advantage estimation through mean squared error (MSE) between the estimated and the true advantage function (Equation 5) on the sampled state-action pairs in each iteration. Additionally, we evaluate the learned policy by computing the true (undiscounted) expected return by E [2, ra E, (u((s)) Note that the optimal policy in this case is T (ul-)) with expected return Σ, क (u = 128.
The results in demmonstaae that DAE can approximate the true advantage function accurately, while GAE struggles. The inability of GAE to accurately estimate the advantage function is caused by the variance of the n-step returns used in the estimates, which slowly decreases as the policy converges to the deterministic greedy policy. For policy optimization, we see that GAE performs better in the early phase of training, but is later surpassed by DAE. This is likely because estimates from GAE include unbiased estimates of n-step returns that are useful to policy optimization, while DAE relies entirely on the approximated function, which is heavily biased at the beginning. As DAE becomes more accurate later in training, we see substantial gains in performance, which demonstrates drawback of using high variance estimates from GAE
7

---

120
Moan Squared Enor
10
Expected Return
110
100
10"4
80
DAE
GAE
200
200
Iteration (policy updates)
Figure 4: Results for the synthetic environment. Lines and shadings represent the mean and one standard error over 100 random seeds. Left: Mean squared error (in log scale) between the true and estimated advantage function on the state-action pairs sampled each iteration. Right: The (undiscounted) expected return of the policy.
Algorithm 1 PPO with DAE (shared network)
1: Initialize Á, and the policy with 0
2: while not converged do
3: D = 0
4: for L..... Nactors do
5: Sample n-step trajectory 7; (8o, Ao, ru.-.., 8n) with policy
6: D DU (Ti)
7:
end for
8:
② + cop
9:
Vinger, 11 To
10:
for = 1,...... ......,Nuuu Nupdates do
Sample mini-batch of n-step trajectories B from D
Compute Á(s, a) for all (s, a) E B (centered according to μ)
13:
Compute PPO clipping loss L (Equation 49) with stop_ soop_grrdieet(A(G a)), u. and
14:
Compute DAE loss LA (Equation 13 with Å, V and Varget
15:
Optimize loss L (L LA)
16:
end for
end while
5.2 MinAtar & Arcade Learning Environment (ALE)
We test DAE's performance in the challenging domains of the MinAtar and the ALE environments using PPO as the base algorithm. We use the PPO implementation and the tuned hyperparameters from Raffin al. [202]] Raffin (2020). A practical implementation of PPO with DAE can be found in Algorithm (see Appendix for more details 3 For DAE, we tune two of the hyperparameters, namely the scaling coefficient of the value function loss and the number of epochs in each PPO iteration using the MinAtar environments, which are then fixed for the ALE experiments. Note that DAE has one less hyperparameter (A) to tune because it does not rely on TD(A), Additionally, we increase the number of parallel actors for both methods as it substantially speeds up PPO training terms of wall-clock time and has become the predominant way to train PPO agents in complex environments [Rudin et al.) 2022 Freeman et al.) (021). We also test how network capacities impact the performance of both methods by increasing the width of each layer in the baseline network (denoted "Wide"). For ALE, we further test deep residual network architecture (denoted "Deep"") based on [Epeholt al. (2018). See Appendix for a more detailed description of the network architectures and hyperparameters.
Each agent is trained independently for 10 million and 40 million in-game frames for the MinAtar and the ALE environments, respectively. We compare both methods on two metrics, (1) Overall: average undiscounted score of all training episodes, and (2) Last: average undiscounted score of the
Codde available at athhtps://// pettps://githule com httpp/tu...ooo//aa////aaaa hwwph/raaanaeeeeeeaeeea

---

MinAtar
ALE
15.0
DAE (Wide)
DAE (Deep)
Mean Normalized Score
DNE
DAE (Wide)
GAE (Wide)
DAE
10.0
GAE
(Deep)
1.5
GAE (Wide)
7.5
GAE
1.0
2.5
0.5
0.0
0.0
8
10
5
10
15
20
25
30
35
40
(Millions)
Frames (Millions)
Figure 5: Mean GAE-normalized score for the MinAtar (left) and the ALE (right) experiments. The lines represent the average all runs (50 and 10 random seeds for the MinAtar and the ALE, respectively) with each run corresponding to the average over all environments with one random seed, and the shadings correspond to standard error of the mean. The scores are normalized based GAB's (baseline network architecture) average score in the first 1 % frames and the last % frames.
last 100 training episodes, as in Schulman al. [2017]. The metrics reflect how fast agent lears and the final performance of the learned policy, respectively.
From Table D we observe that, given the same network architecture, DAE outperforms GAE in all the MinAtar environments and most of the ALE environments on both metrics. In Figure 5 we present the GAE-normalized learning curves for each domain and architecture, where also a similar trend that DAE is much more sample efficient while achieving better final performance Results for individual environments can be found in Appendix E. Finally, we found that increasing the network capacities consistently improves DAE'S performance while having less impact on GAE. This is likely because the advantage function is directly represented in DAE using the network, such that network capacities should have a larger impact on the performance. Interestingly. we see significant improvements in the MinAtar experiments by switching the Wide network for DAE but not GAE. Note that this gap cannot be explained by the increased representation power in the policy network alone, should also see significant improvements with GAE. Therefore, it suggests that the improvements are likely originated from the ability to more accurately approximate the advantage function. In the ALE experiments, we also observe such jumps in performance for some environments (see Appendix C), but for most environments, both DAE and GAE improved significantly. This suggests that the improved performance can be attributed to the increased capacities in the policy networks, which benefit both methods.
6 Related work
The advantage function was first introduced by Baird 1993 to tackle problems with fine-grained time steps, where Q-Learning could be slow. In policy optimization methods, the advantage function has also been widely used as an alternative to Q-functions to lower the variance of policy updates [Greensmith et al.) 2004 Kakade and Langford. 2002] [Sutonn et al., 20000. More recently, Wang cct al. [2016b] developed a neural network architecture which can jointly estimate the value function and the advantage function to achieve better performance with DQN [Mnih et al. 20155. Our work complements theirs by justifying the use of policy centered functions estimating the advantage function, and showing how it can be combined with multi-step step learning. SShhulman et al. [2015b] proposed GAE, a method which combines TD(A) and TD error to estimate the advantage function, and demonstrated strong empirical results in policy optimization.
The link between causality and credit assignment in RI. has also been explored in Mesnard al. [020)] where the authors used the idea of counterfactuals to construct future-conditiona value function to reduce the variance of policy gradient updates. Connections between causal effects and the
'Hard exploration environments such as Montezuma's 's Revenge are excluded in the normalization because the GAE baseline does not improve consistently.

---

Table 1: Number of environments won by each method. The metrics averaged across all runs before comparison. Results are counted as "Similar" their standard error interval overlaps. See Appendix E for detailed results.
Metric
Model Size
Method
Overall
Last
MinAtar
Atari
MinAtar
Atari
GAE
0
10
Baseline
DAE
5
32
5
30
Similar
0
7
0
8
GAE
0
8
0
Wide
DAE
5
35
Similar
0
6
0
GAE
6
Deep
DAE
35
32
Similar
8
advantage function also studied by Corcoll and Vicente 2020). where they model the dynamics of environments a hierarchical manner with a differently defined advantage function.
7 Conclusions and future directions
In this work, we tackled the problem of credit assignment in the RL setting. Inspired by previous studies in causality, we found that the advantage function can be interpreted as the causal effect of an action on the expected return. We then proposed DAE, novel method that can directly model the advantage function and estimate it from data. Our experiments three different domains demonstrated that DAE can converge to the true advantage function, improve the performance of policy optimization compared GAE in most environments, while also enjoying better scalability with increased network capacities.
Here note some possible extensions DAE. (1) Environments with confounders (c.g., partially observable MDPs) are common in real-world applications, and causal inference is known to be effective in such problems [Bareinboim et al. 2015 Tennenholtz al. 2020, Kumor af. 2021]. We expect similar ideas based on the causal effect proposed in this work may also be useful in those settings. (2) Due to the difficulty of enforcing the centering constraint (Def. [T] in continuous domains, we currently restrict ourselves to discrete action space domains. One potential solution to this problem was outlined in Wang et al. [0166a] where they used a sampling method approximate the centering step for continuous action spaces. (3) In this work we assumed the data is on-policy, an interesting extension would be to use importance sampling techniques such as Retrace [Munos et al. 2016 or V-Trace [Espeholt et al. 2018 to extend DAE to off off-policy settings.
Acknowledgments
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Hsiao-Ru Pan and Alexander Neitz. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S and by the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645.
References
L. C. Baird. Advantage updating. Technical report, WRIGHT LAB WRIGHT-PATTERSON AFB OH, 1993.
E. Bareinboim, Forney, and J. Pearl. Bandits with unobserved confounders: A causal approach. Advances in Neural Information Processing Systems, 28, 2015.
10

---

M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
O. Corcoll and R. Vicente. Disentangling causal effects for hierarchical reinforcement learning. arXiv preprint arXiv:2010.01351, 2020.
S. De and S. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33:19964-1999 19975, 2020.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
C. D. Freeman, Frey, Raichuk, Girgin, I. Mordatch, and O. Bachem. Brax differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 13281, 2021.
S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
J. Y. Halpem and C. Hitchcock. Graded causation and defaults. The British Journal for the Philosophy of Science, 66(2):4134577, 2015.
P. W. Holland. Statistics and causal inference. Journal of the American Statistical Association, 81 (396):945-960, 1986.
S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning, In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 2rXiv:2006.04779, 2020.
D. Kumor, J. Zhang, and E. Bareinboim. Sequential causal imitation learning with unobserved confounders. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan. editors, Advances in Neural Information Processing Systems, volume 34, pages 14669-14680. Curran Associates, Inc., 2021.
Levine, Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Mesnard, Weber, F. Viola, S. Thakoor, A. Saade, Harutyunyan, W. Dabney, T. Stepleton, N. Heess, A. Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. arXiv preprint arXiv:201 /.09664, 2020.
M. Minsky, Steps toward artificial intelligence. Proceedings of the IRE, 49():--33, 1961.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 533, 2015.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
R. Munos, T. Stepleton, A. Harutyunyan, and M. G. Bellemare. Safe and efficient off-policy reinforcement learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1054-1062, 2016.

---

A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278-287, 1999.
Pearl. Causality. Cambridge University Press, 2009.
A. Raffin. RI basclines3 z0o.
com/DLR- RM/rr--baselines3 200, 2020. 2020.
A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. /RLhtttp://mml wttp://tpppppp////////.// 2022242.021....... htmm
D. B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5),688, 1974.
N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference Robot Learning, pages 91-100. PMLR, 2022.
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634, 2021.
J. Schulman, S. Levine, Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference machine learning, pages 1889-1897. PMLR, 2015a.
Schulman, P. Moritz, Levine. M. Jordan, and Abbeel. High digh-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: arXiv:1506.024358 2015b.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, arXiv preprint arXiv: (ttd::170700377, 2017.
Splawa-Neymane D. M. Dabrowska, and T. Speed. On the application of probability theory to
agricultural experiments. essay principles. section 9. Statistical Science, pages 465-472, 1990.
R. S. Sutton. Learning predict by the methods of temporal differences. Machine learning, 3(1): 9-44, 1988.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 2018.
R. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances neural information processing systems, pages 1057-1063, 2000.
G. Tennenholtz, U. Shalit, and S. Mannor. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276-10283, 2020.
Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.0122 2016a.
Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. International conference on machine learning, pages 1995- 2003. PMLR, 2016b.
C. J. C. H. Watkins. Learningfrow delayed rewards. PhD thesis, 1989.
R. J. Williams. Simple statistical gralowng-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
K. Young and Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments, arXiv preprint arXiv:1903.03176. 2019.
12


Reviews:
[{"review_id": "w-pjrOuDEDp", "reviewer": "Reviewer_9WbF", "summary": "The paper considers the problem of learning the advantage function directly (with no need for learning Q first). The authors coined the term $\\pi$-normalized [a bit weird terminology] as any function whose expectation over $\\pi$ is zero. Advantage function is an obvious example. Clearly, if the reward is shifted by any $\\pi$-normalized function, the expected return remains unchanged. This property is then used to minimize the variance of return by making it disentangled from the policy. One main result (via Theorem 1) is that the optimal $\\pi$-normalized function (in terms of minimizing the variance) to be subtracted from the reward is indeed the advantage function of the current policy for on-policy learning. This also gives rise to a learning paradigm with the loss function coming directly from the grounded minimization result in Theorem 1. This is then extended to N-step learning using the value function. Finally, promising experimental results are provided to confirm the method. ", "questions": "- L54: Your definition of reward is indeed *expected* reward (expectation taken oven next state $s’$).\n\n- Paragraph of L150: I generally accept that A is lower variance than Q but using PPO to demonstrate this is not adequate. In particular, I would examine this with a value-based method that directly reinforces Q, instead of AC methods, which normally does not give rise to accurate value function. \n\n- The term $\\pi$-normalized is somehow misleading, as normalization is a process of mapping to one, not zero. Consider a different term; perhaps something like  $\\pi$-neutralized or $\\pi$-nullified. \n\n- From proof of theorem 1 (L473), it looks like that non-zero policy is a *missing* assumption, $\\pi(s,a)\\neq 0$ for all $(s,a)$. This kind of assumption is very normal in policy gradient results. I do not consider it as a downside. \n\n- Using value-function for the N-step case is somehow defeating the purpose as the main premise is to disentangle advantage learning from value learning. Isn't it? \n\n- The last part of the theory (from L232 onwards) is not well-written and requires more explanation and expansion. I recommend removing several parts from the first 4 pages (which are simply reviewing textbook stuff) and expand on the last part of the theory with more scrutiny. In particular, $V_{target}$ is not clearly defined and its dependence on learning $V$ is not clear. The trajectory augmentation seems also to be required. From the text, it is very hard to come up with a clear algorithmic machinery. I strongly suggest to add a algorithm block and summarize the details. \n", "limitations": "No limitation is explicitly mentioned, though from the future work discussion there are some comprehendible limitations: \n\n- Action space is discrete. \n- On-policy learning. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- I liked the theoretical results. Direct learning of advantage function is novel and useful. Additionally, the minimization formulation from Theorem 1 is also interesting. \n\n- Disentanglement of learning from current policy is indeed interesting, as it is also evident from the experimental results. \n\nWeaknesses:\n\n- The implementation of this paper requires several steps, which are somehow distributed in various parts of the paper. A clear algorithm block is missing. \n\n- The second part of the theory is not presented clearly, and is too condensed. \n\n- Minor: I do not understand the emphasis on some \"causal\" concepts. These are neither needed (or even used) nor are they any addition to the presented theory. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c42tCOFCRDK", "reviewer": "Reviewer_8k7E", "summary": "This paper outlines a connection between the causality literature and the advantage function in reinforcement learning. The advantage function can be viewed as an estimate of the causal effect of an action on the expected return. As a second contribution, a theorem shows that the advantage function can be viewed as the solution of a variance-minimization problem subject to a certain constraint. This leads to a different objective for learning the advantage function without resorting to estimating the Q and V value functions. This approach is shown to outperform generalized advantage estimation (GAE) in a suite of standard benchmark tasks. ", "questions": "- Table 1 presents counts for how often an algorithm was better (DAE or GAE). Does this account for the confidence intervals? As a rule-of-thumb, if the confidence intervals overlap, it wouldn't be fair to say one was better than the other. This table could include a number for the the number of environments with approximately equal performance.\n\n- The paper makes a connection between the advantage function and causal effects but, afterwards, doesn't make use of this link. Do you have any other thoughts on how this interpretation could lead to better RL algorithms?\n\n- Concerning Fig. 2, do you have an explanation why the advantage function tends be more stable? Do you think it's related to Prop. 1 and actions have limited effects across time in these RL environments?\nAlso, have you looked at the plot of Fig. 2 for other environments? Do you still find the same thing?\n\n- Lines 162-166: Is it true that in a multi-step setting, the advantage function would be more stable? It seems like delayed rewards are an important factor in explaining the difference between Q-values and advantage functions.\n\n- Any thoughts on viewing the advantage-function as a variance-minimizing state-action-dependent baseline (with a constraint)? How would you compare that to the solution of the unconstrained version of the same optimization problem?\n\n", "limitations": "Yes, I don't see any major issues here.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has two distinct contributions: the connection of advantage functions to causal effects and the new objective for estimating the advantage function. I find both of these to be interesting and of interest to the community.\nMoreover, the paper is well-written and easy to follow, from the organization to the choice of notation.\n\nThe exposition of the benefits of the advantage function over Q-values was well-done. The intuitions was nicely explained and supported by the synthetic experiment and Fig. 2.\nAfter checking some proofs, they seem to be correct and are adequately explained. I had no trouble following them. To me, it's a nice result that you can view the advantage function as a variance-minimizing state-action-dependent baseline (under some constraint). \nThe experiments seem sound with reasonable choices as baselines and good experimental procedures. Sufficient details are presented in the appendix to understand the experiements in-depth. In particular, I thought the shuffling of actions in the synthetic experiment (as described in the appendix) was nice experimental design. The experiment varying the size of the network  also yielded some interesting results. \n\nAlthough I am not very familiar with the causality literature, the link between causal effects and advantage functions is intriguing to me. While this connection wasn't developed too thoroughly in this work, it does outline a direction that could be explored further in the future. \nOverall, I think this is a solid paper and haven't found any major issues, although I do have some questions which I've written in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bd2BxJEX1o8", "reviewer": "Reviewer_tFqS", "summary": "This paper proposes Direct Advantage Estimation, an algorithm which estimates\ndirectly the advantage function instead of the traditional state(-action) value\nestimation. They draw some links with a causal interpretation and perform\nexperiments on a synthetic environment as well as MiniAtari and ALE, comparing\nthemselves to PPO with GAE.\n", "questions": "- As you are sampling on a trajectory by trajectory basis instead of minibatch of\n  frames (l 497): how does your algorithm perform in terms of wallclock time\n  compared to PPO?\n- Fig 4: Does DAE still have lower variance in a more general setting rather\n  than one designed so that DAE can reach 0 variance easily? I am\n  thinking a simple gridworld, the Q and V function can be computed in closed form.\n- Do you use p for the discounted or undiscounted stationary distribution? It seems in eq 39-41 it would be the discounted if you are using an unbiased estimator of policy gradient, however it seems to be used as the undiscounted in the main text.\n- To make sure I understand clearly, what exactly is \\hat{A} in alg 1? Is it just the learned advantage or the n-step TD with advantage function?", "limitations": "Some limitations are addressed with Fig 3. However I think the authors need to make a stronger case for their method:\n- First, a comparison with a more similar algorithm (n-step returns) would be better than GAE, as it would showcase that incorporating the advantage in the TD error is important\n- Then they would need to show why estimating directly the advantage as they do is useful/important/interesting. This could be tested experimentally by comparing with other methods for estimating the advantage.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe method proposed is motivated theoretically, is simple to implement and seems\nto show improved performance compared to the baseline in the experiments. The\npaper is overall easy to read.\n\n# Weaknesses\n- The paper displays a collection of different results and interpretations but\n  few seems actually necessary to explain the method. More precisely\n    1. I find the connection to causality weak overall. In this light any\n       centered variable can be thought of as an instance of a causal power, this does not necessarily mean there is any causal estimation behind.\n    2. The content of section 3 is mostly motivated by the synthetic example.\n       I find this problematic as I believe the synthetic example is a very\n       contrived one. While one motivation for DAE is the credit assignment\n       problem (l22-23) this environment is specifically designed so there is no\n       credit assignment. \n\n- Baselines: while in practice DAE ends up being very close to a n-step return\n  method, the only comparison point in the paper is with GAE. A minimum I would\n  need to recommend this paper for acceptance would be a fair comparison with\n  its 1. n-step return ablation (by removing the effect of A in eq 17 if I\n  understood correctly).\n  Then I think, to show the interest of directly estimating the advantage,\n  another baseline could be to replace your directly learned A by\n  2. Q - V where Q and V are learned like usual\n  3. A = Q - V learned with a duelling architecture.\n  1) would show if your estimator which incoporates the advatange is superior to\n     the tradional TD one\n    2) and 3) would be useful to show whether learning the advantage directly\n       is useful. Indeed while you make a case for the advantage being a quantity of interest, I find there is little evidence in the paper for why learning it directly with your method is important compared to the methods 2) and 3).\n\nMinor points:\n- eq 4: maybe rather than the approximate symbol it would be more precise and\n  exact to say that it is a sample?\n- As you do not seem to be Vtrace or the Impala algorithm, I think using \"IMPALA\" just to mean the architecture can be misleading.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CVDjwzQQQb0", "reviewer": "Reviewer_c2up", "summary": "The authors propose a new method for estimating the advantage function of a policy that does not require first estimating the Q or value function. They demonstrate that on a variety of tasks, it works better than the \"industry-standard\" GAE.", "questions": "(1) Much of the motivation in the intro of the paper centers around being able to directly estimate the advantage function without having to first fit a value function. However, it seems like from Thm. 2 and beyond, the authors actually end up fitting a value function. Why does your proposed method not suffer from the same issues that plague value-based methods that you point out prior (e.g. small policy changes leading to large distribution shifts and inaccurate value function estimates)?\n\n(2) You mention in the appendix that for DAE, one needs to sample entire trajectories rather than mini-batches. Do you have a wall-clock time comparison between DAE and GAE?\n\n(3) The concern about estimating the sum of advantages for normalization reminds me a lot of the difficulties estimating the partition function for maximum entropy / energy-based / exponential family models. I wonder if you could borrow from the rich literature there to solve your problem.\n\n(4) The DAE-Wide results are particularly good. Could you provide some reasons as to why this might be the case?", "limitations": "The authors correctly acknowledge that their method is dependent on discrete action spaces for enforcing the normalization constraint and provide some potential solutions. I think the related work section could be improved as well (i.e. there are other advantage estimation methods outside of GAE).", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) This paper was a joy to read. I appreciated the nuance of the discussion (e.g. the examples of where the advantage fn. was more or less policy dependent, the example of why TD-based methods can't work for advantage functions in general). The paper felt fundamental yet accessible, which is a hard balance to strike.\n\n(+) The experiments were well-done and involved results averaged across many environments, indicating a significant gain in performance when switching from GAE to DAE.\n\n(+) I appreciated the connection to causal effects and especially the Halpern citation :).\n\n(-) In general, making decisions based on expected returns (rather than greedily) is a good idea. Some parts of the first few pages could be read as indicating otherwise. If I were you, I would focus on why advantages are good (cite PPO/Kakade&Langford) and delve into why (they are sort of an appropriate level of \"local\"), rather than discussing how Q functions depend on the policy at future steps (which is somewhat obvious and useful).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept