CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration
Authors: Changnan Xiao
Haosen Shi
Jiajun Fan
Shihong Deng
Haiyan Yin
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
policy iteration

Abstract: We study the problem of model-free reinforcement learning, which is often solved following the principle of Generalized Policy Iteration (GPI). While GPI is typically an interplay between policy evaluation and policy improvement, most conventional model-free methods with function approximation assume the independence of GPI steps, despite of the inherent connections between them. In this paper, we present a method that attempts to eliminate the inconsistency between policy evaluation step and policy improvement step, leading to a conflict averse GPI solution with gradient-based functional approximation. Our method is capital to balancing exploitation and exploration between policy-based and value-based methods and is applicable to existed policy-based and value-based methods. We conduct extensive experiments to study theoretical properties of our method and demonstrate the effectiveness of our method on Atari 200M benchmark.

Under review conference paper at ICLR 2023
CASA: BRIDGING THE GAP BETWEEN POLICY IMPROVE- MENT AND POLICY EVALUATION WITH CONFLICT AVERSE POLICY ITERATION
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of model-free reinforcement learning, which is often solved following the principle of Generalized Policy Iteration (GPI). While GPI is typically an interplay between policy evaluation and policy improvement, most conventional model-free methods with function approximation assume the independence of GPI steps, despite of the inherent connections between them. In this paper, we present a method that attempts to eliminate the inconsistency between policy evaluation step and policy improvement step, leading conflict averse GPI solution with gradient-based functional approximation. Our method is capital to balancing exploitation and exploration between policy-based and value- based methods and is applicable to existing policy-based and value- methods. We conduct extensive experiments to study theoretical properties of our method and demonstrate the effectiveness of our method on Atari 200M benchmark.
1 INTRODUCTION
Model-free reinforcement learning has made many impressive breakthroughs in wide range of Markov
Decision Processes (MDP) (Vinyals et al. [2019] Pedersen 2019 Badia et al. 2020). Overall, the methods
could be cast into two categories, value-based methods such DQN (Mnih et 2015) and Rainbow (Hessel
cet al. 2017). and posecy-based methods such as TRPO (Schulman et al. 2015), PPO (Schulman et al. 2017) and IMPALA [Espcholt et al. 2018).
Value-based methods learn state-action values and select the action according to their values. The main target of value-based methods is to approximate the fixed point of the Bellman equation through the generalized policy iteration (GPI) (Sutton & Barto 2018), which generally consists of policy evaluation and policy improvement. One characteristic of the value-based methods is that unless more accurate state-action value is estimated by iterations of the policy evaluation, the policy will not be improved. Previous works equip pasue-based methods with many carefully designed structures to achieve more promising reward learning and sample efficiency (Wang et al. 2016; Schaul et al. 2015 Kapturowski et al. 2018).
Policy-based methods learn a parameterized policy directly without consulting state-action values. One characteristic of policy-based methods is that they incorporate policy improvement phase in every training step, while in contrast, the value-based methods only change the policy after the action corresponding to the highest state-action values is changed. In principle, policy-based methods perform policy improvement more frequently than value-based methods.
We notice that value-based and policy-based methods locate the two extremes of GPI, where value-based methods won't 't improve the policy until more accurate policy evaluation is achieved, while policy-based methods improve the policy for every training step even when the policy evaluation hasn't 't converged. To

---

Under review conference paper at ICLR 2023
mitigate the defect of each, we pursuit technique that is capable of balancing between the two extremes flexibly. We first study the gradients between policy improvement and policy evaluation and notice that they show a positive correlation statistically during the entire training process. To find out if there exists a way that the gradients of the policy improvement and the policy evaluation are parallel, we propose CASA, Critic AS an Actor, which satisfies a weaker compatible condition (Sutton et al. 1999 and enhances gradient consistency between policy improvement and policy evaluation.
With further delving into the properties of CASA, we find CASA is an innovative combination of value-based and policy-based methods. When the policy-based methods are equipped with CASA, the collapse the sub-optimal solution as the entropy goes to zero is prevented by the evaluation of the state-action values, which encourages exploration. When the value-based methods equipped with CASA, the policy improvement via policy gradient is equivalent to the evaluation of the state-action values and a self- saff-bootstrapped policy improvement, which enhances exploitation.
To enable CASA for a large scale off-policy learning, we introduce Doubly-Robur Trace (DR-Trace), which exploits doubly-robust estimator (Jiang & Li 2016) and guarantees the synchronous convergence of the statc-action values and the state values.
Our main contributions are as follows:
(i) We present novel method CASA which enhances gradient consistency between policy evaluation and policy improvement and present extensive studies on the behavior of the gradients.
(ii) We demonstrate CASA could be freely applied to both policy-based and value-based algorithms with motivating examples.
iiii We present extensive empirical study on Atari benchmark, where our conflict averse algorithm brings substantial improvements over the bascline methods.
2 PRELIMINARY
Consider an infinite-horizon MDP, defined by tuple A, 7), where S is the state space, A is the action space, p x A x S - [0, 1] is the state transition probability functioo,, r : x A + R is the reward function, and is the discounted factor. The policy is mapping π S x A + jo, which assigns a distribution over the action space given a state.
The objective of reinforcement learning is maximize the return, cumulative discounted rewards,
Entr(se, ox)
maximize J Etrajew
where traj = {su, ag, .o.... } is a trajectory sampled by with policy- enviromenvi interaction.
Value-based methods maximize J by estimating various type of value functions: the state value func- tion is defined as (x)
Ex |E, 1s0o = s). the state-action value function is defined as Q" (s, a) E, [x, >rre|5p = 8, ag a): the advantage function defined as A a) = Q" (s, a) v*(ss, The objec- tive of maximizing the value functions in value-based methods can be improved through GPI until converging to the optimal policy. For the approximated state-value function Qa that estimates Q", the policy evaluation is conducted by:
minimize E=[(Q" Qo(s, )))
where Q3 is estimated by various methods, e.g., A-return (Sutton 1988) and Re Muunos et al.) 2016). The policy improvement is usually achieved by greedily selecting actions with the highest sate-action values.
2

---

Under review conference paper at ICLR 2023
= Q
x². O
x, Q
2.
x = p
x".""
Fy ps(l2))
Figure 1: The GPI process in our work. Un- like (Sutton & Barto 2018), we evaluate и by Q instead of V. and improve using policy gradient ascent (pg for brevity) instead greedy. The learning procedure shown by the black arrows, i.e., E + I E I....
Figure 2: GPI with function approximation. Due to the constraint of approximated function space, the idcal policy iteration cannot be actually achieved. The underlying process of GPI with function ap- proximation be regarded doing policy improvement and policy evaluation in an ideal space then being projected back into the approxi- mated function space (Sutton & Barto 2018 Ghosh et al. 2020).
Policy-based methods maximize J by optimizing some parameterized policy mh according to the policy gradient theorem (Sutton Barto 2018).
VoJ E, V (s, a) Vo log To (a s)].
The vanilla policy gradient uses V > 3rr-- Actor-critic algorithms approximate (ss a) in the form of baseline, e.g., IMPALA (Espeholt et al. 2018) adopts (( a) + (s') Vo(s) and uses V-Trace estimate V*
3 METHODOLOGY
3.1 MOTIVATION
We use Vo to estimate V", Qo to estimate Q* and ny to represent the policy, where 0 represents parameters to be optimized. In this work, there is one backbone and two individual heads after the backbone. The advantage function and the policy share one head, and state value function is the other head. Hence the policy reuses all parameters of value functions except that temperature is only for the policy. We keep r static in this work. We use E to represent the policy evaluation, which gives the ascent direction of the gradient by 8 <- A 1
VoQol. We use I to represent the policy improvement, which gives 0 + 0 + E, [[Q Vo) Va log mo).
Let's recap the GPT process shown in Figure 1 To get rid of the function approximation error, first assume the approximation function enjoys infinite capacity. We use < x, y > to denote the angle between two vectors, where x, y >= arccos( |[=]>T with arccos [-1, 1] > [0, T.. We define an important notion B, which represents the angle between the gradient ascent directions and E, as follows,
8 def
F F - -o) [_ Vo) V. log To) >
(4)
When i.e.cos(3) = 1, I and E become parallel to each other, which the blue arrow Figure and there is no conflict between the gradient ascent directions of I and E anymore. When 3 = x/2 i.e.cos (5) 0, I and E are perpendicular. When 3 x Le.cos(3)) = -1, and E are toward exactly opposite directions.
Next, assume the representation capacity of the approximation function is limited. When the function approximation is involved, i.c. Q is estimated by Q9 and π is approximated by ne. from the view of operators (Ghosh et al. 2020), each of I and E can be further decomposed into two operators, shown Figure One is to do the policy improvement and the policy evaluation, the other to project into the restricted function space. When 8 0, GPI with function approximation would involve two projection
3

---

Under review conference paper at ICLR 2023
390
PP0
2.5
Figure 3: Return, X- cos(8) and entmpy. PPO adjusted with two additional versions to evaluate state- action values. R2D2 uses a surrogate policy to approx- imate policy gradient. En- tropy of R2D2 is entropy Boltzmann policy on state- action values. Details are in Appendix B
Entrope
339-
230-
130
10 ww
Millions frames
Millions frames
Millions frammes
Millions of frames
2.7
Breakout
1.00
2375
232
Millions of trames
Millions of trames
Milllons of trames
Millions frames
operators in each iteration, which introduces inevitable approximation error. When 3 O, if the function approximation error not considered, find that the gradient conflict between and would be totally climinated. If we consider the limitation of the approximation function, similar to the blue arrow in Figure [ one iteration (represented by two black arrows and dotted arrows) can be united into one arrow and one dotted arrow (not shown in Figure 2 but analogy to the blue arrow in Figure where the gradient conflict is eliminated and the two projection operators are reduced to one correspondingly.
As stated above, if 0 holds, we can expect that the gradient conflict between the policy improvement and the policy evaluation eliminated and the function approximation error could be reduced. However, B is usually estimated by sampling with stochasticity. It's difficult to let 3 by optimizing 0. Instead, we consider another notion X by removing step sizes and taking expectation outside, where the angle each state is fully controllable by 6.
def E. |cos VoQo, V log u >|.
fact, x is highly correlated to compatible value function (Sutton et al. 1999). and Theorem |shoss that x 1 is the necessary condition for the compatible condition V&Qa= Ve log Te. which is weaker compatible condition. More details about compatible value function are in Appendix A
To further understand the behavior of 3 and x. we track cos(3) and x of two algorithms PPO and R2D2 as representatives for policy-based and value-based methods, respectively. We show an important fact in Figure Ihhtt both x and cos(3) (3) are statistically positive for both original version and adjusted versions, which means that arccos (x) and /3 are likely to be less then т 7/2 with neural network approximated functions. The aforementioned conceptual and empirical findings inspire raise the following question GPI: whether we can guarantee 1, so that cos is also closer to 1.
3.2 FORMULATION
Denote T E R4 be a positive temperature and sg be stop gradient operator. CASA can estimate Vi and Ag by any function parameterized by 6, where тe and Qo are derived as follows:
(-(+s)) softmax (Ao(s, )//r),
As(s,a) Ao(s, a) Σ (o)))
(6)
a²
Q0(8,a) = Āo(s, a) (s))
Note that there exist two sg operators in equation 6 The first sg operator is used for computing advantage Ag Ag EA[1A) Ag sg(m)) Ag. where the sg operator here guarantees the gradients between
4

---

Under review conference paper at ICLR 2023
policy improvement and policy evaluation are parallel, which elaborate later. Intuitively, this sg operator also means that we keep To unchanged when evaluating the policy no. The second sg operator exists in Qo = Ag + sg( Vq). As (Chen & He 2020) regards sg in siamese representation learning as case of EMMalgorithm DDmmsteer et al. 1977), similar interpretation exists here. Qo As + sg decomposes the estimation of Qo into a two stage problem, where the first is to estimate the advantage each action without changing the expectation, the second is to estimate the expectation.
The equation
G includes a straightforward refinement of dueling-DQQ We know dueling- DDQ estimates Q" by Qe Ag Va. but it cannot guarantee E, [Aa] 0 i.c. Ex [Qa] V9 due to the function approximation error. But if we estimate Qx by Qe = Ag - B [Ag] + Va. it satisfies the necessary condition E, [Qa] = Va without loss of generality.
3.3 PATH CONSISTENCY BETWEEN POLICY EVALUATION AND POLICY IMPROVEMENT
For brevity, we omit 0 and V, Q, A, π are all approximated functions. Denote the estimations of V and Q as V and Q" respectively. For instance, one choice is to calculate V and Q* by V-Trace (Espeholt et al. 2018) and ReTrace (Munos et al. 2016) respectively.
At training time, the policy evaluation is achieved by updating 9 minimize,
(0) [,v - V) LQ( En[ - Q)
which gives the ascent direction by:
Volv(0) E,, [(V V) Volo(0) Ex [[Q Q)
(7)
And we make the policy improvement by policy gradient, which gives the ascent direction by:
VoJ(T, E. [T(@" v)Va log T|
(8)
where J(T, -F. TE >rr.) It takes an additional т, which frees the scale of gradient from т.
The final gradient ascent direction of f is given by:
oi VoLv + VoLo + az VoJ.
(9)
With (V, Q, T) defined in equation 6 by Lemma E.1 we have,
V (1 π) VoA = T Vo log
(10)
For brevity, denote the shared gradient path as g (1 V.A
Plugging equation into equation eqqutionn g we have,
VOLQ E | Q)g VoJ
E |( -Vgg
(11)
By equation V VLLQ and VeJ walk along the same vector direction of gradient path for each state. By equation 10 this exactly the case X = 1. Since parameters to estimate Q and π are shared except for T, we call it Critic AS an Actor
If we make a subtraction between VoLq and VoJ, we have,
VoJ VoLq + E. [Q - V)g).
(12)
We know Ex [[Q - V'g] is a colf-bootstrapped policy gradient with function approximated Q. Recalling the fact that the balue-based methods improves the policy by greedily selecting actions according Q, if we apply VoJ on f, it additionally utilizes Q to do policy improvement. This a more greedy usage of Q to improve policy than its usual usage.
5

---

Under review conference paper at ICLR 2023
If we exploit the structural information (V.Q. .) defined by equation by Lemma E.2
E [[Q
V)gl = FB IQ - V) log = - V m
then we have,
V VoLo = VoJ
(13)
The equation 13 shows VaLq is a policy gradient with entropy regularization. If we apply VoLo on 0 for policy-based methods, an entropy regularization works implicitly by 02 Vo I.Q in equation which prevents the policy collapse sub-optimal solution.
3.4
DR-TRACE AND OFF-POLICY TRAINING
DR-Trace
VTaaae ReTrace
60" =r; + (V(4)) Q(s),
8.74 =n + YV (o (Ft /0(e)))))))))1))))))))) (
va
E, + Exso 2 -up
1 Ex 2 -u44
Qa
BplQc+ Exz0 2 19u41444- (1 (ess0))e-2)00
B, + 2220 2 qu1144)
V
Bu V.
B,lpddn + 1/, V. loga
Table I: Comparison between DR-Trace and V-Trace/ReTrace.
To enable off-policy training with behavior policy J4, one choice is to estimate V and Q equation 7 and equation 8 by V-Trace and ReTrace As CASA estimates (V, Q, T). applying Doubly Robust (Jiang & Li, (016)) is feasible and suitable. We propose DR-Trace and find the convergence rate and the fixed point of DR-Trace are the same as V-Trace's according to its convergence proof. For completeness, we provide DR RR-Trace and its comparison with V-TraceReerras in Table I More details are in Appendix
4
EXPERIMENTS
4.1
BASIC SETUP
We employ Leamer-Actor pipeline (Espeholt et al. 2018) for large-scale training. Motivation and ablation experiments on PPO and R2D2 don't use LSTM, only experiments on CASA+DR- use LSTM (Schmid) huber 1997), which is for comparison with other algorithms. We use burn-in (Kapturowski et al. 2018) when LSTM is used. All estimated values share the same backbone, which is followed by two fully connected layers for each individual head. We use no intrinsic reward and no entropy regularization in any experiment. We find that using life information can greatly increase the performance of some games. However, to be general, we will not end the episode life lost. All hyperparameters are in Appendix F
For brevity, we denote VLV = [V VoVVV, VLQ = E,I -o) VQol and VJ = ELI V log Te]. where expectation batch-wiss wise average in our implementation. When we write < a. > with a E {VLv, LQQ, VJJ. we firstly calculate batch-wise averaged gradient of and b, then we calculate the angle in-between. When we write cos < VQ, V log > or X. we mean Fr [cos VoQo, Vo log me >), which firstly calculates element-wise cosines and then takes batch-wise average. To avoid numerical problem, 2-2
we calculate z-y || by maxx[[]]
_10 [ymaa([[pl]]1003 3
4.2 APPLICATION OF CASA ON REPRESENTATIVE ALGORITHMM
CASA is applicable to existing algorithms. We take PPO and R2D2 for demonstration. The application of CASA PPO is straightforward. Applying CASA on R2D2 is impossible either e-greedy policy
6

---

Under review conference paper at ICLR 2023
PPO
PPO+CASA
R2D2
R2D2+CASA
(V. A) = Ab)
(V. A) = (Vo, Ap)
Func. Approx.
(V, logit) = (Vo, logito)
= softmax(A//
(V, (Vo,
== softmax (A// (A/T)
softmax(logit)
A= A =g(TT .A
Q = A+
A A (g(
Q = 1 + sg(V)
Q Ā VV(V)
Gradient
0.5VL - VJ
L + VLO + VJ
VL
0.5 + VL + VJ
Table 2: Examples of applying CASA on policy-based methods (PPO) and value-based methods (R2D2).
arg max Q policy breaks the gradient. This problem is the same calculating the gradients of policy improvement for value-based methods. We use a surrogate policy Nsurrogate = softmax A/T), which discussed in Appendix B Table Tablee[summnn summarizes adjustments of function approximations and training gradients.
Since PPO+CASA and R2D2+CASA have the same function approximation, recalling the fact that value- based methods improve the policy when improve the policy for every step, we can balance the two flexibly with
accurate evaluation is achieved and policy-based methods
by on, 02, 0g in equation |
In Figure 3 algorithms with CASA show much higher (3s and X- PPO+CASA does exploration than the original PPO, as the entropy of π doesn't easily drop to zero. R2D2+CASA tends to distinct the state-action values, where we use the entropy of Q measure how greedy the current state-action values are.
4.3
BEHAVIOR OF GRADIENTS ON DIFFERENT STRUCTURES
PPO+CASA
As sg(TT) Ao sg( (Vo)
Table 3: Behavior of gradient on different types. Type 1&:2 are CASA-like structures, where type removes of tt and type 2 removes sg of Vs. Type 3&44 are tueling-like structures, where type adds sg to V dueling-Q and type dueling-Q. Type uses a new head to estimate Qo separately, which can considered as an auxiliary task to estimate Q".
type
Q Ag mg Ay + sg( Va)
type 2
Q Ag sg (Ta) Ao + Vo
type
Q Ag + sg(Vo)
type
Ag Vi
type 5
Q Q8
Though we show that CASA satisfies VQ x log æ, which means X = 1, it's unknown if the structure of try several different
CASA is unique. As Q A A-E,, [A] sV() is direct refinement of Bueling-DQN structures of PPO+CASA. All settings of estimating state-action values are shown Table 3 We always use 0.5 VL VLQ + JJ as the training gradient. We present Breakout and Qbert in Figure 4
For the sake of clarity, we group PPO+CASA and type 3 as sg-V group, type and type 4 as no-sg-V V group. The sg-V group has higher x and higher (Bs( which is closer to the compatible condition and the consistency between two GPI steps, and no-sg-V group is always worst than its contrast in sg-V group.
PPO+CASA has X and the highest cos(3). Type has less returns than PPO+CASA. Hence, when applying a CASA-like structure, stopping the gradient of π is always preferred.
Type 5 uses individual head to estimate Q", which performs the worst. Hence, well-designed CASA-like or dueling-like structure is always preferred.
By scatter plot and box plot in Figure X and cos( positive correlated depending different structures. This phenomenon answers part of the last question of Section 3.1 for these specific designed structures, and cos(3) show positive correlation.
7

---

Under review conference paper at ICLR 2023
Con 2 22
02
Million Trammss
Miliions of frammss
Miliom Trames
Cle Wile 7/'
Figure The ablation results evaluated on Breakout (top row) and Qbert (bottom row). From left to right the return curve, X. c3s(3), scatter plot of (x, (3)( and box plot of (x, cos( Each scatter point is one batch sampled from every consecutive 100 batches. Each box is the interquartile range of scatter points.
4.4 EVALUATION OF CASA ON ATARI GAMES
We present an extensive evaluation on CASA, where we train CASA + DR-Trace on 57 Atari games and report the results in terms of two metrics. The first is Human Normalized Score (HNS), which normalizes the reward by random policy and human expert policies. The other is Standardized Atari BEnchmark for RL (SABER), which normalizes the reward by random policy and human world records, where the normalized score is capped by 200%. SABER considered because recent studies show that the median HNS could easily get hacked by the algorithm since it is sensitive improvement on a small subset of games. Table summarizes the results.
Mean HNS
Median HNS
Mean SABER
Median SABER
Rainbow
873.97
230.99
28.39
4.92
IMPALA
957.34
191.82
29.45
4.31
LASER
1741.36
454.91
36.77
8.08
CASA
1941.08
246.36
36.10
10.29
Table 4: Evaluation scores for the methods on Atari benchmark presented in %.
Note that CASA is a variant of IMPALA with DR-Trace, and it achieves substantially better records than IMPALA across all the evaluation metrics. It also scores substantially better than all the methods in terms of mean HNS and median SABER scores. Though off-policy methods are known as privileged for HNS evaluation due replay, CASA outperforms strong off-policy bascline Rainbow. Though LASER outperforms CASA in Median HNS and Mean SABER, CASA outperforms it in median SABER and mean HNS. Overall, the aforementioned results demonstrate the conflict-averse strategy efficiently boosts the performance in large- scale training scenarios and outperform strong on/off-policy algorithms. Hyperparameters and individual games are presented Appendix and Appendix G respectively.
5 RELATED WORKS
Both value-based policy-based approaches comply with the principle of GPI, but two GPI steps coarsely related to each other such that jointly optimizing both functions might potentially bring conflicts. Despite of such erucial issue in GPI with function approximation, most decent model-free freef algorithms adopt standard
8

---

Under review conference paper at ICLR 2023
policy improvemeendevallttio regime without considering conflict diminishing properties. The issue of reducing conflicts among multiple models trained simultaneously was considered in earlier machine learning literature, such as for robust parameter estimation for multiple estimators under incomplete data (Robins & Rotnitzky 1995; Lunceford & Davidian 2004 Kang & Schafer 2007) and multitask learning with gradient
similarity measure (Chen et al. 2020 et al. 2020 Javaloy
Valera
2022).
When the idea is introduced to reinforcement learning, earliest attempts tackle conservative and safe policy iteration problems (Kakade & Langford 2002] Hazan & Kale 2011 Pirotta et al. 2013). Recently, more works have emerged study GPI in fine-graines manner. In (Ghosh et al. 2020). new Bellman operator is introduced which implements GPI with a policy improvement operator and a projection operator, where the projection attempts find the best approximation of policy among realizable policies. In (Raileanu & Fergus 202I). the policy and value updates are decoupled by approximating two networks with representation regularization. In (Cobbe et al. 202T), GPI is separated into a policy improvement and feature distillation step. On contrast to the aforementioned works, we tackle the conflicts in GPI at the gradient-level, with theoretical analysis. Our work is related (Nachum et al., 2017). which utilizes both the unbiasedness and stability of on-policy training and the data efficiency of off off-policy training to form a soft consistency error. Our work bridges the gap between the two GPI steps from an alternative angle of establishing closer relationship between policy and value functions in their forms, without the focus on off-policy correction. Due the difficulty of controlling the gap between GPI steps, we instead consider X. The condition X = 1
close to compatible value function (Sutton al. 1999 Kakade 200I), shown in Section 3.1 and Appendix A
6 LIMITATION
It's noticeable that CASA is only applied on discrete action space for now. We further find CASA applicable to any function approximation that is able to estimate advantage functions of all actions. We provide additional discussion on continuous action space in Appendix C
Since shares all parameters of value functions, it brings X but sacrifices the freedom of to be parameterized by other parameters. We conjecture that CASA is one endpoint of a trade-off curve between X and the freedom of t, where the other endpoint is that ı shares parameter with value functions.
7 ETHICS AND REPRODUCIBILITY STATEMENT
This paper is aimed at academic issues in deep reinforcement learning, and the experiment used is also in the early stage, but may provide opportunities for malicious applications of reinforcement learning the future. We describe all details to reproduce the main experimental results in Appendix F
8 CONCLUSION
This paper attempts to eliminate gradient inconsistency between policy improvement and policy evaluation. The proposed innovative actor-critic design Critic AS an Actor (CASA) enhances consistency of two GPI steps by satisfying a weaker compatible condition. We present both theoretical proof and empirical evaluation for CASA. The results show that our proposed method achieves oftte-of-the-at performance standards with noticeable performance gain over several strong basclines when evaluated on ALE 200 million (200M) benchmark. We also present several ablation studies, which demonstrates the effectiveness of the proposed method's theoretical properties. Future work includes studying the connection between the compatible condition and the gradient consistency between policy improvement and policy evalution.
9

---

Under review conference paper at ICLR 2023
REFERENCES
Adrià Puigdomênech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. arXiv preprint a:X1v:2003.13350, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566. 2020.
Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020.
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2020- 2027, 2021.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B, 39(1):1-38, 1977.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint wrXiv:1802.01561 2018.
Dibya Ghosh, Marlos C Machado, and Nicolas Le Roux. An operator view of policy gradient methods. Advances Neural Information Processing Systems, 33:3397-3406, 2020.
Elad Hazan and Satyen Kale. Better algorithms for benign bandits. J. Mach. Learn. Res., 12:1287-1311 2011.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot. Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298, 2017.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization multitask learning, In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
Nan Jiang and Lihong Li. Doubly robust off- policy value evaluation for reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 652-661. PMLR, 2016
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales. Sydney, Australia, July 8-12, 2002, pp. 267-274, 2002.
Joseph DY Kang and Joseph L Schafer. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical science, 22(4):523-539, 2007.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. International conference on learning representations, 2018.
Jared K Lunceford and Marie Davidian. Stratification and weighting via the propensity score in estimation of causal treatment effects: comparative study. Statistics in medicine, 23(19):2937-2968 2960, 2004.
10

---

Under review conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Remi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-ppolicy reinforcement learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1054-1062, Curran Associates, Inc., 2016.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2775-2785, 2017.
Carsten Lund Pedersen. Re: Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 2019.
Mattco Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandricllo. Safe policy iteration. In Proceedings of the 30th International Conference Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 307-315, 2013.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8787-8798, 2021.
James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429):122-135 1995.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05552 2015.
Sepp Hochreiter; Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. In International Conference on Machine Learning, pp. 4555-8554. PMLR, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: /tfv://70.0000 107770074436 06347, 2017.
Richard S. Sutton. Learning predict by the methods of temporal differences. Mach. Learn., 3:9 44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances neural information processing systems, 12, 1999.
Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. Is deep reinforcement learning really superhuman on atari? leveling the playing field. arXiv preprint arXiv: 04683, 2019.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):500-355 354, 2019.

---

Under review conference paper at ICLR 2023
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning. pp. 1995-20M0 2016.
Tianhe Yu, Saurabh Kumar, Abhishck Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
12

---

Under review conference paper at ICLR 2023
A COMPATIBLE VALUE FUNCTION
The original policy gradient with compatible value function is stated as follow.
with parameter w and Th be a policy function with parameter If Qu satisfies E, [Q Qw) Qwll 0 and V V log ne, then
Theorem (Sutton et al. 1999 ). Let Qu be a state-action
VoJ Ex[Qu Vo lo]
If we let w = 0 in Theorem where Qu and Ha share parameters, we have the following theorem.
Theorem Let Qo state-action function with parameter and ma policy function with parameter 0. If Qo satisfies E,, 1(Q Qo) VoQl = 0 and VoQu Va log :Ta then
V ToJ = B QQ V log mo).
Define
X def E, jcos < VoQo, V log u >|
We show that X is the necessary condition for the compatible condition VoQo Vo log mg. Theorem 1) VoQ x V log me for all states, then X
ii) 1f 1, then VaQa & Ve log Ta for all states.
By Theorem 3 X = 1 equivalent to VaQa x Va log ne, and VoQa x Vg log ma is the necessary condition for VoQs= = Ve log ne, hence X = 1 is the necessary condition for VaQa Va log ma.
Proof. i) Since VaQa x Ve log we have < VaQa, V log >= 0. By definition of x, we have
X
E jcos
VoQo. V log T >| E [1] 1.
ii) Since X and cos(x) monotonic decreasing as x goes from æ, the equality X 1 only holds when all states satisfy cos < VeQe, V log ma >= 0, which means VoQa oc V log TT.
13

---

Under review conference paper at ICLR 2023
B GRADIENTS BETWEEN POLICY IMPROVEMENT AND POLICY EVALUATION
Function Approximation
Train Gradients
Cosine of Interested Angles
PPO
(V,logit) = (Vo, logilo) softmax(logii)
Ly + VJ
PPO ver.1
(Q, logit) = (Qu, logits),
L.5V + VJ
008 < V V4g. VJ >
π softmax(logit)
C0s < VQ, V log >
V = sg(z))
PPO ver.2
(Q, logit) = (Qo. logits),
0.5V + V VJ
008 < Vl.g. VJ> cos < <V.
pi softmax(logit) V = =g(x))
V Vlog=>
PPO+CASA
(V, = (Vo, Ao):
0.5V + -L4 + VJ
008 < V4g, VJ> COS < VQ, 1 log >
T = soffmax(A//))
= A - $T( A
A vg(V)
Table 5: PPO is the original PPO. PPO ver.1 and PPO ver.2 are adapted versions to calculate VLQ. PPO+CASA applying CASA on PPO, which is described in Sec. 4.2
Function Approximation
Train Gradients
Cosine of Interested Angles
R2D2
(V. (Vo As) =
VLQ
cos < VLQ, VJ >
= softmax(A/7
R2D2 ver.
(V, A) = (Vo, As)
L.5VL + VLQ
00S < VLo, VJ> >
Q ++1 V
softmax (A/T)
R2D2+CASA
(V, A) = (Ve, Ag),
0.5V + +VL +VJ
COS < VLg, VJ >
π = softmax A/T)),
Á A (
Q = + VV)
Table 6: R2D2 is the original R2D2. R2D2 ver. is adapted version to include V Lv for training. R2D2+CASA is applying CASA on R2D2, which described in Sec. 4.2
To understand the hehavior of
def E, 1(1 Qo) VoQol I. Vo) Vo log
and
[cos VoQo, Vo log n
in reinforcement learning algorithms, choose PPO representative for policy-based methods and R2D2 representative for value-based algorithms.
Define
(0) E-((V= Vo)a), LQ(0) [T Q.)
and
VoJ(0)) E, [(Q Vo) V, log T|.
14

---

Under review conference paper at ICLR 2023
We usually have above three kinds of loss functions in reinforcement learning, which aim to estimate the state values, state-action values and the policy. We do not talk about the estimations of V and Q" as they are estimated as their usual way of PPO's and R2D2's. All hyperparameters are listed in Appendix F
For brevity, we write
COS < VQ. Vlog >= B, ccos < VoQu, Vo To
no >/,
and
COS VLQ, VJ >= cos < lL Qo) EII(Q" V6) log kT. 2,
Cos VLV, VJ >= cos < E, [IV Vo) I[Q" - V.) log πol >
COS 1V,, VLQ > cos E [(V VoVo EN
>
The fact that PPO only has VoLv and V and R2D2 only has VoLg the main difficulty to track cos(3) and X. To solve the problem, we adjust PPO and R2D2 with different versions.
For PPO, we displace the estimation of Vo by sg()) Qo, where Qo estimated by function approximation and Vo is estimated by taking the expectation of Qs. All versions of PPO are listed in Table
For R2D2, we point out that though we apply e-greedy to interact with environments, e is only used for exploration and the final target policy of value-based methods is simply arg max Qo. Because arg max Qa breaks the gradient, we use a surrogate policy to approximate the gradient policy improvement. Since
softmax(Qo/1) +=0+ arg max Qo. use ssurrogate softmax(Ao) (Aa/T) to calculate the policy gradient. We only use Teurrogate on learner to calculate the gradient, where the policy that interacts with environments is still e-greedy, All versions of R2D2 are listed in Table 6
R2D2 uses dueling structure and softmax(Ae/T)
15

---

Under review conference paper at ICLR 2023
C ON DISCUSSING APPLICATION OF CASA ON CONTINUOUS ACTION SPACE
As we can see CASA is only applied to discrete action space in the main context, we make discussion on whether CASA is applicable on continuous action space. For brevity, we let and write equation 6 as:
π = softmax (A),
Å = -=
(14)
Q +=A++s(V))
The difficulty comes from estimating two quantities, one is softmax(AA) (A), the other EE, [A]. This comes from the fact that discrete action space is countable so these two quantities are expressed in a closed-form, while continuous action space is uncountable so an accurate estimation of these two quantities is intractable. We can surely apply Monte Carlo methods to approximate, but more elegant close-form form expression may be preferred. Then this becomes another problem: how to estimate (stale-action values / advantages policy probabilities) of all actions in continuous action space efficiently without loss of generality? This is another representational design problem, which is out of scope of this paper. so we don't touch much about it. But with the hope of inspiring better solution to this problem, provide one practical way of applying CASA on continuous action space based on kernel-based machine learning.
Let ag..... ag to be basis actions in the action space. Let A(s, au),...... A(s, a&) to be advantage functions for tuples of states and basis actions. They can either share parameters or be isolated. Let K(..)) be a kernel function defined the product of two action spaces. For any a in the action space, we can estimate A(s, a) by a decomposition such like
A(s,a) 2. (K(a),a)A(ssa) ao) + +... + K(ax, a) A))(s) ax),
1
Zu
where Zn EEE a(a;,a) is a normalization constant.
Since K( K(, a) a closed-form function of a, and [A(s, oo), A(s, ag) H is finite, we can make a closed- form expression of both softmax A) and E, [A]. Then we can apply CASA directly on this expression, with one function estimates V and the other function estimates advantages of all actions closed-form with only state as input. The policy is defined directly by softmax of all advantages. In details, we define
(s. = exp(A(s.a))/ f. exp((A(s. a))da
(15)
A(s,a) A(s,a)
A()
((s,a) = A(s, + ss())s)).
Then it satisfies the consistency of CASA on continuous action space.
Vlog VA(s,a) V.a exp( A(s, d)
S. exp( A(s, a)
Ja exp(A(ss a) VA(s, a)da
VA(s,a)
J. exp( ((s,,ss a))da
VA(s,)) / exp(A a))
Jo expl (osp(A((sa))dl VA(s,
VA(s,)) /. v(s,a)VA(s,cddd
V)(,s)) = ()),,)).
16

---

Under review conference paper at ICLR 2023
D DR-TRACE
As CASA estimates WV,QrT] n), we would ask i) how to guarantee that TVTraee = 5 ReTraces ii) how to exploit (V. Q, ④T) to make better estimation. Though we can apply V-Trace estimate V and ReTrace estimate Q with proper hyperparameters to guarantee TVTrace = Recrce it's more reasonable to estimate (V, Q) together. Inspired by Doubly Robust, which is shown to maximally reduce the variance, we introduce RR-Trace, which estimates V by
VRR(s)) def (x((( + +I+ 420
x Cjeeth-
where j is the behavior policy, 5D def rt + (V(+++1) Q(st, az) is one-step Doubly Robust error, def I Ct+i-
def Pt min Se p) and et def min p c) are clipped per-step importance sampling, C(pt44]
With step Bellman equation, we estimate by
Qbn(s, at) del
bu + VB((s++1
= =Ep(Q(s,, at) + Σ *tth -
k20
where Pi,k = 1ęk= + 1[ks0)Pttk.
Theorem Define Ā [A], Q = A sg(V),
T(Q) def ( and Σ x (111:44--100.000F
k20
((V) del (V) +
* ClCttB--1PP Dil
k20
(QQ.V) (g(Q) - E. [Q] + 9 (V). (V)).
4 (Q,V= = 2 (2/(1))) (Q.)))
then (Q,VV + that corresponds to min (pu( (a) s), ([als))
=aass) EvE min {p ()) π (b|s)
as n + +00.
Proof. See Appendix E Theorem
F.1
Theorem shows that DR- Trace is a contraction mapping and (V, Q) converges to (V* (*) that corresponds
min (pu(als), T(als))
(alss)
Ese min (p (), n (b|=)
According to our proof, DR-Trace should work similar to V-Trace and ReTrace, as the convergence rate and the limitation are same. We compare DR-Trace with V-Trace+ in Figure where we replace estimation of state values by V-Trace and estimation of state-action values by ReTrace. We call V-Trace+ReTras as No-DR-Trace for brevity. No-DR- Trace performs better on Breakout and ChopperCommand, but fails to make a breakthrough on Krull. Recalling the fact that Doubly Robust can maximally reduce the variance of Bellman error, No-DR-Trach is less stable but also potential achieve better performance. conclusion cannot be made about No-DR- Trace, as this phenomenon means that No-DR- Trace is less stable than -Tacee,e, but it also holds the potential to achieve better performance.
17

---

Under review conference paper at ICLR 2023
Breakout
ChopperCommand
Krull
700
retrace
100000
Average Return
600
drtrace
Average Return
800000
Average Return
500
600000
400
60000
300
400000
40000
200
200000
20000
100
25 50 100125 150155200 Millions of frames
50 100125 1501752000
50 100125
Millions of frames
Millions of frames
Figure 5: Ablation study for w/wo DR-Trace Breakout, ChopperCommand and Krull.
PROOFS
Lemma E.1. (i) Define = sofbmaz(///)) then V log π (1 (1-1) T) VA (ii) Denote sg to be stop gradient and define A = A - Ex[A], Q À + VV).. then VQ VA.
Proof. As Q = A+ sg(V) A xg(m) (V). it's obvious that VQ = (1 111-x)) x) VA.
For log n. it's standard derivative of cross entropy, so we have V log (1 (I-x)V(A/7) (1- EA.
TA.
Lemma E.2. Define Ā A [A], Q Ā Vg(V). (A///m/r(A//)... then
E, [Q V log T| -
Proof. Since
म exp(A/T)/2. Z= / exp(A/T).
we have
A = 7 log + r log Z.
Based the observation that E,, [f(s) log (- [s] = 0, we have
F E [A]
V log ग| = 0,
Ex [log
log π| 0.
On the one hand,
Ex [[Q - V)V log π = E, [AV log π] - E. [A] V log π
riE,, [log V log क| + 7E, (log Z V Vlog
-Ex [log . V log T].
18

---

Under review conference paper at ICLR 2023
On the other hand,
VH
mi log m
VT, log T2 J/ V log T.
VT; ni ni
m. log T log ITς
E, [log V log
Hence, EE, [[Q V)V lo -
Theorem E.1. Define A = A [A A sg(V). Define
T(Q) def E, [Q(1 an)
#20
g(V) E (s.)V(s)) + Σ k20
7xt - DR
(QQ (g( Ex [Q] + 9 (V), (V)), 2 (Q.V) = (Q.
then W (Q,V) + (Q*, va) that corresponds to
min (pp(als), [(a|s)) #(als) EsEA min (pu(bls), [b]s))
11 +00
Remark. = exactly how Q is updated training time. Since Q A sg(),, we apply gradient ascent on Q and V in directions (0) and V (0) respectively, change of Q comes from two aspects. One comes from V (f), which changes A, the other comes from Vyy which changes V. Because the gradient of V is stopped when estimating Q, the latter is captured by "minus old baseline, add Theorem E.1
new baseline", which - E5 [Q] + (VV)
Proof. Define
F(Q) (Q. (F(Q), (7, (7/
By Lemma E.3 9 (<) (Q) converges to some A" as n + co. This process will not influence the estimation of V as the gradient of V is stopped when estimating Q. According to the proof, A" does not depend on V. By Lemma E.4 (n()) (V) converges some V as oc.
Hence, we have
as n +00
By definition,
" - (30)
19

---

Under review conference paper at ICLR 2023
we can regard ((Q)+
+ (V) as Q and regard (V) as V. then
(Q, (33(0+++1,)
- (V)( (V
= (V)
By induction.
2 (Q, v) = (g (2) + (V) (())
(A +V* V*) as + to0.
Same as (Espeholt et al., 2018).
min (pp(als), T(als))
(als) EsEA min (pu(b)s), π (b|s)}
is the policy s.t. the Bellman equation holds, which is
pplp((r + VV2+1 v.) =0, 0,
and V/ (Q", V*) = (*). So we have (A" + = v*).
Lemma E.3. Define À A E2-EE114 [A], Q A + sg(V), then operator
T(Q) def E, (0 ar) + Σ
k20
is a contraction mapping w.r.f. Q. Remark. Note that T (Q) is exactly equation D
Since Q = A + sg(V), the gradient of is stopped when estimating Q, updating Q will not change V, which is equivalent to updating A. Without loss of generality, we assume V is fixed as V in the proof.
Proof. Ā A - [A] shows E=(AA] 0, which guarantees that matter how update A, we always have E, [Q] = V*
Based on above observations, define
F(Q) def
+ [E+
It's obvious that we only need to prove F(Q) is a contraction mapping. For brevity, we denote
Q. = Q(s,
at). A, = A(st, V (st).
Noticing that òr.0 = 1, let & represent filtration, we can rewrite F as
F(Q)= =
[ [A +
* *
A20
(16)
-
7 b -
Σ k+z (1414+
k20
20

---

Under review conference paper at ICLR 2023
where
4 Ey P 1C
(17)
By definition of Q.
B 2+ ++1]] BB #4h]
we can rewrite equation
17 as
4 = Ep
T++k+1
(18)
Mu+k+1
For any Q1 = A1 + (V),, ) Q2 A + sg( (V( since
E Lpr T++++1 (c+ 20.
M4+k+1
by equation
T6 equation
18
we have
- Q2ll:
-
where
C = Ep
G141:44-1 -1
Te+k+1 C++xPrA+1)
1x++++
k20
Ep 1+
*+1 c141+++-11 (P cc+RP(
k20
=1 (1 E
k20
< - (1 - <1
Hence, (Q) a contraction mapping and converges to some fixed function, which we denote as A". So T(Q) also contraction mapping and converges A + V*
Lemma E.4. Define Q A + sg(V) with Ex [A] 0, then operator
(FVV) daf E (((t Σ
x -jsttk-1PPtt
DR
A:0
is a contraction mapping w.r.t. V. Remark. Note that (V) is exactly equation] D
Proof. Same as Lemma E.3 get
E, [[pr+ C++RPE+R+)) V4+++1 - A4+ [{t11]
so we have
A A E,, (P(44- - Cc+kP(+++)))) (V (Vi-4+1 V 11 The remaining proof is identical to (Espeholt et al. 2018)'s.
21

---

Under review conference paper at ICLR 2023
F HYPERPARAMETERS
Our python packages are shown in Table 7
Package
Version
ale-py
0.6.0.dev22200207
gym
0.19.0
tensorflow
1.15.2
opencv-python
4.1.2.30
openev-contrib-yyhhee
4.4.0.46
Table 7: Versions for python packages among all experiments.
All experiments follow the shared hyperparameters as Table 8 The specific hyperparameters for PPO, R2D2 and CASA+DR-Tracs are shown in Table Table I0 and Table [[] The only exceptions are V-loss scaling, Q-loss scaling and m-loss scaling, which may be zero depending on some specific ablation settings. We will state these three hyperparameters every time in all experiments.
Parameter
Value
Atari Version
NoFrameskip-v4
Atari Wrapper
gym, wwn.waappeer.taarr www.pprrs.aaiii.ppppoessssieeeee preprocessing
Image Size
84)
Grayscale
Yes
Num. Action Repeats
4
Num. Frame Stacks
4
Action Space
Full
End of Episode When Life Lost
No
Num. Environments
160
Random No-ops
Bur--in Stored Recurrent State
Yes
Bootstrap
Yes
Optimizer
Adam Weight Decay
Weight Decay Rate
0.01
Weight Decay Schedule
Anneal linearly to
Learning Rate
5e-4
Warmup Steps
4000
Learning Rate Schedule
Anneal linearly to
Adam W 81
0.9
Adam 82
0.98
Adam E
1e-6
Adam W Clip Norm
50.0
Learner Push Model Every n Steps
Actor Pull Model Every n Steps
Table 8: Configurations for shared hyperparameters among all experiments.
22

---

Under review conference paper at ICLR 2023
Parameter
Value
Num. States
50M
Sample Reuse
Reward Shape
clip(r; 0, 1)
Burn-in
Seq-length
40
Discount (?)
0.995
Batch size
Backhone
IMPALA,shallow without LSTM
PPO clip e
0.2
GAE A
0.8
Temperature (7)
0.1
Table 9: Hyperparameter confiaatton for PPO.
Parameter
Value
Num. States
50M
Sample Reuse
2
Target Shape
Q = MEEO n-1 + (D
Target Shape Function h
h(x) sign(z) (v ||| + 10-
Bootstrap Length
5
e-gredyy
qunifioommm... 1,8)
PER Sample Temperature a
0.9
PER Buffer Size
400000
Burninn
0
Seq-length
Discount (?)
0.997
Batch size
8
Backbone
IMPALAshhlllww without LSTM
Temperature (7)
0.1
Table 10: Hyperparameter configurations for R2D2.
Parameter
Value
Num. States
200M
Sample Reuse
2
Reward Shape
log(Ir) + 1.) (2 (22-1(220) I(rco))
Burn-in
40
Seq-length
80
Discount (?))
0.997
Batch size
Backbone
dMpep.,
LSTM Units
256
V-loss Scaling (a.)
1.0
Q-loss Scaling (og)
10.0
Tr-loss Scaling (ax))
10.0
Temperature (7)
1.0
Importance Sampling Clip
1.05
Importance Sampling Clip P
1.05
Table 11: Hyperparameter configurations for CASA + DR-Trace.
23

---

Under review conference paper at ICLR 2023
G EVALUATION OF CASA ON ATARI GAMES
Random scores and average human's scores are from (Badia et al. 2020). from (Toromanofi al, 2019). Rainbow's 's scores are from (Hessel et al. 2017). IMPALA's scores are from
Human World Records (HWR) are
(Espeholt et al. 2018). LASER's scores are from (Schmitt al. 2020 sweep at 200M.
Games
RND
HUMAN
RAINBOW
HNS(E)
IMPALA
HNS(
LASER
HNS(/))
CASA
HNS(
Scale
200M
200M
200M
200M
allen
227.8
7127.8
9491.7
134.26
15962.
228.03
35565.9
$12.15
261.37
375.50
amidaar
106.
32.34
assault
222.4
7412
1/198.55
2689.76
19148.47
3612.43
21560.4
4106.62
16228
2080.37
asterix
210
8503.3
4282100
5160.67
300732
3623.67
240090
2892.46
213580
2572. 80
asteroids
213025
atlantis
12850
290228.
826660
5030.32
849967
5174.39
8412000
5120.19
3211600
19772.10
bank heist
14.2
753.1
181.66
1223.15
163.61
569.4
75.14
895.3
119.24
37187.5
64953.3
246.36
beam rider
363.9
16926.5
16890.2
99.54
32463.47
193.81
908816
$46.52
57456
344.70
herzerk
2630.4
1852.7
25579.5
bowling
160.7
5.01
59.92
26.76
48.3
18.31
162.4
101.24
boxting
0.1
12.1
99.6
$29.17
99.96
832.17
100
832.5
98.3
818.33
brcokout
centipede
2090.9
12017
8167.3
61.22
11049.75
90.26
292792
2928.63
102600
1012.57
chopper command
811
7387.8
16654
240
28255
417.29
761699
11569.27
616690
9364 42
crazy
defender
2874.5
18688.9
55103
330.27
185203
1152.93
334953
2112.50
4216/H
2647.75
demon atlack
152.1
1971
111185
6101,40
132826 98
7291.24
133530
7332.89
291590
16022.76
doublee dunk
-164
K2
-0.33
830.45
14
1765.91
enduro
860.5
2125.9
217.05
0
0.00
0
0.00
10019
1164.32
fishing derby
-9.77
-3.8
252.51
44.85
258.13
45.2
258.79
$3.24
273.99
freeway
29.6
114.86
0.00
0.00
3.46
11.69
fronhite
65.2
4334.7
9590.5
223,10
317.75
5:92
5083.5
117.54
1583
35.55
prpher
gravitar
173
338144
1419.3
39.21
359.5
5.87
1106.2
29.36
4311
130.19
hero
1027
30826.4
55867.4
184.10
33730.55
109.75
31628.7
162.69
24236
77.88
hockey
17.4
jameshond
29
302.8
19809
72.24
601.5
209.09
37999.8 8
13868.08
12468
4543 10
kanguroo
3035
14637.5
488,05
1632
52.97
14308
477.91
179 25
8147.4
9387.5
729.70
kung fu master
258.5
22736.3
52181
230.99
43375.5
191,82
607444
2701.26
12463301
$53.31
revenge
4753,
8.08
0.3
$2.35
ms pucman
6951.6
S380,4
76.35
7342.32
105.88
6565.5
94.19
7579
109.44
name this game
2292.3
8049
19136
188.57
21537.2
334,30
26219.5
415.64
32098
517.76
phusnix
pitfall
229.4
6463.7
3.43
-1.66
3.40
.0.6
3.42
11.8
3.16
pong
-20.7
11.6
20.9
117.85
20,98
118.07
21
118.13
20.39
116.10
private eye
0.16
ghen
163.9
13455.0
33W17.5
253.20
351211.12
2641.14
21449.6
160.15
27371
204.70
riverniid
1338.5
17118.0
22920.8
136.77
29608.05
179.15
40062.7
247.31
11182
62.38
read ronner
7845
62041
57121
578.00
251360
robotank
2.2
11.9
61.1
610.31
12.96
110.93
62.1
617.53
10.11
$1.95
scaquest
42164.77
15K9K
28.09
skiing
-17998
4336.9
12957.8
32.14
10180038
51,21
29968.1
-10.86
12230
34,23
solaris
1236.3
12326.7
3560.3
20.96
2365
10.18
2273.5
4.35
2319
9.76
space
star gunner
10250
127029
1318:22
200625
205597
321528
3347.21
337150
3510.18
surround
6,5
9,7
119.39
7.56
106.42
3.4
111.52
-10
0.00
tennis
12.2
time pilot
3568
$229.2
12926
563. 36
48481.5
2703.84
105316
6125.34
84341
4862 62
tutankham
167.6
27K.9
381
236.62
up n down
533,4
11693.2
125755
1122.08
332546.75
2975.08
345727
3093.19
416020
3723.06
venture
1187.5
5,5
0.45
0
0.00
0
0.00
0.00
vidco
$33436.5
$72898.27
5135
1686
wizard of wee
563.5
4756.5
17862.5
412.57
9157.5
204.96
29089.3
679.60
26008
606.83
revenge
3092.9
$4576.9
102557
193.19
84231. 14
157.60
166292.3
316.99
118730
234.61
MEAN HNS(%)
0.00
100.00
$73.97
957.34
1741.36
1911.08
HNS(%)
24

---

Under review conference paper at ICLR 2023
Games
RND
HWR
RAINBOW
SABER(S)
IMPALA
SABER(S)
LASER
SABEROS)
CASA
SABER/S)
Scale
200M
200M
200M
200M
amadar
5.8
104159
$131.2
4,92
1554,79
1.49
1829.
1.75
560
0.53
secault
222.4
8647
14198.5
165.90
19148.47
200.00
21560.4
200.00
16228
189.99
1000000
4282000
213580
atlantis
12850
10601544
826660
7.68
849967.5
7.90
841200
7.82
3211600
30.20
bank heist
14.2
#2058
1358
1.64
1223.15
1.47
509.4
0.08
895.3
1,477
64953.1
rider
berzerk
123.7
1057940
2545.6
0.25
1852.7
0.16
25579.5
2.41
1648
0.14
howling
23,1
300
30
2.49
59.30
13.30
48.3
9.10
162.4
50.31
hoxing
787.34
72.20
centiped:
2000.9
1301709
8167.3
0..47
11049.75
0.69
292792
22.37
102600
7.75
chopper command
811
999999
16654
1.59
28255
2.75
761099
76.15
616990
61.64
climber
10780.5
demon atack
152.1
1556345
111185
7.13
152826 98
8.53
133530
8.57
291590
18.73
double dunk
-186
21
-0.1
46.21
4.33
46.14
14
82.32
20.25
98.1
cuduro
105.46
fishing dedby
freeway
89.47
0.00
0.00
3,46
9.11
frosthite
65.2
454830
159015
2,09
317,75
0,06
5083.5
1.10
1583
10.55
gopher
66782.3
grwitar
359.5
0.57
bero
1027
1000000
55887.4
5.49
35730.55
3.27
31628.7
3.06
24236
2.82
ice hockey
-11.2
1.1
26.06
3,48
31,10
17.4
60.59
1.56
27.03
jumeshond
kangaroe
krall
1598
104100
8741.5
6.97
8147.4
6.39
9387.5
7.60
64347
61.22
kung fu master
258.1
10000000
52181
5.19
43376.5
4.31
667441
661.73
124630
12.44
revenge
ma pieman
240090
7342.32
6563.5
2.16
7579
2.51
same this game
2202.3
25230
19136
47.30
21537.22
83.94
26219..5
104
32098
131,00
phuenix
761.5
4014640
L08529
2.69
210996 45
5.24
519304
12.92
496590
12.40
pong
99.76
20.98
private
24.9
101800
4234
4.14
98.5
0.07
96.3
0.07
134.1
0.11
aphnt
163.9
2400000
33817.5
1,40
351200
14.63
21449.6
0.80
27571
1.13
diveraid
10000000
ned runner
2038100
62041
57121
2.80
45289
2.22
251360
12.33
rebotonk
76
61.4
80.22
12.96
14.98
62.1
81.17
10:44
11.17
68.4
999999
15896.9
1,58
1753.2
0.17
2890.3
0.28
11862
1.18
skinng
-17088
-2730
2365
2319
space invaders
148
621535
18769
3.00
43595.78
6.99
$1007.4
$.19
3031
0,66
664
77400
127029
164.67
200625
200,00
301528
200.00
337150
200,00
tennis
-2388
21
53.13
0.55
54.35
12.2
80.36
21,05
6.14
time pilot
3568
65300
12926
15.16
48481.5
72.76
165316
164.82
84341
130.84
tutankham
292.11
278.9
4.98
6.88
up
venture
0.00
video pinball
89218328
533936.5
0.60
$72898.27
0.64
511835
0.57
297920
0.33
wizai of wor
563.5
395300
17862.5
4.38
9157.5
2.18
29069.3
7.22
26008
6:65
yon revenge
15000105
102557
84231.14
118730
zasxon
32.5
83700
22209.5
26.51
32935.5
29.33
41118
69.11
46070.8
$5.03
MEAN SABER/S)
000
100000
28.39
29.45
36.78
36.10
MEDIAN SABER(S)
25

---

Under review conference paper at ICLR 2023
3
(1.) C000C4
(2) Amidar
(3-)
(4.) Asterix
(3.) Asteoolds
(6.) Atlantis
(T.) BankHeist
(8.) BattleZone
(9.) BesunRider
(10.) Berzerk
(11.) Bowling
(12.) Boxing
(13.) Breakout
[14.) Centipede
(15.) mond
ChopperCom-
(16.) Crazy Clibber
(17.) Defender
(15.) DemonAttack
(18.) DoubleDunk
(20.) Enduro
(21.) FishingDerly
(22.) Freeway
(23.) Frostbite
(24.) Gapher
(25.) Gravitor
(26.) Hero
(27.) TosHorkny
(28.) Jameshond
(29.) Kangaroo
(30.) Krall
(31.) KungFuMantee
(32.) MonlesamraR-- venge
(33.) MsPacman
(31.) Name ThisGame
(35.) Phoeixx
(36.) Pitfall
(37.) Pong
(38.) PrivastEEye
(39.) Qbeet
(10.) Riverraid
(11.) RosdRunner
(42.) Robstank
(43.) Seaquest
(44.) Skiing
(45.) Solaris
[46.) Spaosenvdders
(47.) StarGunner
[48.) Surround
(19.) Teamis
(50.) TimePilol
(51.) Twannkham
(32.) UpNDowu
(33.) Veniuze
(51.) VideoPinball
344
(56.) WizardOrWor
(56.) YarsRevenge
(57.) Zaxxon
26


Reviews:
[{"review_id": "m9mpPrfMBO", "reviewer": "Reviewer_aLPu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This works studies the relation between policy evaluation and policy improvement in the context where the value-function(s) and the policy share the same parameterization, and suggests a way to consolidates the two type of updates. ", "summary_of_the_review": "The paper deals with an interesting question and some parts or aspects of it could be a valid contribution. However in its current form it has too many open gaps and issues that should be fixed before it is ready for publication.", "strength_and_weaknesses": "The basic ideas underlying the paper are interesting and could potentially be a valid contribution. However there are several key issues that make the claims either difficult to understand (in the good case) or wrong/inaccurate (in the worse case). The evaluation is also not satisfying, relying on complex architectures and environments (see more below) to try and support what are fundamentally abstract theoretical claims.\n\nI would recommend the authors to reconsider the evaluation experiments that are used to support the theoretical claims. Ultimately, even a much simpler tabular settings where everything is much more controlled can be a much better starting point for this kind of paper, even if not all claims or properties can be rigorously proven theoretically.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PhQwsuAt0K_", "reviewer": "Reviewer_hwhf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel approach to generalized policy iteration (GPI) exploiting the inherent connection between policy evaluation and policy improvement. It firstly shows that the policy evaluation and policy improvement steps are connected in the case of parameter sharing between the value function and the policy, it describes the benefit of this connection and proposes the method CASA to exploit this connection. CASA is tested on the Atari 200M benchmark where it achieves remarkable state-of-the-art results.", "summary_of_the_review": "This paper studies the important problem of generalized policy iteration in reinforcement learning, and the proposed method is intriguing and able to achieve remarkable results in the challenging Atari 200M benchmark. However, I'm not convinced about the theoretical correctness of the method. It seems to me that conflict between policy evaluation and policy iteration is avoided only in the case where the parameters of value functions and policy are exactly the same. This is never the case in practice. As also stated in the paper, the parameters are mostly shared, except for individual heads. It is not clear to me whether the fact that MOST parameters are shared, is considered sufficiently good to meet the assumption of CASA; in that case, it would be nice to have a better discussion about it. Unfortunately, the low quality of the writing makes everything slightly harder to grasp.\n\nIt would be really nice if the authors could solve these doubts in the rebuttal, and I'd gladly increase my score in case the answer will be convincing enough.\n\nPost-rebuttal feedback\n------------------------------\nI have read the revised paper and appreciate some improvements on the previous version. I am still doubtful about the approach, sharing similar concerns to those expressed by Reviewer aLPu, in particular\n> In that sense, this is really a value-based method as there is no \"independent\" learning of a parameterized policy.\n\nThus, I raise my score for the improved version but I still consider this work under the threshold of acceptance.", "strength_and_weaknesses": "Strenghts\n-----------\n- The paper studies the interesting problem of generalized policy iteration from the original point of view of bridging the gap between policy evaluation and policy improvement;\n- Empirical results on Atari are impressive.\n\nWeaknesses\n--------------\n- The quality of the writing is not satisfactory and makes the paper hardly readable and understandable;\n- The proposed method CASA seems to rely on the fact that ALL parameters should be shared between value functions and policy. However, in the paper it is stated that only \"most parameters\" are shared. Doesn't this critically go against the assumptions of CASA? Please clarify;\n- The figures are too small and hardly readable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J2_DdxvHJAZ", "reviewer": "Reviewer_QNNj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of solving the alleged inconsistencies between the policy evaluation and policy improvement steps of the “policy iteration” algorithm. The paper proposes a method called CASA as a solution which suggests some modifications on the gradient calculations of standard algorithms such as PPO.", "summary_of_the_review": "The studied problem of the paper is not stated in a technically rigorous and unambiguous and intuitively clearly motivated way. The solution attempt also does not appear to follow a consistent deductive process. The reported experiment results have weak relevance to the main hypothesis.", "strength_and_weaknesses": "*Strengths:*\n\nThe formulation of the beta variable is interesting.\n\n*Weaknesses:*\n  \n * I cannot make sense of the studied problem at all. What does it mean to have a gradient conflict between two operational steps (policy evaluation and policy improvement) which have complementary contributions to the solution?\n * The paper states its aim as to “eliminate the inconsistency between the policy evaluation and policy improvement steps”. However, I do not understand how the proposed CASA method solves this problem. It only sets up a tunable continuum between these two steps. But one would also get the same using the \\lambda-policy iteration formula, which is introductory textbook stuff.\n * I do not find the reported ablation study informative. The benefit of CASA can be more directly quantified if the paper compares PPO vs PPO+CASA and R2D2 vs R2D2+CASA directly in the Atari game use case. \n * Overall the experiments section is rather slim. The paper does not compare against some strong candidates such as SAC and MBPO.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7Mc8-BXRTgn", "reviewer": "Reviewer_o19T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for reducing gradient interference between different components (Q, V, and policy) in deep RL with discrete actions. By carefully placing stop-gradient (sg) operators in the objective, the authors show (both theoretically and empirically) that the gradients between the Q function and the policy can be made parallel, so that there is no interference. The method, CASA, can be applied to value-based or policy-based algorithms using the same computations, and attains strong performance on Atari.", "summary_of_the_review": "I think the paper is a useful contribution, as it tackles a relatively underexplored problem with a clever and effective solution.", "strength_and_weaknesses": "Strengths:\n* The paper addresses an issue that many ignore in deep RL – the relationship between the policy evaluation and policy improvement steps in policy iteration.\n* The clever usage of sg in the objective leads to the desired result without any approximation or extra computationally intensive steps.\n* CASA substantially improves the performance of the base algorithm (IMPALA) on the Atari benchmark.\n\nWeaknesses:\n* The method as presented only applies to discrete-action problems, owing to the softmax parameterization of the policy and the summation over actions when computing the advantage.\n* The paper shows plots for many different gradient angles, but it’s not clear which we should care the most about. It would be helpful to have a better understanding of how the angles between the gradients correlate with performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject