Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
Authors: Hiroki Furuta
Tadashi Kozuno
Tatsuya Matsushima
Yutaka Matsuo
Shixiang Gu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Control as Inference
Inference-based Reinforcement Learning
Off-Policy
Deep Reinforcement Learning

Abstract: Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-independent and sometimes under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements across algorithms difficult. In this work, we focus on a series of off-policy inference-based actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation or code details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations.

Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
Hiroki Furuta
Tadashi Kozuno University of Alberta
The University of Tokyo
furutaOweblab. ttraaabbuuu.. t.u-tokyo ac. jp
Tatsuya Matsushima
Yutaka Matsuo The University of Tokyo
Shixiang Shane Gu Google Research
The University of Tokyo
Abstract
Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are gggorithm-indepenenn and some- times under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources performance improve- ments across algorithms difficult. In this work, focus series of off- off-policy inference-best based actor-critic algorithms MPO, AWR, and SAC to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single
objective, where we can catego- rize each algorithm based either Expectation- pntecttttioo-Maiiiizzziiee Maximization (EM) direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation code details are co-adapted and co-evolved with algorithms,, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's s performances but also transfer noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations
Introduction
Deep reinforcement learning (RL) has achieved huge empirical successes in both continuous 36
and discrete [39 266 problem settings with on-policy [52) [54] and off off-policy [13] [2
23 algorithms.
Especially in the continuous control domain, interpreting RL as probabilistic inference
has
yielded many kinds of algorithms with strong empirical performances (34 50 51
29
16] 24
Recently, there has been series of off-policy algorithms derived from this perspective for learning policies with function approximations (2 22 48). Notably, Soft Actor Critic (SAC) 22 23 based
on maximum entropy objective and soft Q-function, significantly outperforms on-policy [52
54 and off-policy [36] [8 [33 methods. Maximum a posteriori Policy Optimisation (MPO)
Thee implementation is available httpp//ss//gtttttoooooooo/////ae based-rr
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

inspired by REPS [50], employs sgeudo-likeliben objective, and achieves high sample-efficieney and fast convergence compared the variety of policy gradient methods [66 54 Similar to MPO, Advantage Weighted Regression (AWR) 48), and its variant, Advantage Weighted Actor- Critic (AWAC) [2], also employ the pseudo-likelihood objective weighted by the exponential of advantages, and reports more stable performance than baselines both in online and offline [35 settings.
While these inference-based algorithms have similar derivations to each other, their empirical perfor- mances have large gaps among them when evaluated on the standard continuous control benchmarks, such as OpenAI Gym [6] or Deep.ind Control Suite (58). Critically, as each algorithm has unique low-level implementation or code design decisions such as value estimation techniques, action distribution for the policy, and network architectures aside from high-level algorithmic choices, it is difficult to exactly identify the causes of these performance gaps as algorithmic or implementational.
In this paper, we first derive MPO, AWR, and SAC from a single objective function, through either Expectation- iitraationn-Mamiizztiioo (EM) or KL minimization, mathematically clarifying the algorithmic connections among the recent state-of-the-art off-policy actor-critic algorithms. This unified deriva- tion allows us precisely identify implementation techniques and code details for each algorithm, which are residual design choices in each method that are generally transferable to other algorithms. To reveal the sources of the performance gaps. we experiment with surefully-selecter ablations of these identified implementation techniques and code details, such as tanh-squashed Gaussian policy, clipped double Q-learning [3], and network architectures. Specifically, we keep the high-level algorithmic designs while normalizing the implementation designs, enabling proper algorithmic com- parisons. Our empirical results successfully distinguish between highly co-adapted design choices and no-co-adapted e (2) We identified that clipped double Q-learning and tanh-squashed policies, the sources of SoTA performance of SAC, are highly co-adapted. specific to KL-minimizztion-baase method, SAC, and difficult to transfer and benefit in EM-based methods, MPO AWR. In contrast, we discover that ELU (8] and layer normalization 4. the sources of SoTA performance of MPO, are transferable choices from MPO that also significantly benefit SAC. We hope our work can inspire more future works to precisely decouple algorithmic innovations from implementation or code details, which allows exact sources of performance gains to be identified and algorithmic researches to better build on one another.
2 Related Work
Inference-based RL algorithms RL. as probabilistic inference has been studied in several prior
contexts [60 61 34 57 45). but many recently-propose algorithms (2) 22 48) are derived separately and their exact relationships are difficult to get out directly, due to mixing of algorithmic and implementational details, inconsistent implementation choices, and benchmark differences. Our work organizes them as a unified policy iteration method, to clarify their exact mathematical algorithmic connections and tease out subtle, but important, implementation differences. We center our analyses around MPO, AWR, and SAC, because they are representative
crrroomentt-spcciin
tunings,
algorithms that span both EM-based 49 50 43
56 42 and KL-
[59] [1] 12, 29
21
33
RL and achieve some of the most competitive performances on popular benchmarks [88)
REPS 50 an EM approach, inspired MPO, AWR, and our unified objective in Eq. [] while Soft
O-learning 21 practical extension of KI. control to continuous action space through Liu and Wang
38). directly led to the development of SAC.
Meta analyses of RL algorithms While many papers propose novel algorithms, some recent works focused on meta analyses of some of the popular algorithms, which attracted significant attention due to these algorithms' high-variance evaluation performances, reproducibility difficulty g 28 65) and
frequent code-level optimizations (25 63 10 3). Henderson et al. (25) empirically showed how these
RL algorithms have inconsistent results across different official implementations and high variances even across runs with the same hyper-parameters, and recommended a concrete action item for the community - use more random seeds. Tucker et al. [63 show that high performances of action- dependent baselines [99
67 were more directly due to different subtle implementation choices. Engstrom et al. 10 focus solely on PPO and TRPO, two on-policy algorithms, and discuss
17
37
20
We here regard as co-adaptation the indispensable implementation or code decisions that do not stem directly from the conceptual algorithmic development, but from empirical considerations.

---

Implementation
Method
Algorithm
By update
Be update
9 yestimate
МPО
EM
Analytic TR
SG TR
Q
Retrace(1)
Tpp (iua(s) Ma(s))
AWR
EM
Analytic
Mixture SG
A
TD(A)
Up (Nua(a), 2)
AWAC
EM
Analytic
Mixture SG
Q"
TD(O)
Ap N(ua(() Mi.
SAC
KL
SG
Fixed Unif.)
Que
TD(0) TD3
Rv Tanh(N"(paaal Eo()))
POWER
EM
Analytic
Analytic
log Q
TIXI)
3p [(po((s) Ma(s))
RWR
Analytic
SG
1 logr
7p ((())(s))>..)
REPS
EM
Analytic
Mi
A
TDXO)
4p Softmax
UREX
EM
Analytic
SG
Q"
TI(I)
3p Soltmax
V-MPO
EM
Analytic TR
SG TR
4"P
n-step TD
7p NN(o(()), Ea(s))
TRPO
KL
TR
A"r
TD(I)
Niuulss) En)
PPO
KL
SG + TR
Tyy
1"
GAE
(()o(()) Mn)
DDPG
(Fixed)
9"
TD(O)
41 Ma(e)
TD3
(Fixed)
Q
TDXO) TD33
B4 (o(3)
Table I: Taxonomy based on the components of inference-basee off off-policy algorithms: MPO 22 AWR 48
AWAC [42). SAC (22). and other algorithms (see [Appendix B). We follow the notation of Sec. 3 We characterize them with the algorithm family (EM KI.), how policies
updated (Ty and pp; SG stands for succastic-gradient--eee and stands trust-regionhhased)) the choice (()), how G is estimated, and the parameterization of the policy. While the advantage function just can he interpreted as Q-function with baseline subtraction, explicitly write A when the stale-value function is parameterized, not Q-function. ( Note that DDPG and TD3 not "inference-based"" but be classified KL control variants.)
how code-level optimizations, instead of the claimed algorithmic differences, actually led more to PPO's superior performances. Andrychowicz al. 3 describes low-level (e.g. hyper-parameter choice, and regularization) and high-level (e.g. policy loss) design choices in on-policy algorithms affects the performance of PPO by showing results of large-scale evaluations.
In contrast to those prior works that mainly focus on single family on-policy algorithms, PPO and TRPO, and evaluating their implementation details alone, our work focuses on two distinet families of off- pollcc algorithms, and more importantly, presents unifying mathematical connections among peapeenddnntl--ppposseeee
state-of-the-ee algorithms. Our experiments demonstrate how some implementation choices in Table and code details are co-evolved with algorithmic innovations and/or have non-trivial effects the performance of off- offlccl inference-base methods.
3 Preliminaries
We consider Markov Decision Process (MDP) defined by state space S, action space A, state transition probability function S A x x S reward function 8 A R, and discount factor y [0, 1). Let Rt denote a discounted return D ~ ""0-"(" a.). We assume the standard RL setting, where the agent chooses actions based on parametric policy my and seeks for parameters that maximize the expected return Ex, [R.). Value functions for a policy π are the expected return conditioned by state-action pair or a state, that is, (st a.) := Eg [R] sts a.). and V == E,, [R, st). They are called the iuute-action-valus function (Q-function), and state-value function, respectively. The ad advantage function (5) is defined as A a) == Q" (s, a) V**s)) (s). The Q-function for a policy π is the unique fixed point of the Bellman operator T defined by (=)(ss = r(s.a) + r (s')p(s' (s, (ss)) a') ds' da' for any function Q x A + R. We denote trajectory or successive state-action sequence as : (81, a1, 82, ......). We also define an unnormalized state distribution under the policy by d. = - (8: s/T).
+ [0, xo), initial state distribution Pl S - [0, oc),
Inference-based Methods For simplicity, consider finite-horizon MDP with time horizon T for the time being. As a result, a trajectory т becomes a finite length: T =
($1, 11-
As in previous works motivated by probabilistic inference graphical model of an MDP binary event variable O, E {0, 1). which represents whether the action time step optimal or not. To derive the RI. objective, we consider the marginal log- likeliolddood log Pr(O- = 1/ where up policy. Note that O means O, 1 for every time step. As is
34
we introduce to the standard
3

---

well known, we can decompose this using a variational distribution of a trajectory as follows:
log Pr (r = 11 = Ry log Pr(O = -1/7) - log g(T)
p(T) + log g(r)
p(r|0 = 1)
= - + (KL(((T)) p 1)),
where [(p 9) E, Pr(O- (g( p(r)) is the evidence lower bound (ELBO). Inference-basee methods aim to find the parametric policy which maximizes the ELBO J (p, g).
There are several algorithmic design choices for q and Pr(O 1(T). Since any q is valid, popular choice is the one that factorizes in the same way as p. that is,
p(T) p(s)) II p(s 1 18c, ar) (p)ll)))))))
((7) p(si) II p( (e)
where Tp is prior policy, and TTy is variational posterior policy. We may also choose which policy (Tp or TT to parameterize. As for Pr(O = 1[r), the most popular choice the following one
21 22
34 48
Pr(O = 1/T) & exp (T
where 17 0 is temperature, and G is a function over S x A, such immediate reward function r, Q-function Q* and advantage function A". While we employ exp(-) in the present paper, there
alternatives [46 47 55
66 for example, Siegel al. (55) and Wang al. [66] consider
indicator function f x IR
- 1 1 2 0). whereas Oh et al. 46 consider the rectified linear unit
f: x E R [a, 0}.
Incorporating these design choices, we can rewrite the ELBO J [(, q) in a more explicit form as;
T
[TTT, = Σ En g a.) - DxL. (+
the following section, we adopt this ELBO and consider its relaxation to the infinite-horizon setting. Then, starting from it, we derive MPO, AWR, and SAC.
4 A Unified View of Inference-based Off-Policy Actor-Critic Algorithms
In Sec. 3 we provided the explicit form of the ELBO I (Tpp Ny). However, in practice, it is difficult to maximize it as the expectation E, depends on TT. Furthermore, since we are interested in umite-horizon setting, we replace Eierry Eg with (s) Note that the latter expectation Ed, (s) is taken with the unnormalized state distribution d, under an arbitrary policy æ. This commonly
assumed in previous works [2] [2]. This relaxation leads to the following optimization:
max [TT, TT) s.t. f do( / mp(a(s) dads and / dz<(s) / Ty.ttg
To (als)) dads = 1, (I)
where [(pp: mq) (7)
DKL(TT Tp)] With this objective, we can regard recent popular SoTA off- off-policy algorithms, MPO 12], AWR [48) and SAC [22) variants of unified policy iteration method. The components are summarized in Table We first explain how these algorithms can be grouped into two categories of approaches for solving Eq. and then we elaborate additional implementation details each algorithm makes.
4.1 Unified Policy Iteration: Algorithmic Perspective
Eq. allows the following algorithmic choices: how or parameterize Ty, and nyz what optimizer to use for them; and if to optimize them jointly, individually while holding the other fixed. We show that the algorithms in Table T can be classified into two categories: Expectation Expeetation-Maximizaaiee Maximization control (EM control) and direct Kullback-Lein divergence minimization control (KL control).

---

4.1.1 Expectaton-Maaxiizztion (EM) Control
This category subsumes MPO (2) (similarly REPS (50). ) AWR 48, and AWAC [42] and we term it EM control. At high level, the algorithm non- pon-parametrically solves for the variational posterior my while holding the parametric prior Tp = Tæ, fixed (E-Step), and then optimize xp holding the new Ty fixed (M-Step). This can be viewed as either generic EM algorithm and as performing coordinate
ascent on Eq. We denote Up and Fq after iteration & of EM steps by (%) and xq (k) respectively.
(k) be close to the optimal In E-step at iteration k. we force the variational posterior policy my posterior policy, i.c., the maximizer of the ELBO J (Tgg (Ty) with respect to TT. EM control converts the hard-constraint optimization problem in Eq. tto solving the following Lagrangian,
I (T B) / d ds(s) / To(als))) g a) dads
- f dz(() / (als))) log
To (a(s))
(1- dz(()) / mo(als) duds): (2)
dads + B
Tz (als)
We analytically obtain the solution of Eq.
Ty (als) Z(s) _ 1a (als) exp (1) (n-'g((sa))
where (s)) is the partition function.
Tp) with respect to Tp. Considering the optimization with respect to xp in Eq. Tresults in forward KL minimization, which is often referred as a gpeudo-likelihood or policy projection objective,
In M-Step at iteration k we maximize the ELBO J [[m
log Te, (als) Z(s) exp (1 (1-'919 g(s ())
max E 6r (s)ns [log Fa, (a|s)] = max (s) (y
0p
of
where we may approximate Z(s) M E/ exppppppp(ppp as)) with a, ~ T oo (-1s) in practice.
4.1.2 Direct Kullback-Leibler (KL) Divergence Minimization Control
In contrast to the EM control in Sec. 4.1.1 we only optimize the variational posterior Tg while holding the prior x. fixed. In this scheme, F. is parameterized,
denote it Ta. This leads to
KL control [59
21).
Equivalent to E-step in Sec. 4.1.1 we force the variational posterior policy ne, to be close to the optimal posterior policy, i.e. the maximizer of J (Tp: ng) with respect to Tg. The difference is that instead of analytically solving J [TTp ng)) for Tg. we optimize [(Tp no,) with respect to 0g. which results in the following objective,
Ta, (a|s) Tpp (a(ss))
n ((s,a) log
max (s) (als) 0g
4.1.3 "Optimal" Policies of EM and KL Control
While we formulate EM and KL control in unified framework, we note that they have a fundamental difference in their definition of "optimal" policies. KL control fixes Tp and converges a regularized- optimal policy in exact case [1., 64). In contrast, EM control continues updating both тp and Tq. resulting in convergence to the standard optimal policy an exact case [5T]. We also note that EM control solves KI. control as sub-problem: for example, the first E-step exactly corresponds to KI. control problem, except for the policy parameterization.
4.2 Unified Policy Iteration: Implementation Details
In this section, we explain the missing pieces of MPO, AWR, and SAC in Sec. 4.1 Additionally, we also describe the details of other algorithms 30 49
56
from the EM and KI.
control perspective. See Appendix B for the details.
5

---

4.2.1 MPO
Algorithm MPO closely follows the EM control scheme explained in Sec. 4.1. wherein Tp =0, parametric, and Mg is non-parametric.
Implementation b Update] This corresponds to the E-step. MPO uses a trust-region (TR)
method. Concretely, it replaces the reverse KL. penalty (second term) in the Lagrangian (Eq. (2) with a constraint and analytically solves it for kg. As result, is also optimized during the training by minimizing the dual of Eq. 2 which resembles REPS [50]:
g(n) y/e + 7) log (s)(() 1; (ale) exp Q1'1Q"""~" (0,
ITp Update] This corresponds to the M-step. MPO uses a combination of Stochastic Gradi- ent (SG) ascent and trust-region method based
forward KL divergence similarly to TRPO [52. Concretely, it obtains 0f*) by maximizing the objective (Eq. 3 with respect to 0, subject to || Te, (+ss)] S E. MPO further decompose this KL divergence to two terms, assuming Gaussian policies; one term includes only the mean vector of Ta, (-|s), and the other includes only its covariance matrix. Abdolmaleki al. (2) justify this as log-prior of MAP estimation, which is assumed as a second-order approximation of KL divergence.
(s) (Dxt))))
[G and G Estimate] MPO uses the Q-function of TOp-D 1) as G in the k-th E-step. It originally uses the Retrace update 40 while its practical implementation 27 uses a single-step Bellman update.
[Te]] For parameterized policy, MPO uses a Gaussian distribution with sute-dependent mean vector and diagonal covariance matrix. The trust-region method in MPO's M-step heavily relies on this Gaussian assumption. Since a Gaussian distribution has an infinite support, MPO has penalty term in its policy loss function that forces the mean of the policy stay within the range of action space.
4.2.2 AWR and AWAC
Algorithm AWR slightly deviates from the EM control. In the M-step, AWR simply Z(s) 1.
Implementation [n, Update] This corresponds the E-step. AWR and AWAC analytically solves the Lagrangian (Eq. 2}. In contrast to MPO, they don't use trust-region method.
ITp Update] This corresponds to the M-step. At iteration k, AWR uses an average of all previous policies pp == x k-1 Ta()) instead of - (cf. Eq. 3). In practice, the average policy Tp. is replaced by samples of actions from replay buffer, which stores action samples of previous policies.
[9 and Estimate] AWR uses the advantage function u as 9 in the k-th E-step, and learns the state-value function of Tglk with TD(A) (53] Due to its choice of ãgt, this avoids importance corrections 40 In contrast, AWAC estimates the advantage via the Q-function with TD(0).
[To] For a parameterized policy, both AWR and AWAC use a Gaussian distribution with state- dependent mean vector and state-independent diagonal covariance matrix constant one for AWR). As in MPO, they uses penalty term to keep the mean of the policy within the range of action space.
4.2.3 SAC
Algorithm Contrary to MPO and AWR. SAC follows the KL control scheme explained in Sec. 4.1.2 wherein the variational posterior policy = Tø, is parameterized, and the prior policy up is fixed to the uniform distribution over the action space A. SAC uses & soft Q-fucction:
FBg (st, ar) r(s,, + yETT,
[v (5(+1)
50 (st) = y (st) + Emog soft
20 (11)
with H(TO, (-1s:)) being - EM,, [log ke, (ar [sc]-

---

SAC
AWR
AWAC
MPD
Hopper-v2
Walker2d-v2
HairCheetah-22
poward
1200
cumu sover1
6000
cumulative rowards
17526
9926
2200
2300-
5000
12530
4000
1oo0
5000
Icee
2000
9020
1000
2300
Training Stepsite6]
Training Stopsr1e6)
Training Stops(led)
Ant-v2
Humanoid-v2
Swimmee--2
140
rewards
7:00
5200
Lumulativer rewards
10000
120
5200
5066
2000
2000
16
L000
8
Training Stepstle6]
Training Stops(106)
Traning Stapat
Figure I: Benchmarking results on OpenAl Gym MuJoCo locomation environments. methods are run with 10 random seeds. SAC seems to perform consistently better.
Implementation [, and Tp Update] SAC performs stochastic gradient (SG) ascent updates of only Ty. where the objective function is shown in Eq. SAC has operation equivalent to M-step in MPÒ and AWR, since it keeps the prior policy xp to the uniform distribution.
Similar to the temperature tuning in MPO, Haarnoja et al. [3] consider the dual function of entropy- constrained Lagrangian and treat the temperature 77 as a Lagrange multiplier, which results in,
gn) -nf n, (s) (a|a) [log Te, (al))]
where H. is the target entropy to ensure that the entropy of the policy should be larger than it.
[G and G Estimate] SAC uses Q as G, and TD(O)-like algorithm based on this soft Q-function. In policy evaluation, clipped double Q-lermiig [33, which retains multiple Q-functions (typically two) and takes the minimum, is employed to suppress the overestimation of Q value.
[Te] The policy of SAC is squashed Gaussian distribution: SAC first samples random variable from Gaussian distribution (s), (s)) with state-dependent mean vector and diagonal covariance; then, it applies the tanh function to and obtain the action a E [-1, 1]1-1, (If a dimension
of the action space is not [-1,1], 1], appropriate scaling and shift by an affine function is applied after the squashing.) Tanh squashing prevents out-of-bounde action.
4.3 Empirical Comparison
To evaluate the empirical performance of these off-policy algorithms (MPO, AWR, AWAC, and SAC), we compare their performances on Open AI Gym MuJoCo environments, namely, Hopper, Walker2d, HalfCheetah, Ant, Humanoid, and Swimmer, following Haarnoja et al. 23 We reproduce all
algorithms based on pytorch RL library [4]. referring their original implementations 22
48 42 27]. Figure Ihoww that SAC outperforms others in four environments (Hopper, Walker2d. HalfCheetah, and Humanoid), while MPO in Ant and AWR in Swimmer achieves the best performance. Generally,
SAC seems to perform consistently better. We extensively evaluate MPO, AWR, and SAC on the 28 tasks on DeepMind Control Suite and 3 MuloCo manipulation tasks. See Appendix C) [ndnd [ for the details.
5 Evaluation on Implementational Choices
In Sec. 4.3 SAC shows notable results most environments, while we revealed that the derivation and formulation of those methods resemble each other. To specify the effect of each implementational
7

---

SAC (D)
(S)
AWAC (D)
ASAC
(D)
MPO (S)
Hopper-v2
3013 + 602
1601 + 733
2329 = 1020
2540 + 755
2352 = 959
2136 + 1047
Walker2d-v2
5820 4 411
1888 + 922
3307 4: 780
3662 -: 712
4471 4: 281
3972 + 849
heethhhetah--22
15254 751
15701 + 630
7396 + 677
7226 = 449
12028 :: 191
11769 = 321
Ant-v2
5332 + 1266
1163 + 1326
3659 + 523
3008 = 375
7179 + 190
6584 = 455
Humamoid-v2
8081 + 1149
768 + 215
5243 + 200
2738 = 982
6858 = 373
5709 # 1081
Swimmer-v2
114 = 21
143 + 3
35 + 8
38 7
69 + 29
70 + 40
Table Ablation of Clipped Double Q-Learning. (D) denotes algorithms with clipped double Qllearning, and (S) denotes without it. We test original SAC (D). AWAC (D), MPO (S) and some variants; SAC without clipped double Q (S), AWAC (S), and MPO with clipped double Q (D), SAC (S) beats SAC (D) in HalfCheetah and
Swimmer, while it fails in Hopper, Walker, Ant and Humanoid, which implics that SAC (S) obtains highly exploratory policy, since fails termination environments. The learning curves are shown in Appendix F
code detail on the performance, we experiment with extensive and careful one-by-one ablations on; (1) clipped double Q-learning, (2) action distribution for the policy, (3) activation and normalization, and (4) network size. The former two correspond to implementation details (the choice of g estimate and the parameterization the policy), and the latter two correspond to code details (see [Appendix G for further experiments on other implementation details, such as TTp update or the choice of G). We also conclude several recommendations for the practitioners (Table 6).
5.1 Clipped Double Q-Learning
We hypothesize that clipped double Q-learning has large effect on the notable performance of SAC, and could be transferable in EM control methods. To verify this, we test the effect of clipped double Q-learning. Instead of AWR, here we evaluate AWAC since it uses Q-function. Table 2 shows that single-Q SAC outperforms original one in HalfCheetah and Swimmer that do not have the termination of the episode, while struggles to learn in Hopper, Walker, Ant and Humanoid that have the episodic termination conditions. A hypothesis is that single- Q SAC obtains to larger overestimation bias, which can help in environments where explorations are safe, but hurt in environments where reckless explorations lead immediate terminations. In contrast, clipped double Q-learning does not help MPO or AWAC as significantly as SAC; most results do not change or lead to slight improvements over the originals. This suggests that clipped double Q-learning might be a co-dependent and indispensable choice to KL control methods.
exploratory policy due
Use clipped double Q-learning as a default choice, but you can omit in EM control methods non-terminal environments, such as HalfCheetah or Swimmer.
Recommendation
5.2 Action Distribution for the Policy
Another hypothesis is that the tanb-squashed policy (last column in Table 1 is an important and transferable design choice. We compare SAC without tanh transform (with MPO action penalty instead) to the original one, which results in drastic degradation (Table 3). This can be caused by the maximum entropy objective that encourages maximizing the covariance. These observations suggest that the high performance of SAC seems to highly depend on the implementational choice of the policy distribution. In contrast, MPO and AWR with tanh squashing don't 't enjoy such large performance gain, or achieve worse cumulative rewards. This also suggests that tanh-squashed policy might be highly co-adapted in SAC and less transferable to EM control methods. Note that for EM control, we must clip the actions keep them within the supports of distributions; E [-1 + e, e| We found that no clipping cause significant numerical instability. See AeeeAndndix H for the details.
Use the original distributions for each algorithm. If you use the tanh-squashed Gaussian in EM control methods, clip the action to be careful for the numerical instability.
Recommendation
5.3 Activation and Normalization
The implementation of MPO 27 has some code-level detailed choices; ELU activation S and
layer normalization 4. We hypothesize that these, sometimes unfamiliar, code details in practical implementation stabilize the learning process, and contribute to the performance of MPO most.
8

---

SAC(W))
(W/(W)))
AWR (w/)
AWR (w/o)
MPO (w/)
MPO (w/o)
Hopper-v2
3013 + 602
6 10
2709 + 905
3085 = 593
2149 849
2136 + 1047
Walker2d-v2
5820 + 411
00
3295 1: 335
4717 1: 678
3167 + 815
3972 1: 849
HaltCheetah-v2
15254 = 751
00
3653 + 652
5742 + 667
9523 = 312
11769 = 321
Ant-v2
3532 + 1266
- 00
445 + 106
1127 + 224
2880 = 306
6584 = 455
Humanoid-v2
8081 1149
108
821
2304 +: 1629
5573 +: 1020
6688 1: 192
5709 +: 1081
Swinmer--x
114 1. 21
28
1.111 11
121 1
128
110 42
70 1 40
Table 3: Ablation of Tanh transformation (" ('numerical error happens during training). We test SAC without tanh squashing, AWR with tanh, and MPO with tanh. SAC without tanh transform results in drastic degradation of the performance, which can caused by the maximum entropy objective that encourages the maximization of the covariance. In contrast, EM Control methods don't It enjoy the performance gain from the tanh-squashed policy, which seems a less transferable choice. The learning curves are shown in Appendix F
Hopper-v2
Walker2d-v2
HalfChoctah-v2
Ant-v2
Humanoid-v2
Swimmer-v2
SAC
3013 1 602
5820 411
15254 1 751
5532 1 1266
8081
1149
114 21
SAC-E
2337 903
5504 = 431
15350 594
6457 828
8196 892
146
SAC-L*
2368 179
5613 = 762
13074
2218
7349 = 176
8146 470
99 = 18
SAC-E
1926 1: 417
5751 + 400
12555
1250
7017 1: 132
7687 1: 1385
143 1
MPO
2136 = 1047
3972 = 849
11769
1 321
6584 1: 455
5709 - 1081
70 = 40
MPO-E
2700 + 879
3653 = 1145
11638 +: 664
5917 + 702
4870 = 1917
108 +: 28
MPO-L-
824 + 250
2413 + 1352
6064
+ 4596
2135 + 2988
5039 838
MPO-E L
843 + 168
1708 663
-1363
20965
807
2351
5566 787
-00
AWR
3085 593
4717 678
5742 667
1127 224
5573 1020
128
AWR-E
1793 = 1305
4418 = 319
5910 754
2288 + 715
6708 226
128 4
AWR-L
2525 = 1130
4900 = 671
5391 + 232
639 + 68
5962 + 376
129 + 2
AWR-E L*
3234 = 118
4906 = 304
6081 + 753
2283 + 927
6041 + 270
130 +
Table 4: Incorporating ELUUlayer normalization into SAC and AWR. E+ indicates adding, and /I. indicates removing ELU/layer normalization. Introducing layer normalization ELU into SAC improves the performances in Ant (beating MPO), Swimmer (beating AWR), HalfCheetah, and Humanoid. AWR also shows the improvement several tasks. MPO removing layer normalization largely drops its performance. Both code-level details seems transferable between EM and KL control methods.
MPO significantly outperforms SAC. To investigate this much deeper, we add them into SAC AWR and remove them from MPO, while maintaining the rest of the implementations. Table 4 shows that layer normalization can contribute to a significantly higher performance of SAC in Ant, and replacing ReLU with ELU also improves performance lot in Swimmer, where AWR is the best in Figure I In contrast, the performance of MPO drastically collapsed when we just removed ELU and layer normalization. This observation suggests that these code-level choices are not only indispensable for MPO, but transferable and beneficial to both KL and EM control methods. Additionally, we tested incorporating ELU and layer normalization SAC in 12 DM Control tasks where MPO outperformed SAC, and observed that they again often benefit SAC performances substantially. See See Appendix Cffor the details.
Especially in Ant (Sec. 4.3
Recommendation It is worth considering replace the activation function from ReLU to ELU and incorporate the layer normalization, to achieve the best performance in several tasks. Both of them are transferable between EM and KL control methods.
5.4 Network Size
While on-policy algorithms such PPO and TRPO can use the common network architecture as in prior works [10 (2), the architecture that works well in all the off-policy inference- based methods still not obvious, and the RL community does not have agreed upon default choice. To validate the dependency on the networks among the inference-basee number of hidden layers in the policy and value networks. We denote the network size of MPO ((256, 256, 256) for policy and (512, 512, 256) for value) as large (L), of SAC ((256, 256) for policy and value) as middle (M), and of AWR ((1 (128, 64) for policy and value) as small (S) (see Appendix A for the details). Table 5lillustrates that SAC with large network seems to have better performance. However, as shown in Appendix F Figure 8), the learning curve sometimes becomes unstable. While the certain degree of robustness to the network size is observed in SAC, EM control methods, MPO
methods, we exchange the size and

---

Hopper-v2
Walker2d-v2
HaliCheetah-v2
Ant-v2
Humanoid-v2
Swimmer-v2
SAC (L)
2486 + 746
3188 + 2115
16528 + 183
7495 = 405
8255 = 578
118 + 26
SAC (M)
3013 + 602
5820 +: 411
15254 = 751
5532 + 1266
8U81 = 1149
114 +: 21
SAC (8)
3456 81
4939 = 284
12241 = 400
3290 + 691
7724 = 497
59 + 11
MPO (L)
2136 = 1047
3972 = 849
11769 = 321
6584 + 455
5709 = 1081
70 + 40
MPO (M)
661 = 79
1965 + 1426
-00
5192 + 538
6015 + 771
81 = 28
MPO (S)
430 + 99
2035 + 990
5003 + 1567
3587 + 957
4745 = 1428
59 + 28
AWR (L)
3221 + 193
4688 + 648
4360 + 542
35 + 43
665 + 54
133 + 3
AWR (M)
2816 910
4826 + 547
5538 + 720
413 + 117
3849
1647
133
2
AWR (S)
3085
1 593
4717
1 678
5742 1 667
1127 E 224
5573
1020
128
E
Table 5: The performance of each algorithm with different network size. (S) stands for the small network size
from AWR, (M) for the middle network size from SAC, and (L) for the large network size from MPO. Generally, SAC, a KT. control methad seems more robust to the network size than EM control methods; MPO and AWR.
and AWR, seem fragile and more dependent on the specific network size. This trend is remarkable in MPO. AWR (M) or (L) also struggles to learn in high-dimensional state tasks, such as Ant (111 dim), or Humanoid (376 dim). The results also imply that, in contrast to on on-policy methods [3. the network size in the off-policy siterence-bssed methods seems less transferable choice.
Recommendation For SAC, use medium size network. Large size will also work, but the learning curve might be unstable. For MPO, strongly recommend to stick to large size, because it is very sensitive to the network size. For AWR, using small size is a better choice, especially in high-dimensional state tasks, such Ant, Humanoid.
6 Conclusion
In this work, we present a taxonomy of inference-basee algorithms, and successfully identify seoorthh--specing as well as algorithm -nddependeett implementation details
MPO
Clipped Double 0
5.1
4
Tanh Gaussian 522
4
that cause substantial performance improvements. We first reformulated recent inference-based off-policy algorithms -ucch as MPO, AWR and SAC into a unified mathemat- ical objective and exhaustively clarified the algorithmic and implementational differences. Through precise ab- lation studies, empirically show that implementation choices like tanh-squashed distribution and clipped double Q-learning are highly co-adapted to KL control methods (e.g. SAC), and difficult to benefit in EM control methods (e.g. MPO or AWR). As an example, the network archi- tectures of inference-based off- off-policy algorithms, especially EM controls, seem more co-dependent than on-policy methods like PPO TRPO, which therefore have significant impacts the overall algorithm performances and need to be carefully tuned per algorithm. Such dependence of each algorithmic innovation specific hand-tuned implementation details makes accurate performance gain attributions and cumulative build-up of research insights difficult. In contrast, we also find that some code-level implementation details, such as ELU and layer normalization, are not only indispensable choice to MPO, but also transferable and beneficial to SAC substantially. We hope our work can encourage more works that study precisely the impacts of algorithmic properties and empirical design choices, not only for one type of algorithms, but also across a broader spectrum of deep RL algorithms.
ELU & Norm
3.3
0
Large Network
5.4
Medium Notwors
Small Network
Table Intuitive summary of the ablations. O stands for indispensable choice, O stands for recommended choice, A stands for not much different worse choice than expected, and stands for un-recommended choice. See Recommendation for the details.
Acknowledgements
We thank Yusuke Iwasawa, Masahiro Suzuki, Mare G. Bellemare, Ofir Nachum, and Sergey Levine for many fruitful discussions and comments. This work has been supported by the Mohammed bin Salman Center for Future Science and Technology for Saudi-Japan Vision 2030 at The University of Tokyo (MbSC2030).
10

---

References
[1] Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arXiv:1812.02256. 2018.
[2] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018.
[3] Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthicu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on polpol deep actor-critic methods? large-scale study. In International Conference on Learning Representations. 2021.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06453 2016.
[5] Leemon C Baird III. Advantage updating. Technical report, WRIGHT LAB WRIGHT-PATTERSON AFB OH, 1993.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1r65506600.01540, 2016.
Kamil Ciosek and Shimon Whiteson. Expected policy gradients for reinforcement learning. Journal of Machine Learning Research, 2020.
[8] Amó Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). In International Conference on Learning Representations, 2016.
[9] Yan Duan. Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, 2016.
[10] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2019.
[11] Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning, In Advances in Neural Information Processing Systems, 2019.
[12] Roy Fox, Ari Pakman, and Naftali Tishhy. Taming the noise in reinforcement learning via soft updates. In Conference on Uncertainty in Artificial Intelligence, 2016.
[13] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning. 2018
[14] Yasuhiro Fujita, Prabhat Nagarajan. Toshiki Kataoka, and Takahiro Ishikawa. Chainerr:: A deep reinforce- ment learning library. Journal of Machine Learning Research, 2021.
[15] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision processes. In International Conference Machine Learning, 2019.
[16] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, 2020.
[17] Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. arXiv preprint 17 2017.
[18] Shixiang Gu, Timothy Lillicrap. Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, 2016.
[19] Shixiang Gu Timothy Lillicrap. Zoubin Ghahramani, Richard Turner, and Sergey Levine. Q-Prop: Sample-efficient policy gradient with an off-policy critic. In International Conference on Learning Representations, 2017.
[20] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on on-policy and off off-policy gradient estimation for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
11

---

[21] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, 2017.
[22] Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
[23] Tuomas Haarooja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jic Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint
219005 2018.
[24] Danijar Hafner, Pedro A Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. Action and perception as divergence minimization, arXiv preprint arXiv:2009.0179) 2020.
[25] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2017.
[26] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
[27] Mau Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth- Warth-Maron, Feryal Behbahani, Tamara Nor- man, Abbas Abdolmaleki, Albin Cassirer. Fan Yang. Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine. Andrew Cowie, Ziyu Wang. Bilal Piot, and Nando de Freitas. Acme: A rescarch framework for distributed reinforcement learning. arXiv preprint arXiv:2006.c 2000000999,, 2020.
[28] Riashat Islam, Peter Henderson, Maziar Gomrokchi. and Doina Precup. Reproducibility of benchmarked deep reinforvement learning tasks for continuous control. arXiv preprint 1.XXiv1770..41333 2017.
[29] Natasha Jaques. Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobate Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Muchine Learning, 2017.
[30] Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Advances in neural information processing systems, 2008.
[31] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-polics q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
[32] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov, Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, 2020.
[33] Alex X. Lee, Anusha Nagabandi. Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. In Advances Neural Information Processing Systems, 2020.
[34] Sergey Levine. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and review, arXiv preprint 1XXx:1805500009 2018.
[35] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint wwXXiv20055016422 2020.
[36] Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[37] Hao Liu, Yihao Feng Yi Mao, Dengyong 7hou, Jian Peng, and Qiang Liu. Action-depedent control variates for policy optimization via stein's 's identity. arXiv preprint arXiv: ///0/, 2017.
[38] Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. arxiv preprint 1608.04000444 2016.
[39] Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
12

---

[40] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems. 2016.
[41] Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. International Conference on Learning Representations, 2017.
[42] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelcrating online reinforcement learning with offiine datascts, arXiv preprint arXiv:2006.099: 2020.
[43] Gerhard Neumann et al. Variational inference for policy search in changing situations. In International Conference on Machine Learning, 2011.
[44] Mohammad Norouzi, Sany Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neurul Information Processing Systems, 2016.
[45] Brendan O'Donoguue, Ian Osband, and Catalin Ionescu. Making sense of reinforcement learning and prohahilistic inference. In International Conference on Learning Representations, 2020.
[46] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitaioon learning. In International Conference on Machine Learning, 2018.
[47] Masashi Okada and Tadahiro Taniguchi. Variational inference mpe for bayesian model- based reinforcement learning. In Conference on Robot Learning, 2019.
[48] Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-Weigtted Regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[49] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, 2007.
[50] Jan Peters. Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In AAAI Conference on Artificial Intelligence, 2010.
[51] Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. International Joint Conference on Artificial Intelligence. 2012.
[52] John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Picter Abbeel. Trust region policy optimization. In International Conference on Machine Learning, 2015.
[53] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
[54] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1007.063477 2017.
[55] Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin A. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[56] H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg. Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Riedmiller, and Matthew M. Botvinick. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2020.
[57] Yunhao Tang and Alp Kucukelbir, Hindsight expectation maximization for goal-conditioned reinforcement lcarning. arXiv preprint arXiv:2006.075: 2020.
[58] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abhas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint prepritt arXiv: 1801.00690, 2018
[59] Emanuel Todorov. Linearly-solvable markov decision problems. In Advances in Neural Information Processing Systems, 2006.
[60] Emanuel Todorov. General duality between optimal control and estimation. In IEEE Conference on Decision and Control, 2008
13

---

[61] Mare Toussaint. Robot trajectory optimization using approximate inference. In International Conference Machine Learning, 2009.
[62] Mare Toussaint and Amos Storkey, Probabilistic inference for solving discrete and continuous state markov decision processes. In International conference on Machine learning, 2006.
[63] George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of Machine Learning, 2018.
basclines in reinforcement learning. In International Conference on
[64] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin. Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in rl. In Advances in Neural Information Processing Systems, 2021.
[65] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint or 1907.02077 2019.
[66] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bohak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. arXiv preprint arXiv:2006. 15134, 2020.
[67] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. arXiv preprint 1833:1800000244 2018.
[68] Yifan Wu, George Tucker, and Ofir Nachum. Behavior Regularized Offline Reinforcement Leaming. arXiv preprint /r1iv:991..1136/ 2019.
Checklist
1. For all authors...
(a) Do main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We added the failure cases ablations that end up the insufficient coverage and the unclear insights in InAApeedixx H]
(c) Did you discuss any potential negative societal impacts your work? [Yes] Actually, we conducted exhaustive evaluations through the enormous experiments, which might lead to force the future research to spend much computing resources. We hope empirical observations and recommendations help the practitioners to explore the explosive configuration space.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or as a URL)? (Yes] We open-source the codebase at https://github cww/foo///////o hwtm//rtt///aie inf erence erence-basdd- based-r)
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Appendix A
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] Our experimental results averaged among 10 random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See See Appendix
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) your work existing assets, did you cite the creators? [Yes] We cited the authors of the codebase in Sec. 4.
(b) Did you mention the license of the assets?
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'r us- ing/eurating? [N/A]
14

---

(e) Did discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "yt8jMT4R5St", "reviewer": "Reviewer_BvPF", "summary": "This paper fits in the category of survey/review paper. While this kind of paper is not common at Neurips, it doesn't mean that it should be disregarded. It does a good job at trying to unify a popular body of work on policy gradient methods under the umbrella of \"control as inference\". The paper is well-structured: it defines a taxonomy which differentiates between \"EM-based\" methods and those based on \"direct KL minimization\". The paper focuses on four algorithms: Maximum a posteriori Policy Optimisation (Abdolmaleki et al. 2018), Advantage-Weighted Regression (Peng et al. 2019), Soft Actor-Critic (Haarnoja et al. 2017) and Relative Entropy Policy Search (Peters et al. 2010). The  paper starts by explaining how the RL problem can be cast as a probabilistic inference problem and tries to explain how an evidence lower bound can obtained. It then goes into the specifics of each algorithm and how each either parametrizes the prior or the posterior and whether  it uses hard constraints (with closed-form expressions) or soft/regularized re-formulations. \n\nThe empirical section then compares the raw performance three of those algorithms (SAC, MPO and AWR) on six Mujoco environments. \n\nRegarding the \"co-adaptation\" expression in the title, I wouldn't say it defines the paper. The \"co-adaptation\" findings really pertain to the impact of specific implementation details on the final performance in those six Mujoco environments. More specifically, the authors study the impact of double-q learning + clipping, ELU activations vs ReLu, network size and layer normalization. The empirical findings are nicely summarized through tables. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the structure is great and I like the ablatation study. Furthermore, I have learned new things via the EM vs direct minimization taxonomy. That being said, I think that the writing could be improved. I find the technical presentation is not solid enough. Not that the authors should go with a measure-theoretic presentation, but the paper would gain clarity if the notation is improved. Don't get me wrong, I'm not complaining about the choice of symbol, but rather the symbol-to-meaning mapping which I find deficient. For the kind of paper that the authors are aiming for, the background section needs to be near-perfect and crystal clear. \n\nRegarding the main contribution, the co-adaptation claim, I think that it should be de-emphasized. Just in terms of space in the text, pages 1-6 are dedicated to the presentation of SAC, MPO and AWR under a unified narrative. The co-adapation findings are only developped in pages 7-9 and the overall takeaway seems to be that pretty much everything should stay the same for MPO and AWR but SAC is a bit more tolerant and can do better with a few tweaks taken from the other methods. \n\n# Specific Comments\n\nThe expression \"co-adaptation\" is interesting, and I can see how it expresses the idea that you are trying to convey. That being said, I wonder if there would be another way to describe the issue without introducing new terminology (in this case borrowing from biology). For example, from the very begining in the abstract, the sentence:\n\n\"tanh Gaussian policy and network sizes are highly adapted to algorithmic types \" (line 17) is a bit obscure to the non-initated \n\nOn line 53, \"are highly co-adapted to KL-minimization-based\", you could perhaps say \"are highly specific\" instead.\n\nIf you insists on using \"co-adaptation\", you should perhaps state explicitely what it means in the first sentence or so. Something like \"we define co-adaptation as the implementation decisions that do not stem directly from the conceptual algorithmic development, but from empirical considerations.\"\n\nOn line 27, \"there have been a series\" I think that you should write \"has\" \nOn line 29, \"on maximum entropy\", \"on a\"\n\nOn line 88, \"Our experiments prove\", I suggest to keep \"prove\" on the math side. Instead: \"demonstrate\"/\"highlight\" \n\nCaption of figure 1, \"when the state-value function is parameterized instead of Q-function\" (\"instead of the\"). This sentence fragment is unclear.\n\nOn line 93, \"transition probability distribution\" => \"transition probability function\", also \"initial state distribution\" not formally (Puterman) part of the MDP definition\n\nLine 95: The use of \"u\" as a time index is unusual. How about \"k\" instead? \n\nLine 95: \"standard RL setting\", I wouldn't say that this is necessary \"standard\". How about \"we consider a policy gradient setting\"?\n\nLine 95: \" for its parameter that maximizes\" => \"for parameters that maximize\"\n\nLine 99, \" advantage function\" cite Baird 1993.\n\nLine 100, \"is known to be a\" => \"is the\"\n\nLine 105, \"For simplicity, we consider a finite-horizon MDP\" I know that it is common in RL to make this assumption, but it's very important to keep in mind that the properties of finite-horizon MDPs and infinite-horizon discounted ones are not the same. The beauty of infinite horizon discounted is that it allows us to restrict our attention to stationary policies when searching for optimal policies. But it doesn't have to be the case in the finite horizon setting! In fact, the contraction/fixed-point based perspective is no longer appropriate. Instead, you would want to talk about backward induction and so on... \n\nIn the ELBO derivation around line 112, you have $\\tau$ on the rhs but not the lhs. This is because $\\mathcal{O}$ is function of $\\tau$. Perhaps writing $\\mathcal{O}(\\tau)$ would make things clearer. Also, writing $\\text{Pr}(\\mathcal{O}|\\pi_p)$ suggests that $\\pi_p$ is a random variable, and that there is conditional distribution defined over those quantities. If so, I suggest being a bit clearer about that.\n\nLine 116, \"a popular choice is the one that factorizes\", making that assumption of an MDP earlier, I don't think that you have much of a choice here in your factorization.\n\nLine 123-124: are these other choices valid? As in: integrating/summing up to 1?\n\nLine 125: You jump from an expectation over trajectories, to one where there exists a joint over state-action pairs. This is not a trivial step.\n\nLine 132: \"unnormalized state distribution\", if it's unnormalized, it can't be a distribution and writing $\\mathbb{E}_{d_\\pi}$ doesn't quite make sense. think what you mean is something like $d_\\pi = p_1^\\top(I - \\gamma P_\\pi)^{-1}$. You then write an equality-constrained program where your equality constraints is the actual normalization. This is confusing.\n\nLine 152: \"converts the hard-constraint optimization\". Just to be clear, the Lagrangian step is not about \"converting\" an constrained optimization problem into an unconstrained one (the constrained solution need not coincide with an unconstrained min or maximum of the Lagrangian). \n\nLine 170: \"an exact case\". You should be clearer about what you mean by \"exact\" (lack of function approximation). \n\nLine 173: \"control problem, except the policy parameterization.\" The sentence sounds incomplete. Please expand\n\nLine 182: \"reverse KL penalty (second term) in the Lagrangian (Eq. 2) with a reverse KL constraint\". This sentence is confusing, but I think what you mean is that it's actually solving a different constraint optimization problem. One where you not only require to have distributions (integrates to 1) but also that the KL is equal (or perhaps) less than some designated value. You should then write the equivalent constrained program as you did in equation 1. It would be clearer than talking about changes to the Lagrangian. \n\nLine 194: \"For a paramterized policy\" => \"parameterized\" (you should use Grammarly to catch those prior to submitting) \n\nLine 204: \"replaced by samples of actions from a replay buffer\" I don't quite understand how this works out to be similar.\n\nLine 219: \"the dual function of entropy constrained Lagrangian \" I don't think that this sentence makes sense. I think all you want to say is that they treat $\\eta$ as a Lagrange multiplier and they they then do gradient-ascent descent on the Lagrangian. \n\nSection 5.1: \"Distribution of the Policy\". This is about the policy parametrization. Not about some \"distribution over policies\". I think that you should avoid this expression", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't want to be that reviewer who says: \"more experiments\". Of course we all want more experiments... But I need to be\n\nI need to say that I'm becoming increasingly concerned about the over-reliance on the Mujoco domains for benchmarking policy gradient (Berkeley-style) methods in RL. As a community, we're using these environments so often that it's becoming increasingly difficult to assess where we stand in terms of \"real-world\" impact.  Are we overfitting? \n\nBecause this is essentially a survey paper, I would have like to see a more extensive evaluation. \n\nHow about taking more time and writing a much longer a journal paper? Like those great survey papers from the people at Darmstadt. How about the benchmark track at Neurips? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RHZ2J1NlZzQ", "reviewer": "Reviewer_CYmr", "summary": "Provides a unifying mathematical framework for off-policy inference based algorithms such as MPO, AWR (that use pseudo-likelihood objective) and SAC, classifying them into EM based or KL based algorithms (uses the control as inference framework to do so). This allows decoupling the effect of high-level algorithmic choices from low-level code optimizations, to identify where the benefit from these algorithms really comes from, and further analyzing which design choices are transferrable across different algorithm families. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The control as inference framework has been used by the authors of MPO and SAC in their respective papers, and isn't a new way of looking at these algorithms. More specifically, the EM class formulation closely follows a similar characterization in the MPO paper. Nevertheless presenting these together as part of the same framework seems new (where KL control is shown to be almost a sub-problem for EM control). The paper cites a lot of the previous work dealing with meta-analysis of RL algorithms which point out that performance is often dependent on the specific code implementation used. \n\n**Quality**: The mathematical formulation seems technically sound and well supported (and are based on prior work in the field). The experimental evaluation includes 5 openAI gym mujoco environments and is quite exhaustive as it includes ablations over a number of different design choices, such as network size, activations and normalization etc, in order to study which low-level code optimizations are important.\n\n**Clarity**:  The paper is well written and easy to follow and understand. The authors first present their unified control as inference framework, then clearly show how MPO, AWR, SAC fall into it. The experimental evaluation is well organized for analysis into each separate ablation axis. \n\n**Significance**: Irreproducibility is a well known issue for RL, especially the dependence of algorithm performance on the specific implementation used. This work takes a step towards addressing this issue. The exhaustive experimental evaluation might be useful to other researchers, where the authors analyze what aspects of the different algorithms are critical. They find that certain choices that are important for SAC (clipped double Q and squashed tanh), do not improve performance of other algorithms. Further they also find useful design choices in MPO (ELU and LayerNorm) which improve performance of SAC. That said, some of these observations might be specific to the gym environments tested, and may not transfer.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation are novelty of the formulation (which has been used in prior work, albeit separately for the different algorithms), and the uncertainty over whether the low level design choice recommendations made here (with regard to network size, normalization etc) will apply to other agents/robots. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "NYspl1Sk5v", "reviewer": "Reviewer_34Q6", "summary": "Comparing the performance of different RL algorithms can sometimes be difficult because algorithmic contributions can be confounded with differing implementational choices between the agents. This paper attempts to examine the relative contributions of algorithmic vs implementational choices in a series of inference-based, off-policy actor-critic algorithms -- MPO, AWR, and SAC. The authors' starting point is a unified view of these algorithms from the perspective of control-as-inference. In this framework, policies as cast as posterior probabilities that a given action in a given state will be optimal. The authors derive an ELBO for this posterior that, when optimised, yields an approximate optimal policy. The authors then describe how optimization can be approached in two ways: an EM approach (where both a prior and posterior policy are optimized) and a KL approach (where only the posterior policy is optimized). MPO and AWR belong to the EM family of algorithms, while SAC belongs to the KL family. The authors further detail how these algorithms fit into their unified framework.\n\nIn the second half of the paper, the authors focus on the implementational differences between these algorithms. They focus specifically on four choices: clipped double q-learning, tanh-squashing in the policies, the choice of activation function and layer norm, and the network size. Specifically, through a series of ablation studies, the authors examine how critical some of these choices are to specific algorithms and which of these can be transferred to other algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this was an interesting paper. For me, one of the paper's strengths was its theoretical framework casting MPO, AWR, SAC and other algorithms as special cases of a more unified control-as-inference objective, and the discussion of EM control vs KL control was rather enlightening. It was unclear to me, however, whether the authors were regarding the \"Implementational Details\" of Section 4.2 as possible transferable implementational choices, on the same level as activation functions and network sizes, or whether these were to be interpreted as different algorithms.\n\nMy main hesitations, however, lie with the empirical side of this paper, and for the reasons below, I do not believe the paper yet ready for acceptance:\n* the authors ablated the different implementational choices individual, but my concern is that there may be co-dependencies between different implementational choices, and individual ablations may fail to capture that. For instance, whether the tanh-squashing helps or hurts the policy network depends on whether and how the policies are regularised, something which may be more pertinent to KL control rather than EM.\n* there also seems to be a thematic disconnect between the four implementational choices the authors have focused on and the preceding control-as-inference discussion. The four implementational choices do not seem to depend in any way on the fact that MPO, AWR, and SAC can be treated in a unified, inference-based manner. Whether to use layer norm, ELU or RELU, large or small networks, tanh squashing, or double Q-learning is a question that applies equally well to any deep RL algorithm, not just the ones discussed here. Hence it is puzzling why the authors chose these four differences to focus on given the topic of their paper. Furthermore, if the differences in Section 4.2 can be regarded as implementational differences, it would have been more interesting and thematically-consistent with the first half of this paper to examine the transferability of those choices instead.\n* for algorithms that originally included a certain implementational choice, it is clear that ablation leads to a decline in performance. However, I find in several cases that the authors' claim of improved performance less convincing. For instance, the authors claim that using ELU and layer norm across all algorithms will improve performance. While doing this certainly didn't hurt performance and even led to a higher mean score in some cases, the score increases do not seem to reach the level of statistical significance given the size of the standard deviations.\n* finally, the scope of the empirical studies seemed a bit limited to me. The authors only looked at ablating four implementational choices, and it would have strengthened the paper significantly had the authors examined a wider selection (see e.g. Hessel et al, 2017 for an example of a more extensive ablation study in RL).\n\nMinor errors:\n* the authors reference equation 3 several times (e.g. lines 187 and 203), but this equation is never numbered in the main text. Presumably, it is the equation that appears in the M-step, after line 157.\n* typo in line 268:  “achieve” rather than “archive”.\n* typo in 294: “agreed upon default choice” rather than “agreeable default choice”.\n\n\n\n\n\n--- Review update ---\n\nAfter further consideration and based on the clarification and additional results the authors have provided, I have decided to upgrade my score to 6. To me, this work offers two very valuable contributions to the community. The first is the theoretical demonstration that many disparate policy gradient algorithms can actually be seen within a unified control-as-inference framework, a framing that I find novel and exciting. This unified framework not only provides a deeper understanding of pre-existing algorithm, but it also offers a new means to study and inform the design of future algorithms. The second is, as both reviewer CYmr and the AC have pointed out, the paper's attempt to survey the various algorithms already in the literature using this framework, examining how transferable certain implementational or code components are within the KL or EM families. What the authors are attempting with this work is definitely of great interest to the community.\n\nGiven the huge scope, I appreciate that exhaustive exploration on the empirical side is not possible. The new results were rather interesting, but for me, certain questions remain. For instance, the latest results have not entirely addressed my question about co-dependencies. From their limited transfer studies, the authors have tried to draw general conclusions about transferability between the KL or EM families; yet, I still wonder whether there are synergies between components that the authors break with their ablations and whether some of the performance drop has more to do with breaking these synergies than whether the algorithm is KL or EM. Again, as an example, I mention the possibility that soft-Q functions may synergise more with tanh-gaussian policies and clipped double q-learning. In the ablation studies, the authors found that using tanh-gaussianity and/or clipped double q-learning didn't transfer well to EM algorithms. But this could be because tanh-gaussianity and double clipped q-learning work best with a soft-Q function, and these components performed worse because soft-Q  was absent. This is just one example of how possible co-dependencies may confound the conclusions that the authors sought to make in their ablation studies. Regarding the new results, the failure case of alternative $\\pi_p$ was also rather intriguing. In short, there is a lot in the space of algorithms that remains to be explored, but doing an exhaustive exploration would be an enormous undertaking.\n\nIn the end, I am inclined to support accepting support of the paper not only for the reasons I mentioned at the beginning, but also because I believe doing so will provoke much important discussion and inspire new research to address the questions that may come out of it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Given that the foundational nature of this work, there would not be any malignant implications beyond that which is already inherent in basic RL, so a discussion of the potential societal impact is not relevant for this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers thought there were various ways that this paper could be improved, there was also general consensus that this framework which unites multiple existing algorithms was interesting and useful, prompting one reviewer to comment on the new insights it brought them. The authors are encouraged to address the feedback from the reviewers in their camera ready. 


Decision: Accept (Poster)