Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
Year: 2021

Abstract:
Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-independent and sometimes under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements across algorithms difficult. In this work, we focus on a series of off-policy inference-based actor-critic algorithms – MPO, AWR, and SAC – to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation or code details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO’s performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another’s both algorithmic and implementational innovations.1

heading: 1 Introduction
text: Deep reinforcement learning (RL) has achieved huge empirical successes in both continuous [36, 18] and discrete [39, 26] problem settings with on-policy [52, 54] and off-policy [13, 22, 23] algorithms. Especially in the continuous control domain, interpreting RL as probabilistic inference [60, 61] has yielded many kinds of algorithms with strong empirical performances [34, 50, 51, 12, 29, 21, 16, 24].
Recently, there has been a series of off-policy algorithms derived from this perspective for learning policies with function approximations [2, 22, 48]. Notably, Soft Actor Critic (SAC) [22, 23], based on a maximum entropy objective and soft Q-function, significantly outperforms on-policy [52, 54] and off-policy [36, 18, 13] methods. Maximum a posteriori Policy Optimisation (MPO) [2],
1The implementation is available at https://github.com/frt03/inference-based-rl.
35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.
inspired by REPS [50], employs a pseudo-likelihood objective, and achieves high sample-efficiency and fast convergence compared to the variety of policy gradient methods [36, 54, 7]. Similar to MPO, Advantage Weighted Regression (AWR) [48], and its variant, Advantage Weighted ActorCritic (AWAC) [42], also employ the pseudo-likelihood objective weighted by the exponential of advantages, and reports more stable performance than baselines both in online and offline [35] settings.
While these inference-based algorithms have similar derivations to each other, their empirical performances have large gaps among them when evaluated on the standard continuous control benchmarks, such as OpenAI Gym [6] or DeepMind Control Suite [58]. Critically, as each algorithm has unique low-level implementation or code design decisions – such as value estimation techniques, action distribution for the policy, and network architectures – aside from high-level algorithmic choices, it is difficult to exactly identify the causes of these performance gaps as algorithmic or implementational.
In this paper, we first derive MPO, AWR, and SAC from a single objective function, through either Expectation-Maximization (EM) or KL minimization, mathematically clarifying the algorithmic connections among the recent state-of-the-art off-policy actor-critic algorithms. This unified derivation allows us to precisely identify implementation techniques and code details for each algorithm, which are residual design choices in each method that are generally transferable to other algorithms. To reveal the sources of the performance gaps, we experiment with carefully-selected ablations of these identified implementation techniques and code details, such as tanh-squashed Gaussian policy, clipped double Q-learning [13], and network architectures. Specifically, we keep the high-level algorithmic designs while normalizing the implementation designs, enabling proper algorithmic comparisons. Our empirical results successfully distinguish between highly co-adapted design choices and no-co-adapted ones2. We identified that clipped double Q-learning and tanh-squashed policies, the sources of SoTA performance of SAC, are highly co-adapted, specific to KL-minimization-based method, SAC, and difficult to transfer and benefit in EM-based methods, MPO or AWR. In contrast, we discover that ELU [8] and layer normalization [4], the sources of SoTA performance of MPO, are transferable choices from MPO that also significantly benefit SAC. We hope our work can inspire more future works to precisely decouple algorithmic innovations from implementation or code details, which allows exact sources of performance gains to be identified and algorithmic researches to better build on one another.

heading: 2 Related Work
text: Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [60, 61, 34, 11, 57, 45], but many recently-proposed algorithms [2, 22, 48] are derived separately and their exact relationships are difficult to get out directly, due to mixing of algorithmic and implementational details, inconsistent implementation choices, environment-specific tunings, and benchmark differences. Our work organizes them as a unified policy iteration method, to clarify their exact mathematical algorithmic connections and tease out subtle, but important, implementation differences. We center our analyses around MPO, AWR, and SAC, because they are representative algorithms that span both EM-based [49, 50, 43, 44, 1, 56, 42] and KL-control-based [59, 51, 12, 29, 21, 33, 32] RL and achieve some of the most competitive performances on popular benchmarks [6, 58]. REPS [50], an EM approach, inspired MPO, AWR, and our unified objective in Eq. 1, while Soft Q-learning [21], a practical extension of KL control to continuous action space through Liu and Wang [38], directly led to the development of SAC.
Meta analyses of RL algorithms While many papers propose novel algorithms, some recent works focused on meta analyses of some of the popular algorithms, which attracted significant attention due to these algorithms’ high-variance evaluation performances, reproducibility difficulty [9, 28, 65], and frequent code-level optimizations [25, 63, 10, 3]. Henderson et al. [25] empirically showed how these RL algorithms have inconsistent results across different official implementations and high variances even across runs with the same hyper-parameters, and recommended a concrete action item for the community – use more random seeds. Tucker et al. [63] show that high performances of actiondependent baselines [19, 17, 37, 20, 67] were more directly due to different subtle implementation choices. Engstrom et al. [10] focus solely on PPO and TRPO, two on-policy algorithms, and discuss
2We here regard as co-adaptation the indispensable implementation or code decisions that do not stem directly from the conceptual algorithmic development, but from empirical considerations.
how code-level optimizations, instead of the claimed algorithmic differences, actually led more to PPO’s superior performances. Andrychowicz et al. [3] describes low-level (e.g. hyper-parameter choice, and regularization) and high-level (e.g. policy loss) design choices in on-policy algorithms affects the performance of PPO by showing results of large-scale evaluations.
In contrast to those prior works that mainly focus on a single family of on-policy algorithms, PPO and TRPO, and evaluating their implementation details alone, our work focuses on two distinct families of off-policy algorithms, and more importantly, presents unifying mathematical connections among independently-proposed state-of-the-art algorithms. Our experiments demonstrate how some implementation choices in Table 1 and code details are co-evolved with algorithmic innovations and/or have non-trivial effects on the performance of off-policy inference-based methods.

heading: 3 Preliminaries
text: We consider a Markov Decision Process (MDP) defined by state space S, action space A, state transition probability function p : S × A× S → [0,∞), initial state distribution p1 : S → [0,∞), reward function r : S ×A → R, and discount factor γ ∈ [0, 1). Let Rt denote a discounted return∑∞ u=t γ
u−tr(su, au). We assume the standard RL setting, where the agent chooses actions based on a parametric policy πθ and seeks for parameters that maximize the expected return Eπθ [R1]. Value functions for a policy π are the expected return conditioned by a state-action pair or a state, that is, Qπ(st, at) := Eπ[Rt|st, at], and V π(st) := Eπ[Rt|st]. They are called the state-action-value function (Q-function), and state-value function, respectively. The advantage function [5] is defined as Aπ(s, a) := Qπ(s, a) − V π(s). The Q-function for a policy π is the unique fixed point of the Bellman operator T π defined by T πQ(s, a) = r(s, a) + ∫ π(a′|s′)p(s′|s, a)Q(s′, a′) ds′da′ for any function Q : S × A → R. We denote a trajectory or successive state-action sequence as τ := (s1, a1, s2, a2, . . . ). We also define an unnormalized state distribution under the policy π by dπ(s) = ∑∞ t=1 γ tp(st = s|π).
Inference-based Methods For simplicity, we consider a finite-horizon MDP with a time horizon T for the time being. As a result, a trajectory τ becomes a finite length: τ := (s1, a1, . . . , sT−1, aT−1, sT ).
As in previous works motivated by probabilistic inference [34, 2], we introduce to the standard graphical model of an MDP a binary event variable Ot ∈ {0, 1}, which represents whether the action in time step t is optimal or not. To derive the RL objective, we consider the marginal log-likelihood log Pr(O = 1|πp), where πp is a policy. Note that O = 1 means Ot = 1 for every time step. As is
well known, we can decompose this using a variational distribution q of a trajectory as follows: log Pr (O = 1|πp) = Eq [ log Pr(O = 1|τ)− log q(τ)
p(τ) + log
q(τ)
p(τ |O = 1) ] = J (p, q) +DKL(q(τ) || p(τ |O = 1)),
where J (p, q) := Eq [log Pr(O = 1|τ)]−DKL(q(τ) || p(τ)) is the evidence lower bound (ELBO). Inference-based methods aim to find the parametric policy which maximizes the ELBO J (p, q). There are several algorithmic design choices for q and Pr(O = 1|τ). Since any q is valid, a popular choice is the one that factorizes in the same way as p, that is,
p(τ) = p(s1) ∏ t p(st+1|st, at)πp(at|st), q(τ) = p(s1) ∏ t p(st+1|st, at)πq(at|st),
where πp is a prior policy, and πq is a variational posterior policy. We may also choose which policy (πp or πq) to parameterize. As for Pr(O = 1|τ), the most popular choice is the following one [21, 22, 2, 34, 48]:
Pr(O = 1|τ) ∝ exp (∑T
t=1 η −1G(st, at)
) ,
where η > 0 is a temperature, and G is a function over S ×A, such as an immediate reward function r, Q-function Qπ, and advantage function Aπ. While we employ exp(·) in the present paper, there are alternatives [46, 47, 55, 66]: for example, Siegel et al. [55] and Wang et al. [66] consider an indicator function f : x ∈ R 7→ 1[x ≥ 0], whereas Oh et al. [46] consider the rectified linear unit f : x ∈ R 7→ max{x, 0}. Incorporating these design choices, we can rewrite the ELBO J (p, q) in a more explicit form as;
J (πp, πq) = T∑ t=1 Eq [ η−1G(st, at)−DKL(πq(·|st) || πp(·|st)) ] .
In the following section, we adopt this ELBO and consider its relaxation to the infinite-horizon setting. Then, starting from it, we derive MPO, AWR, and SAC.

heading: 4 A Unified View of Inference-based Off-Policy Actor-Critic Algorithms
text: In Sec. 3, we provided the explicit form of the ELBO J (πp, πq). However, in practice, it is difficult to maximize it as the expectation Eq depends on πq. Furthermore, since we are interested in a finite-horizon setting, we replace ∑ t∈[T ] Eq with Edπ(s). Note that the latter expectation Edπ(s) is taken with the unnormalized state distribution dπ under an arbitrary policy π. This is commonly assumed in previous works [2, 22]. This relaxation leads to the following optimization:
max πp,πq
J (πp, πq) s.t. ∫ dπ(s) ∫ πp(a|s) dads = 1 and ∫ dπ(s) ∫ πq(a|s) dads = 1 , (1)
where J (πp, πq) = Edπ(s)[η−1G(s, a)−DKL(πq || πp)] . With this objective, we can regard recent popular SoTA off-policy algorithms, MPO [2], AWR [48], and SAC [22] as variants of a unified policy iteration method. The components are summarized in Table 1. We first explain how these algorithms can be grouped into two categories of approaches for solving Eq. 1, and then we elaborate additional implementation details each algorithm makes.

heading: 4.1 Unified Policy Iteration: Algorithmic Perspective
text: Eq. 1 allows the following algorithmic choices: how or if to parameterize πp and πq; what optimizer to use for them; and if to optimize them jointly, or individually while holding the other fixed. We show that the algorithms in Table 1 can be classified into two categories: Expectation-Maximization control (EM control) and direct Kullback-Leibler divergence minimization control (KL control).

heading: 4.1.1 Expectation-Maximization (EM) Control
text: This category subsumes MPO [2] (similarly REPS [50]), AWR [48], and AWAC [42], and we term it EM control. At high level, the algorithm non-parametrically solves for the variational posterior πq while holding the parametric prior πp = πθp fixed (E-Step), and then optimize πp holding the new πq fixed (M-Step). This can be viewed as either a generic EM algorithm and as performing coordinate ascent on Eq. 1. We denote θp and πq after iteration k of EM steps by θ (k) p and π (k) q , respectively.
In E-step at iteration k, we force the variational posterior policy π(k)q to be close to the optimal posterior policy, i.e., the maximizer of the ELBO J (π
θ (k−1) p , πq) with respect to πq. EM control converts the hard-constraint optimization problem in Eq. 1 to solving the following Lagrangian,
J (πq, β) = ∫ dπ(s) ∫ πq(a|s)η−1G(s, a) dads
− ∫ dπ(s) ∫ πq(a|s) log
πq(a|s) π θ (k−1) p (a|s) dads+ β
( 1− ∫ dπ(s) ∫ πq(a|s) dads ) . (2)
We analytically obtain the solution of Eq. 2, π(k)q (a|s) = Z(s)−1πθ(k−1)p (a|s) exp ( η−1G(s, a) ) ,
where Z(s) is the partition function.
In M-Step at iteration k, we maximize the ELBO J (π(k)q , πp) with respect to πp. Considering the optimization with respect to πp in Eq. 1 results in forward KL minimization, which is often referred to as a pseudo-likelihood or policy projection objective,
max θp E dπ(s)π (k) q (a|s)
[ log πθp(a|s) ] = max
θp Edπ(s)π θ (k−1) p (a|s)
[ log πθp(a|s)
Z(s) exp
( η−1G(s, a) )] ,
(3) where we may approximate Z(s) ≈ 1M ∑M j=1 exp(η −1G(s, aj)) with aj ∼ πθ(k−1)p (·|s) in practice.

heading: 4.1.2 Direct Kullback-Leibler (KL) Divergence Minimization Control
text: In contrast to the EM control in Sec. 4.1.1, we only optimize the variational posterior πq while holding the prior πp fixed. In this scheme, πq is parameterized, so we denote it as πθq . This leads to KL control [59, 62, 51, 12, 29, 21].
Equivalent to E-step in Sec. 4.1.1, we force the variational posterior policy πθq to be close to the optimal posterior policy, i.e, the maximizer of J (πp, πq) with respect to πq. The difference is that instead of analytically solving J (πp, πq) for πq, we optimize J (πp, πθq ) with respect to θq, which results in the following objective,
max θq
Edπ(s)πθq (a|s) [ η−1G(s, a)− log πθq (a|s) πp(a|s) ] .

heading: 4.1.3 “Optimal” Policies of EM and KL Control
text: While we formulate EM and KL control in a unified framework, we note that they have a fundamental difference in their definition of “optimal” policies. KL control fixes πp and converges to a regularizedoptimal policy in an exact case [15, 64]. In contrast, EM control continues updating both πp and πq , resulting in convergence to the standard optimal policy in an exact case [51]. We also note that EM control solves KL control as a sub-problem; for example, the first E-step exactly corresponds to a KL control problem, except for the policy parameterization.

heading: 4.2 Unified Policy Iteration: Implementation Details
text: In this section, we explain the missing pieces of MPO, AWR, and SAC in Sec. 4.1. Additionally, we also describe the details of other algorithms [30, 49, 50, 41, 56, 36, 13, 52, 54] from the EM and KL control perspective. See Appendix B for the details.

heading: 4.2.1 MPO
text: Algorithm MPO closely follows the EM control scheme explained in Sec. 4.1.1, wherein πp = πθp is parametric, and πq is non-parametric.
Implementation [πq Update] This corresponds to the E-step. MPO uses a trust-region (TR) method. Concretely, it replaces the reverse KL penalty (second term) in the Lagrangian (Eq. 2) with a constraint and analytically solves it for πq. As a result, η is also optimized during the training by minimizing the dual of Eq. 2, which resembles REPS [50]:
g(η) = η + η logEdπ(s)π θ (k−1) p (a|s)
[ exp ( η−1Q π θ (k−1) p (s, a) )] .
[πp Update] This corresponds to the M-step. MPO uses a combination of Stochastic Gradient (SG) ascent and trust-region method based on a forward KL divergence similarly to TRPO [52]. Concretely, it obtains θ(k)p by maximizing the objective (Eq. 3) with respect to θp subject to Edπ(s)[DKL(πθ(k−1)p (·|s) || πθp(·|s))] ≤ ε. MPO further decompose this KL divergence to two terms, assuming Gaussian policies; one term includes only the mean vector of πθp(·|s), and the other includes only its covariance matrix. Abdolmaleki et al. [2] justify this as a log-prior of MAP estimation, which is assumed as a second-order approximation of KL divergence.
[G and G Estimate] MPO uses the Q-function of π θ (k−1) p as G in the k-th E-step. It originally uses the Retrace update [40], while its practical implementation [27] uses a single-step Bellman update.
[πθ] For a parameterized policy, MPO uses a Gaussian distribution with state-dependent mean vector and diagonal covariance matrix. The trust-region method in MPO’s M-step heavily relies on this Gaussian assumption. Since a Gaussian distribution has an infinite support, MPO has a penalty term in its policy loss function that forces the mean of the policy to stay within the range of action space.

heading: 4.2.2 AWR and AWAC
text: Algorithm AWR slightly deviates from the EM control. In the M-step, AWR simply set Z(s) to 1.
Implementation [πq Update] This corresponds the E-step. AWR and AWAC analytically solves the Lagrangian (Eq. 2). In contrast to MPO, they don’t use trust-region method.
[πp Update] This corresponds to the M-step. At iteration k, AWR uses an average of all previous policies π̃pk := 1k ∑k−1 j=0 πθ(j)p instead of π θ (k−1) p
(cf. Eq. 3). In practice, the average policy π̃pk is replaced by samples of actions from a replay buffer, which stores action samples of previous policies.
[G and G Estimate] AWR uses the advantage function of π̃pk as G in the k-th E-step, and learns the state-value function of π
θ (k−1) p with TD(λ) [53]. Due to its choice of π̃pk , this avoids importance corrections [40]. In contrast, AWAC estimates the advantage via the Q-function with TD(0).
[πθ] For a parameterized policy, both AWR and AWAC use a Gaussian distribution with statedependent mean vector and state-independent diagonal covariance matrix (a constant one for AWR). As in MPO, they uses a penalty term to keep the mean of the policy within the range of action space.

heading: 4.2.3 SAC
text: Algorithm Contrary to MPO and AWR, SAC follows the KL control scheme explained in Sec. 4.1.2, wherein the variational posterior policy πq = πθq is parameterized, and the prior policy πp is fixed to the uniform distribution over the action space A. SAC uses as G a soft Q-function:
Q πθq soft (st, at) := r(st, at) + γEπθq
[ V πθq soft (st+1) ] ,
V πθq
soft (st) := V πθq (st) + Eπθq [∑∞ u=t γ u−tηH(πθq (·|st)) ∣∣st] ,
withH(πθq (·|st)) being −Eπθq [log πθq (at|st)|st].
Implementation [πq and πp Update] SAC performs stochastic gradient (SG) ascent updates of only πq , where the objective function is shown in Eq. 1. SAC has no operation equivalent to M-step in MPO and AWR, since it keeps the prior policy πp to the uniform distribution.
Similar to the temperature tuning in MPO, Haarnoja et al. [23] consider the dual function of entropyconstrained Lagrangian and treat the temperature η as a Lagrange multiplier, which results in,
g(η) = −ηH̄ − ηEdπ(s)πθq (a|s) [ log πθq (a|s) ] ,
where H̄ is the target entropy to ensure that the entropy of the policy should be larger than it.
[G and G Estimate] SAC uses Qπθqsoft as G, and TD(0)-like algorithm based on this soft Q-function. In policy evaluation, clipped double Q-learning [13], which retains multiple Q-functions (typically two) and takes the minimum, is employed to suppress the overestimation of Q value.
[πθ] The policy of SAC is a squashed Gaussian distribution: SAC first samples a random variable u from a Gaussian distribution N (µθ(s),Σθ(s)) with state-dependent mean vector and diagonal covariance; then, it applies the tanh function to u and obtain the action a ∈ [−1, 1]|A|. (If a dimension of the action space is not [−1, 1], an appropriate scaling and shift by an affine function is applied after the squashing.) Tanh squashing prevents out-of-bounds action.

heading: 4.3 Empirical Comparison
text: To evaluate the empirical performance of these off-policy algorithms (MPO, AWR, AWAC, and SAC), we compare their performances on Open AI Gym MuJoCo environments, namely, Hopper, Walker2d, HalfCheetah, Ant, Humanoid, and Swimmer, following Haarnoja et al. [23]. We reproduce all algorithms based on pytorch RL library [14], referring their original implementations [22, 48, 42, 27]. Figure 1 shows that SAC outperforms others in four environments (Hopper, Walker2d, HalfCheetah, and Humanoid), while MPO in Ant and AWR in Swimmer achieves the best performance. Generally, SAC seems to perform consistently better. We extensively evaluate MPO, AWR, and SAC on the 28 tasks on DeepMind Control Suite and 3 MuJoCo manipulation tasks. See Appendix C and D for the details.

heading: 5 Evaluation on Implementational Choices
text: In Sec. 4.3, SAC shows notable results in most environments, while we revealed that the derivation and formulation of those methods resemble each other. To specify the effect of each implementational
or code detail on the performance, we experiment with extensive and careful one-by-one ablations on; (1) clipped double Q-learning, (2) action distribution for the policy, (3) activation and normalization, and (4) network size. The former two correspond to implementation details (the choice of G estimate and the parameterization of the policy), and the latter two correspond to code details (see Appendix G for further experiments on other implementation details, such as πp update or the choice of G). We also conclude several recommendations for the practitioners (Table 6).

heading: 5.1 Clipped Double Q-Learning
text: We hypothesize that clipped double Q-learning has a large effect on the notable performance of SAC, and could be transferable in EM control methods. To verify this, we test the effect of clipped double Q-learning. Instead of AWR, here we evaluate AWAC since it uses Q-function. Table 2 shows that single-Q SAC outperforms original one in HalfCheetah and Swimmer that do not have the termination of the episode, while struggles to learn in Hopper, Walker, Ant and Humanoid that have the episodic termination conditions. A hypothesis is that single-Q SAC obtains a more exploratory policy due to larger overestimation bias, which can help in environments where explorations are safe, but hurt in environments where reckless explorations lead to immediate terminations. In contrast, clipped double Q-learning does not help MPO or AWAC as significantly as SAC; most results do not change or lead to slight improvements over the originals. This suggests that clipped double Q-learning might be a co-dependent and indispensable choice to KL control methods.
Recommendation Use clipped double Q-learning as a default choice, but you can omit it in EM control methods or non-terminal environments, such as HalfCheetah or Swimmer.

heading: 5.2 Action Distribution for the Policy
text: Another hypothesis is that the tanh-squashed policy (last column in Table 1) is an important and transferable design choice. We compare SAC without tanh transform (with MPO action penalty instead) to the original one, which results in drastic degradation (Table 3). This can be caused by the maximum entropy objective that encourages maximizing the covariance. These observations suggest that the high performance of SAC seems to highly depend on the implementational choice of the policy distribution. In contrast, MPO and AWR with tanh squashing don’t enjoy such large performance gain, or achieve worse cumulative rewards. This also suggests that tanh-squashed policy might be highly co-adapted in SAC and less transferable to EM control methods. Note that for EM control, we must clip the actions to keep them within the supports of distributions; a ∈ [−1 + , 1− ]|A|. We found that no clipping cause significant numerical instability. See Appendix H for the details.
Recommendation Use the original distributions for each algorithm. If you use the tanh-squashed Gaussian in EM control methods, clip the action to be careful for the numerical instability.

heading: 5.3 Activation and Normalization
text: The implementation of MPO [27] has some code-level detailed choices; ELU activation [8] and layer normalization [4]. We hypothesize that these, sometimes unfamiliar, code details in practical implementation stabilize the learning process, and contribute to the performance of MPO most.
Especially in Ant (Sec. 4.3), MPO significantly outperforms SAC. To investigate this much deeper, we add them into SAC or AWR and remove them from MPO, while maintaining the rest of the implementations. Table 4 shows that layer normalization can contribute to a significantly higher performance of SAC in Ant, and replacing ReLU with ELU also improves performance a lot in Swimmer, where AWR is the best in Figure 1. In contrast, the performance of MPO drastically collapsed when we just removed ELU and layer normalization. This observation suggests that these code-level choices are not only indispensable for MPO, but transferable and beneficial to both KL and EM control methods. Additionally, we tested incorporating ELU and layer normalization to SAC in 12 DM Control tasks where MPO outperformed SAC, and observed that they again often benefit SAC performances substantially. See Appendix C for the details.
Recommendation It is worth considering to replace the activation function from ReLU to ELU and incorporate the layer normalization, to achieve the best performance in several tasks. Both of them are transferable between EM and KL control methods.

heading: 5.4 Network Size
text: While on-policy algorithms such as PPO and TRPO can use the common network architecture as in prior works [10, 25], the architecture that works well in all the off-policy inference-based methods is still not obvious, and the RL community does not have agreed upon default choice. To validate the dependency on the networks among the inference-based methods, we exchange the size and number of hidden layers in the policy and value networks. We denote the network size of MPO ((256, 256, 256) for policy and (512, 512, 256) for value) as large (L), of SAC ((256, 256) for policy and value) as middle (M), and of AWR ((128, 64) for policy and value) as small (S) (see Appendix A for the details). Table 5 illustrates that SAC with large network seems to have better performance. However, as shown in Appendix F (Figure 8), the learning curve sometimes becomes unstable. While the certain degree of robustness to the network size is observed in SAC, EM control methods, MPO
and AWR, seem fragile and more dependent on the specific network size. This trend is remarkable in MPO. AWR (M) or (L) also struggles to learn in high-dimensional state tasks, such as Ant (111 dim), or Humanoid (376 dim). The results also imply that, in contrast to on-policy methods [3], the network size in the off-policy inference-based methods seems a less transferable choice.
Recommendation For SAC, use medium size network. Large size will also work, but the learning curve might be unstable. For MPO, we strongly recommend to stick to large size, because it is very sensitive to the network size. For AWR, using small size is a better choice, especially in high-dimensional state tasks, such as Ant, or Humanoid.
MPO AWR SAC Clipped Double Q [5.1] 4 4 © Tanh Gaussian [5.2] 4 4 © ELU & LayerNorm [5.3] © © © Large Network [5.4] © 4 © Medium Network [5.4] × 4 © Small Network [5.4] × © 4
tectures of inference-based off-policy algorithms, especially EM controls, seem more co-dependent than on-policy methods like PPO or TRPO, which therefore have significant impacts on the overall algorithm performances and need to be carefully tuned per algorithm. Such dependence of each algorithmic innovation on specific hand-tuned implementation details makes accurate performance gain attributions and cumulative build-up of research insights difficult. In contrast, we also find that some code-level implementation details, such as ELU and layer normalization, are not only indispensable choice to MPO, but also transferable and beneficial to SAC substantially. We hope our work can encourage more works that study precisely the impacts of algorithmic properties and empirical design choices, not only for one type of algorithms, but also across a broader spectrum of deep RL algorithms.

heading: Acknowledgements
text: We thank Yusuke Iwasawa, Masahiro Suzuki, Marc G. Bellemare, Ofir Nachum, and Sergey Levine for many fruitful discussions and comments. This work has been supported by the Mohammed bin Salman Center for Future Science and Technology for Saudi-Japan Vision 2030 at The University of Tokyo (MbSC2030).

Review summaries:
Comparing the performance of different RL algorithms can sometimes be difficult because algorithmic contributions can be confounded with differing implementational choices between the agents. This paper attempts to examine the relative contributions of algorithmic vs implementational choices in a series of inference-based, off-policy actor-critic algorithms -- MPO, AWR, and SAC. The authors' starting point is a unified view of these algorithms from the perspective of control-as-inference. In this framework, policies as cast as posterior probabilities that a given action in a given state will be optimal. The authors derive an ELBO for this posterior that, when optimised, yields an approximate optimal policy. The authors then describe how optimization can be approached in two ways: an EM approach (where both a prior and posterior policy are optimized) and a KL approach (where only the posterior policy is optimized). MPO and AWR belong to the EM family of algorithms, while SAC belongs to the KL family. The authors further detail how these algorithms fit into their unified framework.  In the second half of the paper, the authors focus on the implementational differences between these algorithms. They focus specifically on four choices: clipped double q-learning, tanh-squashing in the policies, the choice of activation function and layer norm, and the network size. Specifically, through a series of ablation studies, the authors examine how critical some of these choices are to specific algorithms and which of these can be transferred to other algorithms.

This paper fits in the category of survey/review paper. While this kind of paper is not common at Neurips, it doesn't mean that it should be disregarded. It does a good job at trying to unify a popular body of work on policy gradient methods under the umbrella of "control as inference". The paper is well-structured: it defines a taxonomy which differentiates between "EM-based" methods and those based on "direct KL minimization". The paper focuses on four algorithms: Maximum a posteriori Policy Optimisation (Abdolmaleki et al. 2018), Advantage-Weighted Regression (Peng et al. 2019), Soft Actor-Critic (Haarnoja et al. 2017) and Relative Entropy Policy Search (Peters et al. 2010). The  paper starts by explaining how the RL problem can be cast as a probabilistic inference problem and tries to explain how an evidence lower bound can obtained. It then goes into the specifics of each algorithm and how each either parametrizes the prior or the posterior and whether  it uses hard constraints (with closed-form expressions) or soft/regularized re-formulations.   The empirical section then compares the raw performance three of those algorithms (SAC, MPO and AWR) on six Mujoco environments.   Regarding the "co-adaptation" expression in the title, I wouldn't say it defines the paper. The "co-adaptation" findings really pertain to the impact of specific implementation details on the final performance in those six Mujoco environments. More specifically, the authors study the impact of double-q learning + clipping, ELU activations vs ReLu, network size and layer normalization. The empirical findings are nicely summarized through tables.