Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning

1 Introduction . Deep reinforcement learning ( RL ) has achieved huge empirical successes in both continuous [ 36 , 18 ] and discrete [ 39 , 26 ] problem settings with on-policy [ 52 , 54 ] and off-policy [ 13 , 22 , 23 ] algorithms . Especially in the continuous control domain , interpreting RL as probabilistic inference [ 60 , 61 ] has yielded many kinds of algorithms with strong empirical performances [ 34 , 50 , 51 , 12 , 29 , 21 , 16 , 24 ] . Recently , there has been a series of off-policy algorithms derived from this perspective for learning policies with function approximations [ 2 , 22 , 48 ] . Notably , Soft Actor Critic ( SAC ) [ 22 , 23 ] , based on a maximum entropy objective and soft Q-function , significantly outperforms on-policy [ 52 , 54 ] and off-policy [ 36 , 18 , 13 ] methods . Maximum a posteriori Policy Optimisation ( MPO ) [ 2 ] , 1The implementation is available at https : //github.com/frt03/inference-based-rl . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . inspired by REPS [ 50 ] , employs a pseudo-likelihood objective , and achieves high sample-efficiency and fast convergence compared to the variety of policy gradient methods [ 36 , 54 , 7 ] . Similar to MPO , Advantage Weighted Regression ( AWR ) [ 48 ] , and its variant , Advantage Weighted ActorCritic ( AWAC ) [ 42 ] , also employ the pseudo-likelihood objective weighted by the exponential of advantages , and reports more stable performance than baselines both in online and offline [ 35 ] settings . While these inference-based algorithms have similar derivations to each other , their empirical performances have large gaps among them when evaluated on the standard continuous control benchmarks , such as OpenAI Gym [ 6 ] or DeepMind Control Suite [ 58 ] . Critically , as each algorithm has unique low-level implementation or code design decisions – such as value estimation techniques , action distribution for the policy , and network architectures – aside from high-level algorithmic choices , it is difficult to exactly identify the causes of these performance gaps as algorithmic or implementational . In this paper , we first derive MPO , AWR , and SAC from a single objective function , through either Expectation-Maximization ( EM ) or KL minimization , mathematically clarifying the algorithmic connections among the recent state-of-the-art off-policy actor-critic algorithms . This unified derivation allows us to precisely identify implementation techniques and code details for each algorithm , which are residual design choices in each method that are generally transferable to other algorithms . To reveal the sources of the performance gaps , we experiment with carefully-selected ablations of these identified implementation techniques and code details , such as tanh-squashed Gaussian policy , clipped double Q-learning [ 13 ] , and network architectures . Specifically , we keep the high-level algorithmic designs while normalizing the implementation designs , enabling proper algorithmic comparisons . Our empirical results successfully distinguish between highly co-adapted design choices and no-co-adapted ones2 . We identified that clipped double Q-learning and tanh-squashed policies , the sources of SoTA performance of SAC , are highly co-adapted , specific to KL-minimization-based method , SAC , and difficult to transfer and benefit in EM-based methods , MPO or AWR . In contrast , we discover that ELU [ 8 ] and layer normalization [ 4 ] , the sources of SoTA performance of MPO , are transferable choices from MPO that also significantly benefit SAC . We hope our work can inspire more future works to precisely decouple algorithmic innovations from implementation or code details , which allows exact sources of performance gains to be identified and algorithmic researches to better build on one another . 2 Related Work . Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [ 60 , 61 , 34 , 11 , 57 , 45 ] , but many recently-proposed algorithms [ 2 , 22 , 48 ] are derived separately and their exact relationships are difficult to get out directly , due to mixing of algorithmic and implementational details , inconsistent implementation choices , environment-specific tunings , and benchmark differences . Our work organizes them as a unified policy iteration method , to clarify their exact mathematical algorithmic connections and tease out subtle , but important , implementation differences . We center our analyses around MPO , AWR , and SAC , because they are representative algorithms that span both EM-based [ 49 , 50 , 43 , 44 , 1 , 56 , 42 ] and KL-control-based [ 59 , 51 , 12 , 29 , 21 , 33 , 32 ] RL and achieve some of the most competitive performances on popular benchmarks [ 6 , 58 ] . REPS [ 50 ] , an EM approach , inspired MPO , AWR , and our unified objective in Eq . 1 , while Soft Q-learning [ 21 ] , a practical extension of KL control to continuous action space through Liu and Wang [ 38 ] , directly led to the development of SAC . Meta analyses of RL algorithms While many papers propose novel algorithms , some recent works focused on meta analyses of some of the popular algorithms , which attracted significant attention due to these algorithms ’ high-variance evaluation performances , reproducibility difficulty [ 9 , 28 , 65 ] , and frequent code-level optimizations [ 25 , 63 , 10 , 3 ] . Henderson et al . [ 25 ] empirically showed how these RL algorithms have inconsistent results across different official implementations and high variances even across runs with the same hyper-parameters , and recommended a concrete action item for the community – use more random seeds . Tucker et al . [ 63 ] show that high performances of actiondependent baselines [ 19 , 17 , 37 , 20 , 67 ] were more directly due to different subtle implementation choices . Engstrom et al . [ 10 ] focus solely on PPO and TRPO , two on-policy algorithms , and discuss 2We here regard as co-adaptation the indispensable implementation or code decisions that do not stem directly from the conceptual algorithmic development , but from empirical considerations . how code-level optimizations , instead of the claimed algorithmic differences , actually led more to PPO ’ s superior performances . Andrychowicz et al . [ 3 ] describes low-level ( e.g . hyper-parameter choice , and regularization ) and high-level ( e.g . policy loss ) design choices in on-policy algorithms affects the performance of PPO by showing results of large-scale evaluations . In contrast to those prior works that mainly focus on a single family of on-policy algorithms , PPO and TRPO , and evaluating their implementation details alone , our work focuses on two distinct families of off-policy algorithms , and more importantly , presents unifying mathematical connections among independently-proposed state-of-the-art algorithms . Our experiments demonstrate how some implementation choices in Table 1 and code details are co-evolved with algorithmic innovations and/or have non-trivial effects on the performance of off-policy inference-based methods . 3 Preliminaries . We consider a Markov Decision Process ( MDP ) defined by state space S , action space A , state transition probability function p : S × A× S → [ 0 , ∞ ) , initial state distribution p1 : S → [ 0 , ∞ ) , reward function r : S ×A → R , and discount factor γ ∈ [ 0 , 1 ) . Let Rt denote a discounted return∑∞ u=t γ u−tr ( su , au ) . We assume the standard RL setting , where the agent chooses actions based on a parametric policy πθ and seeks for parameters that maximize the expected return Eπθ [ R1 ] . Value functions for a policy π are the expected return conditioned by a state-action pair or a state , that is , Qπ ( st , at ) : = Eπ [ Rt|st , at ] , and V π ( st ) : = Eπ [ Rt|st ] . They are called the state-action-value function ( Q-function ) , and state-value function , respectively . The advantage function [ 5 ] is defined as Aπ ( s , a ) : = Qπ ( s , a ) − V π ( s ) . The Q-function for a policy π is the unique fixed point of the Bellman operator T π defined by T πQ ( s , a ) = r ( s , a ) + ∫ π ( a′|s′ ) p ( s′|s , a ) Q ( s′ , a′ ) ds′da′ for any function Q : S × A → R. We denote a trajectory or successive state-action sequence as τ : = ( s1 , a1 , s2 , a2 , . . . ) . We also define an unnormalized state distribution under the policy π by dπ ( s ) = ∑∞ t=1 γ tp ( st = s|π ) . Inference-based Methods For simplicity , we consider a finite-horizon MDP with a time horizon T for the time being . As a result , a trajectory τ becomes a finite length : τ : = ( s1 , a1 , . . . , sT−1 , aT−1 , sT ) . As in previous works motivated by probabilistic inference [ 34 , 2 ] , we introduce to the standard graphical model of an MDP a binary event variable Ot ∈ { 0 , 1 } , which represents whether the action in time step t is optimal or not . To derive the RL objective , we consider the marginal log-likelihood log Pr ( O = 1|πp ) , where πp is a policy . Note that O = 1 means Ot = 1 for every time step . As is well known , we can decompose this using a variational distribution q of a trajectory as follows : log Pr ( O = 1|πp ) = Eq [ log Pr ( O = 1|τ ) − log q ( τ ) p ( τ ) + log q ( τ ) p ( τ |O = 1 ) ] = J ( p , q ) +DKL ( q ( τ ) || p ( τ |O = 1 ) ) , where J ( p , q ) : = Eq [ log Pr ( O = 1|τ ) ] −DKL ( q ( τ ) || p ( τ ) ) is the evidence lower bound ( ELBO ) . Inference-based methods aim to find the parametric policy which maximizes the ELBO J ( p , q ) . There are several algorithmic design choices for q and Pr ( O = 1|τ ) . Since any q is valid , a popular choice is the one that factorizes in the same way as p , that is , p ( τ ) = p ( s1 ) ∏ t p ( st+1|st , at ) πp ( at|st ) , q ( τ ) = p ( s1 ) ∏ t p ( st+1|st , at ) πq ( at|st ) , where πp is a prior policy , and πq is a variational posterior policy . We may also choose which policy ( πp or πq ) to parameterize . As for Pr ( O = 1|τ ) , the most popular choice is the following one [ 21 , 22 , 2 , 34 , 48 ] : Pr ( O = 1|τ ) ∝ exp ( ∑T t=1 η −1G ( st , at ) ) , where η > 0 is a temperature , and G is a function over S ×A , such as an immediate reward function r , Q-function Qπ , and advantage function Aπ . While we employ exp ( · ) in the present paper , there are alternatives [ 46 , 47 , 55 , 66 ] : for example , Siegel et al . [ 55 ] and Wang et al . [ 66 ] consider an indicator function f : x ∈ R 7→ 1 [ x ≥ 0 ] , whereas Oh et al . [ 46 ] consider the rectified linear unit f : x ∈ R 7→ max { x , 0 } . Incorporating these design choices , we can rewrite the ELBO J ( p , q ) in a more explicit form as ; J ( πp , πq ) = T∑ t=1 Eq [ η−1G ( st , at ) −DKL ( πq ( ·|st ) || πp ( ·|st ) ) ] . In the following section , we adopt this ELBO and consider its relaxation to the infinite-horizon setting . Then , starting from it , we derive MPO , AWR , and SAC .

Summary: Comparing the performance of different RL algorithms can sometimes be difficult because algorithmic contributions can be confounded with differing implementational choices between the agents. This paper attempts to examine the relative contributions of algorithmic vs implementational choices in a series of inference-based, off-policy actor-critic algorithms -- MPO, AWR, and SAC. The authors' starting point is a unified view of these algorithms from the perspective of control-as-inference. In this framework, policies as cast as posterior probabilities that a given action in a given state will be optimal. The authors derive an ELBO for this posterior that, when optimised, yields an approximate optimal policy. The authors then describe how optimization can be approached in two ways: an EM approach (where both a prior and posterior policy are optimized) and a KL approach (where only the posterior policy is optimized). MPO and AWR belong to the EM family of algorithms, while SAC belongs to the KL family. The authors further detail how these algorithms fit into their unified framework.  In the second half of the paper, the authors focus on the implementational differences between these algorithms. They focus specifically on four choices: clipped double q-learning, tanh-squashing in the policies, the choice of activation function and layer norm, and the network size. Specifically, through a series of ablation studies, the authors examine how critical some of these choices are to specific algorithms and which of these can be transferred to other algorithms.