Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning

1 Introduction . Deep reinforcement learning ( RL ) has achieved huge empirical successes in both continuous [ 36 , 18 ] and discrete [ 39 , 26 ] problem settings with on-policy [ 52 , 54 ] and off-policy [ 13 , 22 , 23 ] algorithms . Especially in the continuous control domain , interpreting RL as probabilistic inference [ 60 , 61 ] has yielded many kinds of algorithms with strong empirical performances [ 34 , 50 , 51 , 12 , 29 , 21 , 16 , 24 ] . Recently , there has been a series of off-policy algorithms derived from this perspective for learning policies with function approximations [ 2 , 22 , 48 ] . Notably , Soft Actor Critic ( SAC ) [ 22 , 23 ] , based on a maximum entropy objective and soft Q-function , significantly outperforms on-policy [ 52 , 54 ] and off-policy [ 36 , 18 , 13 ] methods . Maximum a posteriori Policy Optimisation ( MPO ) [ 2 ] , 1The implementation is available at https : //github.com/frt03/inference-based-rl . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . inspired by REPS [ 50 ] , employs a pseudo-likelihood objective , and achieves high sample-efficiency and fast convergence compared to the variety of policy gradient methods [ 36 , 54 , 7 ] . Similar to MPO , Advantage Weighted Regression ( AWR ) [ 48 ] , and its variant , Advantage Weighted ActorCritic ( AWAC ) [ 42 ] , also employ the pseudo-likelihood objective weighted by the exponential of advantages , and reports more stable performance than baselines both in online and offline [ 35 ] settings . While these inference-based algorithms have similar derivations to each other , their empirical performances have large gaps among them when evaluated on the standard continuous control benchmarks , such as OpenAI Gym [ 6 ] or DeepMind Control Suite [ 58 ] . Critically , as each algorithm has unique low-level implementation or code design decisions – such as value estimation techniques , action distribution for the policy , and network architectures – aside from high-level algorithmic choices , it is difficult to exactly identify the causes of these performance gaps as algorithmic or implementational . In this paper , we first derive MPO , AWR , and SAC from a single objective function , through either Expectation-Maximization ( EM ) or KL minimization , mathematically clarifying the algorithmic connections among the recent state-of-the-art off-policy actor-critic algorithms . This unified derivation allows us to precisely identify implementation techniques and code details for each algorithm , which are residual design choices in each method that are generally transferable to other algorithms . To reveal the sources of the performance gaps , we experiment with carefully-selected ablations of these identified implementation techniques and code details , such as tanh-squashed Gaussian policy , clipped double Q-learning [ 13 ] , and network architectures . Specifically , we keep the high-level algorithmic designs while normalizing the implementation designs , enabling proper algorithmic comparisons . Our empirical results successfully distinguish between highly co-adapted design choices and no-co-adapted ones2 . We identified that clipped double Q-learning and tanh-squashed policies , the sources of SoTA performance of SAC , are highly co-adapted , specific to KL-minimization-based method , SAC , and difficult to transfer and benefit in EM-based methods , MPO or AWR . In contrast , we discover that ELU [ 8 ] and layer normalization [ 4 ] , the sources of SoTA performance of MPO , are transferable choices from MPO that also significantly benefit SAC . We hope our work can inspire more future works to precisely decouple algorithmic innovations from implementation or code details , which allows exact sources of performance gains to be identified and algorithmic researches to better build on one another . 2 Related Work . Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [ 60 , 61 , 34 , 11 , 57 , 45 ] , but many recently-proposed algorithms [ 2 , 22 , 48 ] are derived separately and their exact relationships are difficult to get out directly , due to mixing of algorithmic and implementational details , inconsistent implementation choices , environment-specific tunings , and benchmark differences . Our work organizes them as a unified policy iteration method , to clarify their exact mathematical algorithmic connections and tease out subtle , but important , implementation differences . We center our analyses around MPO , AWR , and SAC , because they are representative algorithms that span both EM-based [ 49 , 50 , 43 , 44 , 1 , 56 , 42 ] and KL-control-based [ 59 , 51 , 12 , 29 , 21 , 33 , 32 ] RL and achieve some of the most competitive performances on popular benchmarks [ 6 , 58 ] . REPS [ 50 ] , an EM approach , inspired MPO , AWR , and our unified objective in Eq . 1 , while Soft Q-learning [ 21 ] , a practical extension of KL control to continuous action space through Liu and Wang [ 38 ] , directly led to the development of SAC . Meta analyses of RL algorithms While many papers propose novel algorithms , some recent works focused on meta analyses of some of the popular algorithms , which attracted significant attention due to these algorithms ’ high-variance evaluation performances , reproducibility difficulty [ 9 , 28 , 65 ] , and frequent code-level optimizations [ 25 , 63 , 10 , 3 ] . Henderson et al . [ 25 ] empirically showed how these RL algorithms have inconsistent results across different official implementations and high variances even across runs with the same hyper-parameters , and recommended a concrete action item for the community – use more random seeds . Tucker et al . [ 63 ] show that high performances of actiondependent baselines [ 19 , 17 , 37 , 20 , 67 ] were more directly due to different subtle implementation choices . Engstrom et al . [ 10 ] focus solely on PPO and TRPO , two on-policy algorithms , and discuss 2We here regard as co-adaptation the indispensable implementation or code decisions that do not stem directly from the conceptual algorithmic development , but from empirical considerations . how code-level optimizations , instead of the claimed algorithmic differences , actually led more to PPO ’ s superior performances . Andrychowicz et al . [ 3 ] describes low-level ( e.g . hyper-parameter choice , and regularization ) and high-level ( e.g . policy loss ) design choices in on-policy algorithms affects the performance of PPO by showing results of large-scale evaluations . In contrast to those prior works that mainly focus on a single family of on-policy algorithms , PPO and TRPO , and evaluating their implementation details alone , our work focuses on two distinct families of off-policy algorithms , and more importantly , presents unifying mathematical connections among independently-proposed state-of-the-art algorithms . Our experiments demonstrate how some implementation choices in Table 1 and code details are co-evolved with algorithmic innovations and/or have non-trivial effects on the performance of off-policy inference-based methods . 3 Preliminaries . We consider a Markov Decision Process ( MDP ) defined by state space S , action space A , state transition probability function p : S × A× S → [ 0 , ∞ ) , initial state distribution p1 : S → [ 0 , ∞ ) , reward function r : S ×A → R , and discount factor γ ∈ [ 0 , 1 ) . Let Rt denote a discounted return∑∞ u=t γ u−tr ( su , au ) . We assume the standard RL setting , where the agent chooses actions based on a parametric policy πθ and seeks for parameters that maximize the expected return Eπθ [ R1 ] . Value functions for a policy π are the expected return conditioned by a state-action pair or a state , that is , Qπ ( st , at ) : = Eπ [ Rt|st , at ] , and V π ( st ) : = Eπ [ Rt|st ] . They are called the state-action-value function ( Q-function ) , and state-value function , respectively . The advantage function [ 5 ] is defined as Aπ ( s , a ) : = Qπ ( s , a ) − V π ( s ) . The Q-function for a policy π is the unique fixed point of the Bellman operator T π defined by T πQ ( s , a ) = r ( s , a ) + ∫ π ( a′|s′ ) p ( s′|s , a ) Q ( s′ , a′ ) ds′da′ for any function Q : S × A → R. We denote a trajectory or successive state-action sequence as τ : = ( s1 , a1 , s2 , a2 , . . . ) . We also define an unnormalized state distribution under the policy π by dπ ( s ) = ∑∞ t=1 γ tp ( st = s|π ) . Inference-based Methods For simplicity , we consider a finite-horizon MDP with a time horizon T for the time being . As a result , a trajectory τ becomes a finite length : τ : = ( s1 , a1 , . . . , sT−1 , aT−1 , sT ) . As in previous works motivated by probabilistic inference [ 34 , 2 ] , we introduce to the standard graphical model of an MDP a binary event variable Ot ∈ { 0 , 1 } , which represents whether the action in time step t is optimal or not . To derive the RL objective , we consider the marginal log-likelihood log Pr ( O = 1|πp ) , where πp is a policy . Note that O = 1 means Ot = 1 for every time step . As is well known , we can decompose this using a variational distribution q of a trajectory as follows : log Pr ( O = 1|πp ) = Eq [ log Pr ( O = 1|τ ) − log q ( τ ) p ( τ ) + log q ( τ ) p ( τ |O = 1 ) ] = J ( p , q ) +DKL ( q ( τ ) || p ( τ |O = 1 ) ) , where J ( p , q ) : = Eq [ log Pr ( O = 1|τ ) ] −DKL ( q ( τ ) || p ( τ ) ) is the evidence lower bound ( ELBO ) . Inference-based methods aim to find the parametric policy which maximizes the ELBO J ( p , q ) . There are several algorithmic design choices for q and Pr ( O = 1|τ ) . Since any q is valid , a popular choice is the one that factorizes in the same way as p , that is , p ( τ ) = p ( s1 ) ∏ t p ( st+1|st , at ) πp ( at|st ) , q ( τ ) = p ( s1 ) ∏ t p ( st+1|st , at ) πq ( at|st ) , where πp is a prior policy , and πq is a variational posterior policy . We may also choose which policy ( πp or πq ) to parameterize . As for Pr ( O = 1|τ ) , the most popular choice is the following one [ 21 , 22 , 2 , 34 , 48 ] : Pr ( O = 1|τ ) ∝ exp ( ∑T t=1 η −1G ( st , at ) ) , where η > 0 is a temperature , and G is a function over S ×A , such as an immediate reward function r , Q-function Qπ , and advantage function Aπ . While we employ exp ( · ) in the present paper , there are alternatives [ 46 , 47 , 55 , 66 ] : for example , Siegel et al . [ 55 ] and Wang et al . [ 66 ] consider an indicator function f : x ∈ R 7→ 1 [ x ≥ 0 ] , whereas Oh et al . [ 46 ] consider the rectified linear unit f : x ∈ R 7→ max { x , 0 } . Incorporating these design choices , we can rewrite the ELBO J ( p , q ) in a more explicit form as ; J ( πp , πq ) = T∑ t=1 Eq [ η−1G ( st , at ) −DKL ( πq ( ·|st ) || πp ( ·|st ) ) ] . In the following section , we adopt this ELBO and consider its relaxation to the infinite-horizon setting . Then , starting from it , we derive MPO , AWR , and SAC .

Summary: This paper fits in the category of survey/review paper. While this kind of paper is not common at Neurips, it doesn't mean that it should be disregarded. It does a good job at trying to unify a popular body of work on policy gradient methods under the umbrella of "control as inference". The paper is well-structured: it defines a taxonomy which differentiates between "EM-based" methods and those based on "direct KL minimization". The paper focuses on four algorithms: Maximum a posteriori Policy Optimisation (Abdolmaleki et al. 2018), Advantage-Weighted Regression (Peng et al. 2019), Soft Actor-Critic (Haarnoja et al. 2017) and Relative Entropy Policy Search (Peters et al. 2010). The  paper starts by explaining how the RL problem can be cast as a probabilistic inference problem and tries to explain how an evidence lower bound can obtained. It then goes into the specifics of each algorithm and how each either parametrizes the prior or the posterior and whether  it uses hard constraints (with closed-form expressions) or soft/regularized re-formulations.   The empirical section then compares the raw performance three of those algorithms (SAC, MPO and AWR) on six Mujoco environments.   Regarding the "co-adaptation" expression in the title, I wouldn't say it defines the paper. The "co-adaptation" findings really pertain to the impact of specific implementation details on the final performance in those six Mujoco environments. More specifically, the authors study the impact of double-q learning + clipping, ELU activations vs ReLu, network size and layer normalization. The empirical findings are nicely summarized through tables.