Probabilistic Planning with Sequential Monte Carlo methods

1 INTRODUCTION . To exhibit intelligent behaviour machine learning agents must be able to learn quickly , predict the consequences of their actions , and explain how they will react in a given situation . These abilities are best achieved when the agent efficiently uses a model of the world to plan future actions . To date , planning algorithms have yielded very impressive results . For instance , Alpha Go ( Silver et al. , 2017 ) relied on Monte Carlo Tree Search ( MCTS ) ( Kearns et al. , 2002 ) to achieve super human performances . Cross entropy methods ( CEM ) ( Rubinstein & Kroese , 2004 ) have enabled robots to perform complex nonprehensile manipulations ( Finn & Levine , 2017 ) and algorithms to play successfully Tetris ( Szita & Lörincz , 2006 ) . In addition , iterative linear quadratic regulator ( iLQR ) ( Kalman et al. , 1960 ; Kalman , 1964 ; Todorov & Li , 2005 ) enabled humanoid robots tasks to get up from an arbitrary seated pose ( Tassa et al. , 2012 ) . Despite these successes , these algorithms make strong underlying assumptions about the environment . First , MCTS requires a discrete setting , limiting most of its successes to discrete games with known dynamics . Second , CEM assumes the distribution over future trajectories to be Gaussian , i.e . unimodal . Third , iLQR assumes that the dynamics are locally linear-Gaussian , which is a strong assumption on the dynamics and would also assume the distribution over future optimal trajectories to be Gaussian . For these reasons , planning remains an open problem in environments with continuous actions and complex dynamics . In this paper , we address the limitations of the aforementioned planning algorithms by creating a more general view of planning that can leverage advances in deep learning ( DL ) and probabilistic inference methods . This allows us to approximate arbitrary complicated distributions over trajectories with non-linear dynamics . We frame planning as density estimation problem over optimal future trajectories in the context of control as inference ( Dayan & Hinton , 1997 ; Toussaint & Storkey , 2006 ; Toussaint , 2009 ; Rawlik et al. , 2010 ; 2012 ; Ziebart , 2010 ; Levine & Koltun , 2013 ) . This perspective allows us to make use of tools from the inference research community and , as previously mentioned , model any distribution over future trajectories . The planning distribution is complex since trajectories consist of an intertwined sequence of states and actions . Sequential Monte Carlo ( SMC ) ( Stewart & McCarty , 1992 ; Gordon et al. , 1993 ; Kitagawa , 1996 ) methods are flexible and efficient to model such a ∗both authors contributed equally . distribution by sequentially drawing from a simpler proposal distribution . From the SMC perspective , the policy can be seen as the proposal and a learned model of the world as the propagation distribution . This provides a natural way to combine model-free and model-based RL . Contribution . We depict the problem of planning as one of density estimation that can be estimated using SMC methods . We introduce a novel planning strategy based on the SMC class of algorithms , in which we treat the policy as the proposed distribution to be learned . We investigate how our method empirically compares with existing model-based methods and a strong model-free baseline on the standard benchmark Mujoco ( Todorov et al. , 2012 ) . 2 BACKGROUND . 2.1 CONTROL AS INFERENCE . We consider the general case of a Markov Decision Process ( MDP ) { S , A , penv , r , γ , µ } where S and A represent the state and action spaces respectively . We use the letters s and a to denote states and actions , which we consider to be continuous vectors . Further notations include : penv ( s′|s , a ) as the state transition probability of the environment , r ( s , a ) as the reward function , and γ ∈ [ 0 , 1 ) as the discount factor . µ denotes the probability distribution over initial states . This work focuses on an episodic formulation , with a fixed end-time of T . We define a trajectory as a sequence of state-action pairs xt : T = { ( st , at ) , . . . , ( sT , aT ) } , and we use the notation π for a policy which represents a distribution over actions conditioned on a state . Here π is parametrized by a neural network with parameters θ . The notation qθ ( x1 : T ) = µ ( s1 ) ∏T−1 t≥1 penv ( st+1|st , at ) ∏T t≥1 πθ ( at|st ) denotes the probability of a trajectory x1 : T under policy πθ . Traditionally , in reinforcement learning ( RL ) problems , the goal is to find the optimal policy that maximizes the expected return Eqθ [ ∑T t=1 γ trt ] . However , it is useful to frame RL as an inference problem within a probabilistic graphical framework ( Rawlik et al. , 2012 ; Toussaint & Storkey , 2006 ; Levine , 2018 ) . First , we introduce an auxiliary binary random variable Ot denoting the “ optimality “ of a pair ( st , at ) at time t and define its probability1 as p ( Ot = 1|st , at ) = exp ( r ( st , at ) ) . O is a convenience variable only here for the sake of modeling . By considering the variables ( st , at ) as latent and Ot as observed , we can construct a Hidden Markov Model ( HMM ) as depicted in figure 2.1 . Notice that the link s→ a is not present in figure 2.1 as the dependency of the optimal action on the state depends on the future observations . In this graphical model , the optimal policy is expressed as p ( at|st , Ot : T ) . The posterior probability of this graphical model can be written as2 : p ( x1 : T |O1 : T ) ∝ p ( x1 : T , O1 : T ) = µ ( s1 ) T−1∏ t=1 penv ( st+1|at , st ) exp ( T∑ t=1 r ( st , at ) + log p ( at ) ) . ( 2.1 ) It appears clearly that finding optimal trajectories is equivalent to finding plausible trajectories yielding a high return . 1as in Levine ( 2018 ) , if the rewards are bounded above , we can always remove a constant so that the probability is well defined . 2Notice that in the rest of the paper , we will abusively remove the product of the action priors ∏T t=1 p ( at ) = exp ( ∑T t=1 log p ( at ) ) from the joint as in Levine ( 2018 ) . We typically consider this term either constant or already included in the reward function . See Appendix A.2 for details . Many control as inference methods can be seen as approximating the density by optimizing its variational lower bound : log p ( O1 : T ) ≥ Ex1 : T∼qθ [ ∑T t=1 r ( st , at ) − log πθ ( at|st ) ] ( Rawlik et al. , 2012 ; Toussaint , 2009 ) . Instead of directly differentiating the variational lower bound for the whole trajectory , it is possible to take a message passing approach such as the one used in Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) and directly estimate the optimal policy p ( at|st , Ot : T ) using the backward message , i.e a soft Q function instead of the Monte Carlo return . 2.2 SEQUENTIAL MONTE CARLO METHODS . Since distributions over trajectories are complex , it is often difficult or impossible to directly draw samples from them . Fortunately in statistics , there are successful strategies for drawing samples from complex sequential distributions , such as SMC methods . For simplicity , in the remainder of this section we will overload the notation and refer to the target distribution as p ( x ) and the proposal distribution as q ( x ) . We wish to draw samples from p but we only know its unnormalized density . We will use the proposal q to draw samples and estimate p. In the next section , we will define the distributions p and q in the context of planning . Importance sampling ( IS ) : When x can be efficiently sampled from another simpler distribution q i.e . the proposal distribution , we can estimate the likelihood of any point x under p straightforwardly by computing the unnormalized importance sampling weights w ( x ) ∝ p ( x ) q ( x ) and using the identity p ( x ) = w̄ ( x ) q ( x ) where w̄ ( x ) = w ( x ) ∫ w ( x ) q ( x ) dx is defined as the normalized importance sampling weights . In practice , one draws N samples from q : { x ( n ) } Nn=1 ∼ q ; these are referred to as particles . The set of particles { x ( n ) } Nn=1 associated with their weights { w ( n ) } Nn=1 are simulations of samples from p. That is , we approximate the density p with a weighted sum of diracs from samples of q : p ( x ) ≈ N∑ n=1 w̄ ( n ) δx ( n ) ( x ) , withx ( n ) sampled from q where δx0 ( x ) denotes the Dirac delta mass located as x0 . Sequential Importance Sampling ( SIS ) : When our problem is sequential in nature x = x1 : T , sampling x1 : T at once can be a challenging or even intractable task . By exploiting the sequential structure , the unnormalized weights can be updated iteratively in an efficient manner : wt ( x1 : t ) = wt−1 ( x1 : t−1 ) p ( xt|x1 : t−1 ) q ( xt|x1 : t−1 ) . We call this the update step . This enables us to sample sequentially xt ∼ q ( xt|x1 : t−1 ) to finally obtain the set of particles { x ( n ) 1 : T } and their weights { w ( n ) T } linearly in the horizon T . Sequential Importance Resampling ( SIR ) : When the horizon T is long , samples from q usually have a low likelihood under p , and thus the quality of our approximation decreases exponentially with T . More concretely , the unnormalized weights w ( n ) t converge to 0 with t→∞ . This usually causes the normalized weight distribution to degenerate , with one weight having a mass of 1 and the others a mass of 0 . This phenomenon is known as weight impoverishment . One way to address weight impoverishment is to add a resampling step where each particle is stochastically resampled to higher likelihood regions at each time step . This can typically reduce the variance of the estimation from growing exponentially with t to growing linearly . 3 PLANNING AS PROBABILISTIC INFERENCE . In the context of control as inference , it is natural to see planning as the act of approximating a distribution of optimal future trajectories via simulation . In order to plan , an agent must possess a model of the world that can accurately capture consequences of its actions . In cases where multiple trajectories have the potential of being optimal , the agent must rationally partition its computational resources to explore each possibility . Given finite time , the agent must limit its planning to a finite horizon h. We , therefore , define planning as the act of approximating the optimal distribution over trajectories of length h. In the control-as-inference framework , this distribution is naturally expressed as p ( a1 , s2 , . . . sh , ah|O1 : T , s1 ) , where s1 represents our current state .

Summary: Sequential Monte Carlo (SMC) has since its inception some 25 years ago proved to be a powerful and generally applicable tool. The authors of this paper continue this development in a very interesting and natural way by showing how SMC can be used to solve challenging planning problems. This is a enabled by reformulating the planning problem as an inference problem via the recent trend referred to as "control as inference". While there is unfortunately no real world experiments, the simulations clearly illustrate the potential of the approach.