Title: DUAL SEQUENTIAL MONTE CARLO: TUNNELING FILTERING AND PLANNING IN CONTINUOUS POMDPS

Abstract: We present the DualSMC network that solves continuous POMDPs by learning belief representations and then leveraging them for planning. It is based on the fact that filtering, i.e. state estimation, and planning can be viewed as two related sequential Monte Carlo processes, with one in the belief space and the other in the future planning trajectory space. In particular, we first introduce a novel particle filter network that makes better use of the adversarial relationship between the proposer model and the observation model. We then introduce a new planning algorithm over the belief representations, which learns uncertainty-dependent policies. We allow these two parts to be trained jointly with each other. We testify the effectiveness of our approach on three continuous control and planning tasks: the floor positioning, the 3D light-dark navigation, and a modified Reacher task. Weight particles based on observation SMC planning based on top-M (blue) particles Resample old / propose new particles Transit all particles based on the MPC action from SMC planning Filtering steps Planning step True state Goal * Depending on the task, we can keep K constant or make (K -K ) follow an exponential decay.

1. INTRODUCTION:

Partially observable Markov Decision Processes (POMDPs) formulate reinforcement learning problems where the robot's instant observation is insufficient for optimal decision making (Kaelbling et al., 1998) . POMDPs can easily become intractable in moderately large discrete space, let alone in continuous domains (Papadimitriou & Tsitsiklis, 1987) . As a result, sampling-based methods are typically employed. For example, Monte Carlo tree search (MCTS) methods have shown success in relatively large POMDPs by constructing a search tree of history based on rollout simulations (Silver & Veness, 2010; Seiler et al., 2015; Sunberg & Kochenderfer, 2018) . Cross-entropy methods (CEM) (De Boer et al., 2005) update a distribution over policies based on sampled trajectories and have achieved promising results in Decentralized POMDPs (Oliehoek et al., 2008; Omidshafiei et al., 2016) . Despite their effectiveness, MCTS methods often require a black-box simulator to generate planning trajectory and thus limiting their success to environments with known dynamics. CEM usually assumes all distributions are Gaussian and therefore is restricted to unimodal planning.

On the other hand, in the context of deep reinforcement learning, approximate solutions for solving POMDPs often directly encode past history with a recurrent neural network (RNN) and performs model-free planning on the latent representations of the RNN (Hausknecht & Stone, 2015; Zhu et al., 2018) . Recent work further reduces the burden on the RNN by training an internal generative model for approximate inference of a latent belief (Igl et al., 2018) . By doing end-to-end training on the neural architecture, the resulting models can solve complex POMDPs with visual inputs. Despite being simple and generic, these methods lose the ability to incorporate useful prior knowledge like the state formulation as both the planning is completely based on the latent states of RNN. Moreover, whenever these methods fail to perform well, it is difficult to analyze which part causes the failure.

In this work, we present the dual sequential Monte Carlo (DualSMC) model that aims to solve continuous POMDPs with complex unknown dynamics and high dimensional observations, while preserving the interpretability. In particular, DualSMC consists of two coupled inference processes: one for belief estimation over states, and the other for multi-modal density estimation over the optimal future trajectories. To connect the two parts, we feed top particles and the weighted mean estimate from the first SMC, i.e. an adversarial particle filter, as the belief representation to the second one. From there, the second SMC explicitly takes uncertainty into consideration and does multi-modal planning. Note that the learned dynamics is efficiently shared between these two parts. The overall pipeline of our algorithm is summarized in Figure 1 .

We evaluate our model on three continuous POMDP tasks: the floor-positioning task for explanatory purpose, the 3D light-dark navigation task simulated by DeepMind Lab (Beattie et al., 2016) with rich visual inputs, and a control task in a modified Mujoco (Todorov et al., 2012) environment. Our method consistently outperforms the baseline methods. 

2. PRELIMINARIES:

In this section, we provide a brief review of the key concepts for the background of this work.

Continuous POMDPs. A continuous POMDP can be specified as a 7-tuple (S, A, T , R, Ω, Z, γ), where S, A and Ω are underlying continuous state, action and observation spaces. We denote s t ∈ S as the underlying state at time t. When the robot takes an action a t ∈ A according to a policy π(a t |o ≤t , a <t ), the state changes to s t+1 with probability T (s t+1 |s t , a t ). The robot will then receive a new observation o t+1 ∼ Z(o t+1 |s t+1 ) and a reward r t ∼ R(s t , a t ). Assuming the episodes are of fixed length L, the robot's objective is then to maximize the expected cumulative future reward G = E τ ∼π L t=1 γ t-1 r t , where τ = (s 1 , a 1 , . . . , a L , s L+1 ) are trajectories induced by π, and 0 ≤ γ < 1 is the discount factor. In general, the optimal policy has to take the entire history into consideration, which can grow exponentially in time steps. Instead of remembering the entire history, classical methods often maintain a belief over possible states and gradually filter out the true state. Denote bel(s t ) p(s t |o ≤t , a <t ), then the belief is updated according to bel(s t+1 ) = η bel(s t )Z(o t+1 |s t+1 )T (s t+1 |s t , a t )ds t , where η is a normalization factor.

Particle filter (PF).:

A particle filter uses a set of particles {(s

(k) t , w (k) t )} K k=1 with K k=1 w (k) t

= 1, to approximate the belief distribution bel(s t ). With this approximation scheme, belief update reduces to updating individual particles, s

(k) t+1 ∼ T (•|s (k) t , a t ) and w (k) t+1 ∝ Z(o t+1 |s (k) t+1 )w (k)

t . Particle filter benefits from its flexibility to approximate any distribution. In practice, when the true dynamics T and Z are not known a priori, we can use parametrized functions T ψ (•) and Z θ (•) to approximate their corresponding counterparts. Despite its simplicity and efficiency, particle filter can suffer from the particle degeneracy problem that most of the probability mass concentrates on a few particles. A well-known solution is particle resampling, which bootstraps new particles of equal weights from the old ones. Recent advances in particle filters (Karkus et al., 2018b; Jonschkowski et al., 2018) adopt neural networks as the parametrized transition and observation functions and make the filtering process differentiable. By doing so, particle filters can be more easily applied to problems that have complex and high-dimensional observations like images. While previous work focuses on belief space filtering, we take one step further and propose a planning method.

Sequential Monte Carlo planning (SMCP). The task of planning can also be regarded as an inference problem, provided that the likelihood of any future trajectory is proportional to its expected cumulative rewards. This idea is connected to the control as inference framework (Todorov, 2008; Toussaint, 2009; Kappen et al., 2012) , where control problems are solved by forming a probabilistic graphical model. Specifically, if we denote O t as the optimality variable and define p(O t = 1) ∝ exp(R(s t , a t )) as the probability of time step t being optimal, the optimal plan then corresponds to the maximum a posterior estimate conditioned on the optimality of all future steps. We can solve this inference problem again with an SMC. For a complete derivation, please refer to Levine (2018) and Piche et al. (2018) . Notably, Piche et al. (2018) first uses the sequential Monte Carlo for planning (SMCP) in Markov decision processes. We extend the idea to partially observable domains.

3. METHOD:

In this section, we present DualSMC, which consists of two interrelated sequential Monte Carlo processes. We first introduce a more sampling-efficient adversarial particle filter for learning the belief representation, then an uncertainty-aware planning algorithm based on these representations. 

(k) old } K k=1

) transited from the previous step based on the updated weight and combine them with (K -K ) newly proposed particles ({s

(k) new } K k=K +1 ) * .

Naturally, P φ is trained by regressing the proposed state to the true state.

For successful POMDPs planning, the state estimation particle filter had better be really sampling efficient and provide a belief representation based on the most plausible states for the downstream planner. We observe that P φ and Z θ are opposite yet dependent on each other. Following the intuition that leveraging this adversarial relationship would enhance both parts and thus help to narrow down the belief space of interest, we propose the adversarial particle filter. In particular, we train Z θ to differentiate the true state from all particle states and train P φ to fool Z θ . Formally, denote p real as the real joint distribution over s, o, Z θ and P φ play the following two-player minimax game with function F (Z θ , P φ ):

min φ max θ F (Z θ , P φ ) = E s,o∼p real(•) [ log Z θ (o|s) + E s∼s (k) old log(1 -Z θ (o|s)) + E P ∼N (0,I) log(1 -Z θ (o|P φ (o, P ))) ] .

(1)

3.2. DUALSMC PLANNING WITH BELIEF REPRESENTATIONS:

A straightforward solution to POMDP planning is to train the planning module separately from the filtering module. At inference time, plans are made based on sampled individual particles from the state filter. We call this planning algorithm particle-independent SMC planning (PI-SMCP) and use it as a baseline method. More details about PI-SMCP can be found in Appendix A. Nevertheless, PI-SMCP does not perceive the state uncertainty. By contrast, the proposed DualSMC explicitly considers the belief distribution by planning directly on an approximated belief representation, i.e. a combination of the top candidates from the state filter as well as the weighted mean estimate.

Algorithm 1 Overall DualSMC filtering and planning method

1: {s (k) 1 ∼ Priori(s 1 )} K k=1 , {w (k) 1 = 1} K k=1

Define initial filtering particles 2: for t = 1 : L do At each control time step 3:

{w (k) t ∝ w (k) t-1 • Z θ (s (k) t , o t )} K k=1 Update particle weights 4: bel t = k w (k) t s (k) t

Compute mean belief state 5:

{s (m) t , w(m) t } M m=1 = Top-M({s (k) t , w (k) t } K k=1 ), w.r.t.{w (k) t } K k=1

Take top-M particles 6:

a t = DualSMC-P(bel t , {s (m) t , w(m) t } M m=1 ; π ρ , Q ω ) Perform planning (Alg 2) 7: o t+1 , r t ∼ p env (a t ) Update the environment 8: {s (k) t } K k=1 ∼ Multinomial({s (k) t } K k=1 ), w.r.t.{w (k) t } K k=1

Resample particle states 9:

{s (k) t ∼ P φ (o t )} K k=K +1 , {w (k) t = 1} K k=1

Propose new particles 10:

{s (k) t+1 ∼ T ψ (s (k) t , a t )} K k=1

Predict next-step particles Sample a batch from the buffer and update parameters (ρ, ω, θ, ψ, φ) Train all modules 13: end for We present the details of DualSMC in Algorithm 1. At time step t, when a new observation comes, we first use Z θ to update the particle weights (line 3 Alg 1), and then perform the planning algorithm in Algorithm 2. We duplicate the top-M particles and the mean belief state N times as the root states of N planning trajectories (line 1-2 Alg 2). Different from the previous SMCP (Piche et al., 2018) method under full observations, the policy network π ρ perceives the belief representations and predicts an action based on the top-M particle states as well as the mean belief state (line 4 Alg Algorithm 2 DualSMC planning with filtered belief representations Input: mean belief state bel t , top-M filtering particles {s

(m) t , w(m) t } M m=1 , policy network π ρ , value network Q ω , planning start time t, planning horizon H 1: { w(m) t } M m=1 = Normalize({ w(m) t } M m=1 ) Normalize top-M filtering particle weights 2: {ŝ (m)(n) t = s(m) t } M m=1 , ŵ(n) t-1 = 1, bel (n) t = bel t N n=1

Duplicate initial states set N times 3: for i = t : t + H do At each planning time step 4:

a (n) i ∼ π ρ ({ŝ (m)(n) i } M m=1 , bel (n) i ) N n=1

Sample N actions 5:

ŝ(m)(n) i+1 , r (m)(n) i ∼ T ψ (ŝ (m)(n) i , a (n) i ) M,N m=1,n=1

Predict next-step states and rewards 6:

bel

(n) i+1 ∼ T ψ (bel (n) i , a (n) i ) N n=1

Predict next-step mean belief state 7:

ŵ(n) i ∝ ŵ(n) i-1 • exp m w(m) t A (m)(n) N n=1

Update weights with Equation 28:

x

(n) i = ({ŝ (m)(n) i+1 , ŝ(m)(n) i } M m=1 , bel (n) i+1 , a (n) i ) N n=1

Assemble planning trajectories 9:

x

(n) t:i } N n=1 ∼ Multinomial({x (n) t:i } N n=1 ), w.r.t.{ ŵ(n) i N n=1

Resample trajectories 10: i>t by re-using T ψ † (line 6 Alg 2). We update the planning weight of each planning trajectory by summarizing the advantages of each state using the initial M belief weights (line 7 Alg 2). Here, we introduce an alternative advantage formulation that is equivalent to the one used in Piche et al. ( 2018)

ŵ(n) i = 1 N n=1

EQUATION

where TD

EQUATION

i-1 ) and r

(m)(n) i-1

are set to 0. We emphasize that our formulation is much simpler. Because our formulation only requires a learned Q function and more importantly, it prevents us from estimating the expectation of the value function V . We leave the full derivation in Appendix B.2.

At the end of each planning time step, we apply the sequential importance resampling (SIR) over N planning trajectories (line 9-10 Alg 2). When the planning horizon is reached, where i = t + H, we sample one planning trajectory and feed its first action to the environment (line 7 Alg 1). We then go back to the filtering part and update the belief representations by resampling, proposing, and predicting the next-step particle states (line 8-10 Alg 1). Lastly, we train all modules of DualSMC, including the policy network, the critic network as well as the adversarial particle filtering networks (line 11-12 Alg 1). We add details of each network component to Appendix C.

4. EXPERIMENT:



4.1. FLOOR POSITIONING:

A robot localizes itself and approaches a target region on a 2D plane. It knows the distances to the nearest walls: o t = (d x-, d x+ , d y-, d y+ ) t , but does not know the world coordinates (s x , s y ). It starts from a random location and is headed to different regions according to different floors. To reach the correct target, the robot has to learn to reduce the uncertainty about s y . The action is defined as a = (∆s x , ∆s y ) with a maximum magnitude of 0.05. Only at training time, a reward of 100 is given at the end of each episode if the robot reaches the correct target region. Implementations details of DualSMC can be found in Appendix D.1.

Qualitative results. Figure 2 (a) and 2(b) are results by applying the standard SMCP (Piche et al., 2018) to the top-1 particle state. We use different particle filters (PF) for these two baseline models. We use a regressive PF ‡ for the first one and an adversarial PF for the second one. As shown by the blue crosses in Figure 2(a) , training the proposer with the mean squared loss is equivalent to regressing the proposed particles to the mean values of the multi-modal state distributions under partial observations. Thus, the robot cannot make reasonable decisions. By contrast, as shown in Figure 2 (b), training the PF adversarially leads the proposed particles more akin to plausible states. The robot learns an interesting policy: moving right wherever the starting point is, and then bouncing back at the right wall. However, this policy is clearly not optimal, as it does not consider the uncertainty of the belief state. Figure 2 (c) shows the results by DualSMC. We have three findings. First, the robot learns a policy to localize itself quickly and then approach the target area with converged belief states, so that the robot can finish the task successfully with fewer steps. Second, the robot learns the multi-modal distributions of the planning trajectories and can move up or down with different probabilities and advantage values. Third, the observation model works well. Once the robot steps across the blue line, the belief states quickly converge to the actual values. ‡ The term regressive here indicates that the proposer of the particle filter takes the mean square error between the proposed states and the true states as the training objective function. We may also use the kernel density estimation as an alternative objective function.

Quantitative comparisons. 2018)). DualSMC models with regressive proposers are even worse than one without any proposers, illustrating that the quality of the belief representations is important; erroneous state estimations will harm the final planning results.

Is the DualSMC filter better than the traditional PF? Given partial observations, an ideal filter should derive a complete distribution of possible states instead of point estimation. Figure 3 (a) compares the averaged RMSE between the filtered results of different models and the true states. The adversarial PF performs best, while the PF with the regressive proposer performs even worse than that without a proposer. A natural question arises: as the filtering error is also related to different moving trajectories of different models, can we eliminate this interference? For Figure 3 (b), we train different filters without a planner. All filters follow the same expert trajectories. We can see that the adversarial PF still achieves the best performance. Figure 3 (c) explores the effect of different numbers of filtering particles. Using too few particles will deteriorate the filtering performance. Is the DualSMC planner aware of state uncertainty? In fully observable scenarios, we suppress the filtering part of DualSMC and assume DualSMC plans upon a converged belief on the true state (s x , s y ). The DualSMC planner makes different decisions to walk straight to the target region (see Figure 4 ). It performs equally well to the standard SMCP, with an average of 21.3 steps (v.s. 20.7 steps for SMCP) and a 100.0% success rate. We may conclude that DualSMC does not provide policies by remembering them, but by perceiving the distribution of the belief state. We may also conclude that DualSMC trained under POMDPs generalizes well to similar tasks with less uncertainty. 

4.2. 3D LIGHT-DARK NAVIGATION:

We extend the 2D light-dark navigation task (Platt Jr et al., 2010) to a visually rich environment simulated by DeepMind Lab (Beattie et al., 2016) . At each episode, the robot is placed randomly and uniformly on one of the 4 platforms at the bottom. Then, the robot's goal is to navigate toward the central cave (marked in orange) while avoiding any of the 4 traps (marked by crosses). The maze is divided into upper and lower parts. Within the lower part, the robot travels in darkness, receives noisy visual input of a limited range (up to a fixed depth), and therefore suffers from high state uncertainty.

When the robot gets to the upper part (the blue area), it has a clear view of the entire maze. We place decals as visual hints on the top walls of the maze to help the robot figure out its location. However, it has to be very close to the upper walls to see clearly what these decals are. The robot receives a positive reward of 100 when it reaches the goal and a negative reward of -100 when in a trap. At each time step, the robot's observation includes a 64 × 64 RGB image, its current velocity, and its orientation. Since the moving command in DeepMind Lab is binary (forward/backward), we force the robot to move forward and only let it control its orientation, which we make continuous. The experiment results are summarized in Table 2 . By considering the uncertainty, DualSMC methods outperform other baselines and are the only methods that learned to go up and figure out its location first before going directly towards the goal (see Figure 5 ). However, we notice that DualSMC with the adversarial proposer performs slightly worse than with a proposer trained with density loss. We further validate our model on a continuous control task with partial observation, i.e. a modified Reacher environment from OpenAI Gym (Brockman et al., 2016) . The original observation of Reacher is a 11-D vector including (cos θ 1 , cos θ 2 , sin θ 1 sin θ2, g x , g y , ω 1 , ω 2 , r x , r y , r z ), where the first 4 dimensions are cos/sin values of the two joint angles θ 1 , θ 2 , g x , g y the goal position, ω 1 , ω 2 the angular velocities and r x , r y , r z the relative distance from the end-effector to the goal. We remove g x , g y , r x , r y , r z from the original observation and include a single scalar r = ||(r x , r y , r z )|| 2 + r , where r ∼ N (0, 0.01) is a small noise (notice that r is usually on the scale of 0.1). The resulting observation is therefore a 7-D vector. The robot has to simultaneously locate the goal and reach it. We provide a visualization of one sample run under DualSMC with the adversarial filter in Figure 6 . As expected, initially the proposed particles roughly are in a half-cycle and as time goes on, the particles gradually concentrate around the true goal. Since the final performance of various methods is similar after long enough time of training, we provide the smoothed training curve of these methods in Figure 7 to showcase the advantage of DualSMC. We truncate the results up to 5,000 episodes since no obvious change in performance is observed from thereon. As we can see, DualSMC methods not only achieve similar asymptotic performance as the SMCP method with full observation but also learn faster to solve the task than baseline methods.

5. RELATED WORK:

Embedding useful algorithmic prior to a reinforcement learning model has been a recent trend because pure model-free approaches can suffer from unstable training and long convergence issues, which are exaggerated in partially observable domains. QMDP-Net (Karkus et al., 2017) extends the idea from value iteration network (VIN) (Tamar et al., 2016 ) by representing the bellman update in value iteration as convolution and pooling operations in a neural network. A similar idea has been extended to more visually rich domain (Karkus et al., 2018a; Shankar et al., 2016) . However, the underlying state and action space still remain simple because of the explicit value iteration procedure during the planning step. The idea of connecting a particle filter with a policy has been addressed in Coquelin et al. (2009) . The performance of their policy is only evaluated on the mean particle state and they use finite-difference method to update the policy. 

6. CONCLUSION:

In this paper, we provided a new method named DualSMC to solve continuous POMDPs, which has three advantages. First, it learns plausible belief representations for high-dimensional POMDPs with an adversarial particle filter. Second, it plans future actions by considering the distributions of the learned belief representations. The filter module and the planning module are inter-dependent and jointly trained. Third, DualSMC combines the richness of neural networks as well as the interpretability of classical sequential Monte Carlo methods. We empirically validated the effectiveness of DualSMC on different tasks including visual navigation and control.

A PARTICLE-INDEPENDENT SEQUENTIAL MONTE CARLO PLANNING:

A straightforward baseline method of our planning approach is to compute the future policies of each particle state independently. Similar to DualSMC planning, we initially sample M filtering particles according to their particle weights. Differently, we roll out N future action/state trajectories for each particle independently, following the probabilistic SMCP method (Piche et al., 2018) . In the end, there are M × N future planning trajectories, and we sample one of them according to the importance weights. It is equivalent to the standard SMCP when M = 1. We refer this planning algorithm as particle-independent SMC planning (PI-SMCP) and use it as an alternative of Algorithm 2 in line 6 Algorithm 1. Details of PI-SMCP are shown in Algorithm 3. We train the PI-SMCP module separately from the filtering network. For the policy network π ρ , we perform the standard soft actor-critic (SAC) (Haarnoja et al., 2018 ) on true states. Therefore, one disadvantage of PI-SMCP is that the SAC policy network cannot perceive the uncertainty of the belief representations.

Algorithm 3 Particle-independent SMC planning with filtered belief representations

EQUATION

t = w(m) t } M,N m=1,n=1

Define M × N initial states for planning

2: { ŵ(n) t } M ×N n=1 = Normalize({ w(n) t } M ×N n=1 )

Normalize weights of planning trajectories 3: for i = t : t + H do At each planning time step 4:

EQUATION

EQUATION

i )} M ×N n=1

Predict next-step states and rewards 6:

EQUATION

i , ŝ(n) i+1 ))} M ×N n=1

Update weights, see Eq 2 for A(•) 

EQUATION

i , ŝ(n) i )} M ×N n=1 Assemble planning trajectories 8: {x (n) t:i } M ×N n=1 ∼ Multinomial({x (n) t:i } M ×N n=1 ), w.r.t.{ ŵ(n) i+1 } M ×N n=1 Resample trajectories 9: { ŵ(n) i+1 = 1} M ×N

B THEORETICAL JUSTIFICATIONS:

In this section we provide theoretical justifications for our algorithms.

B.1 ANALYSIS OF THE POMDPS FILTERING STEP.

In the following, we use X n a.s --→ X to denote the sequence of random variables X n converge to X almost surely. We show if we do not use Top-M selection step, the particle filtering estimator is asymptotically consistent. We remark that Top-M is used for reducing the variance and it may increase the biased and thus M is a hyper-parameter to balance the bias-variance trade-off. 

C NETWORK DETAILS:

Network details of the DualSMC model for the two planning tasks are summarized in Table 3 .

D IMPLEMENTATION DETAILS D.1 FLOOR POSITIONING:

We use 100 particles to train the filtering module of DualSMC. Based on the top-3 particles, we roll out 10 steps with 30 planning trajectories. We use Adam optimizer with a 10 -3 learning rate. At test time, we increase the planning horizon to 20 and the number of filtering particles to 300. All models are trained for 10,000 episodes with a 64 batch size. All results are averaged over 1,000 episodes. The maximum number of steps is 200.

D.2 3D LIGHT-DARK:

We use 100 particles to train the filtering module of DualSMC. Based on the top-3 particles, we roll out 5 steps with 10 planning trajectories. We use Adam optimizer with a 0.0003 learning rate. At test 

D.3 MODIFIED REACHER:

We use 100 particles to train the filtering module of DualSMC. Based on the top-3 particles, we roll out 1 step (since no multi-modal planning is necessary here) with 10 planning trajectories. We use Adam optimizer with a 0.0003 learning rate. All models are trained for 20,000 episodes with a 256 batch size. As in the original Reacher environment, the maximum number of steps is 50.

:

† Like QMDP (Littman et al., 1995), DualSMC assumes the uncertainty disappears on the next time step and performs model-based planning.

Figures and Tables:

Figure 1: The pipeline of DualSMC. The planner and filter are linked via belief representatives.

Figure 2: The left column shows the actual trajectories of the robot, and other columns are its planning trajectories at different times. The robot can better localize itself stepping across dashed blue lines.

Figure 3: Averaged filtering error as a function of the number of robot steps for floor positioning.

Figure 4: The DualSMC planner trained under POMDPs generates different polices according to the uncertainty of belief state. For (b), we assign true state values to the top-M particles before planning.

Figure 5: DualSMC learns to go up to the wall of decals first before turning back to the goal. Note that the robot figures out its location at t = 30 and where to start turning back at t = 60.

This might be caused by the unstability of the adversarial training. For simplicity, we use the naïve adversarial training method as in the original generative adversarial networks (Goodfellow et al., 2014), which may easily suffer from mode collapse, leading to particle degeneracy in our case. One may potentially improve DualSMC with modern techniques, i.e. Wasserstein distance (Arjovsky et al., 2017), the gradient penalty (Gulrajani et al., 2017), and the spectral normalization (Miyato et al., 2018), to stabilize training or guarantee the existence of a unique Nash equilibrium.

Figure 6: The modified Reacher environment and the goal estimation of DualSMC v.s. time.

Figure 7: Smoothed training curves on modified Reacher. See experiment details in Appendix D.3.

Select a t = first action of x (n) t:t+H , where n ∼ Uniform(1, . . . , M × N ) Return an action

3.1 ADVERSARIAL PARTICLE FILTERING Similar to the architecture in Jonschkowski et al. (2018), our differentiable PF contains three neural modules, an observation model Z θ (o t |s new |o t , P ) for suggesting new probable particles, and a stochastic transition model T ψ (s t |s t-1 , a t-1 , T ) that mimics the environment dynamics. Here, P and T are Gaussian noise for stochasticity. At time t, we re-sample K particles ({s

Table content: <table><tr><td>(k) t ) that weights each particle given the current observation,</td></tr><tr><td>a proposer P φ (s (k)</td></tr></table>

Floor positioning results averaged over 1,000 test episodes, including the success rate, the mean value and the standard deviation of numbers of steps.

Table content: <table><tr><td>Method</td><td colspan="3">Success rate # Steps Std</td></tr><tr><td>LSTM filter + SMCP</td><td>23.5%</td><td>149.1</td><td>81.2</td></tr><tr><td>DVRL (Igl et al., 2018)</td><td>38.3%</td><td>162.4</td><td>19.9</td></tr><tr><td>Regressive PF (top-1) + SMCP</td><td>25.0%</td><td>107.9</td><td>69.8</td></tr><tr><td>Adversarial PF (top-1) + SMCP</td><td>95.0%</td><td>73.3</td><td>44.0</td></tr><tr><td>DualSMC w/o proposer</td><td>78.9%</td><td>64.9</td><td>62.2</td></tr><tr><td>DualSMC with regressive PF (MSE loss)</td><td>45.7%</td><td>121.6</td><td>69.2</td></tr><tr><td>DualSMC with regressive PF (density loss)</td><td>54.0%</td><td>109.2</td><td>81.8</td></tr><tr><td>DualSMC with adversarial PF</td><td>99.4%</td><td>36.8</td><td>15.1</td></tr></table>

Table 1 shows that, first, the DualSMC planning algorithm achieves the highest success rate using the least number of steps. Second, the adversarial PF outperforms other forms of PFs, as well as a deterministic LSTM model (LSTMs are previously used as strong baselines by Karkus et al. (2018b) and Jonschkowski et al. (

Table content: <table/>

Experimental results on the 3D navigation task. Results are averaged over 100 episodes after 2,000 episodes of training. Methods with a filter are trained (tested) with 100 (400) particles.

Table content: <table><tr><td>LSTM + SMCP</td><td>59% 85.40</td></tr><tr><td>Adversarial PF (top-1) + SMCP</td><td>58% 56.11</td></tr><tr><td>Adversarial PF (top-3) + PI-SMCP</td><td>64% 64.37</td></tr><tr><td>DualSMC with regressive PF (MSE loss)</td><td>93% 64.32</td></tr><tr><td colspan="2">DualSMC with regressive PF (density loss) 97% 73.86</td></tr><tr><td>DualSMC with adversarial PF</td><td>95% 65.13</td></tr><tr><td>4.3 MODIFIED REACHER</td><td/></tr></table>

Igl et al. (2018) introduced a variational lower bound to improve the original black-box learning of RNN while maintaining a set of particles to address uncertainty. But in their work, the latent particle states are not interpretable and known prior knowledge could not apply. Recently,Karkus et al. (2018b)  andJonschkowski et al. (2018) independently discovered methods to make the conventional particle filter differentiable in terms of neural networks, both showing that end-to-end training improves the filtering performance. Our work extends this idea to planning under uncertainty and we introduce an alternative adversarial proposing strategy to further improve the differentiable filter. Control as inference methods (Toussaint & Storkey, 2006; Rawlik et al., 2010; Ziebart, 2010; Levine & Koltun, 2013) regard policy search as a probabilistic inference problem by interpreting rewards as the log-likelihood of task fulfillment. It is, therefore, possible to adopt useful statistical tools to solve the control and planning problem by maximizing the likelihood of high reward future trajectory, or estimating the posterior distribution of actions conditioned on the optimality of future trajectories. Levine (2018) provided a comprehensive review of these methods. While previous work mainly focuses on simplified dynamics and simple state/action space, Piche et al. (2018) extended the idea to more complicated task domains by adopting the sequential Monte Carlo approach to select trajectories based on how promising they are, and learning the model-based knowledge necessary for planning in the meantime. All of the above algorithms assume perfect observability and plan in the state space, while our work generalizes the idea to belief space planning.

Table content: <table/>

Theorem 1. Assuming the transition kernel T is Feller and the the likelihood function Z is bounded, continuous and strictly positive, then we have .2 ALTERNATIVE SMCP UPDATE FORMULA We provide the probabilistic graphical model of planning in Figure 8. Similar to the derivation in Appendix A.4 of Piche et al. (2018), we have t |x 1:t-1 )p(x 1:t |O 1:T ) p(x 1:t-1 |O 1:T ) = w t-1 q(x t |x 1:t-1 ) p(O 1:T |x 1:t )p(x 1:t ) p(O 1:T |x 1:t-1 )p(x 1:t-1 ) = w t-1 q(x t |x 1:t-1 ) p(O 1:t-1 |x 1:t-1 )p(x 1:t )p(O t:T |x t ) p(O 1:t-2 |x 1:t-2 )p(x 1:t-1 )p(O t-1:T |x t-1 ) = w t-1 q(x t |x 1:t-1 ) p(x t |x t-1 )p(O t-1 |x t-1 ) exp(Q(s t , a ) -Q(s t-1 , a t-1 )) = w t-1 p(x t |x t-1 ) q(x t |x t-1 ) exp Q(s t , a t ) -Q(s t-1 , a t-1 ) + r t-1 = w t-1 p env (s t |s t-1 , a t-1 ) p model (s t |s t-1 , a t-1 ) exp Q(s t , a t ) -Q(s t-1 , a t-1 ) + r t-1 -log π θ (a t |s t ) O tis the observed optimality variable that depends on the latent state s t and action a t .

Table content: <table><tr><td>w t =</td><td colspan="2">p(x 1:t |O 1:T ) q(x 1:t )</td><td/><td/><td/></tr><tr><td>=</td><td colspan="2">p(x 1:t-1 |O 1:T ) q(x 1:t-1 )</td><td colspan="2">p(x t |x 1:t-1 , O 1:T ) q(x t |x 1:t-1 )</td><td/></tr><tr><td colspan="2">= w t-1</td><td colspan="2">p(x t |x 1:t-1 , O 1:T ) q(x t |x 1:t-1 )</td><td/><td/></tr><tr><td>=</td><td colspan="3">w t-1 q(x a 1 O 1</td><td>O 2 a 2</td><td/><td>O 3 a 3</td><td>. . .</td><td>O t a t</td></tr><tr><td/><td/><td/><td>s 1</td><td>s 2</td><td/><td>s 3</td><td>s t</td></tr><tr><td colspan="2">Figure 8:</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>lim K→∞</td><td>bel t</td><td>a.s --→ bel t .</td></tr><tr><td colspan="7">Proof of Theorem 1. Consider the sequence {s (k) t } L,K t=1,k=1 . Note the evolution of {s (k) t } L,K t=1,k=1</td></tr></table>

Architecture details of the learnable models. time, we increase the planning horizon to 15 and the number of filtering particles to 400. All models are trained for 2,000 episodes with a 128 batch size. All results are averaged over 100 episodes. The maximum number of steps is 200.

Table content: <table><tr><td colspan="2">Module Floor positioning</td><td>3D light-dark navigation</td></tr><tr><td>Z</td><td>3× fc: 256, 256, 16</td><td>Conv2d: (3, 3), 16, 2, 1, ReLU</td></tr><tr><td/><td/><td>Conv2d: (3, 3), 32, 2, 1, ReLU, MaxPool(2)</td></tr><tr><td/><td/><td>Conv2d: (3, 3), 64, 2, 1, ReLU, Dropout(0.2)</td></tr><tr><td/><td/><td>fc: 128, ReLU (*)</td></tr><tr><td/><td>2× lstm: 128, 128</td><td>concat: velocity and orientation</td></tr><tr><td/><td>3× fc: 256, 256, 1</td><td>5× fc: 128, 128, 128, 128, 1</td></tr><tr><td>P</td><td>3× fc: 256, 256, 64</td><td>same as Z up to (*); fc: 64</td></tr><tr><td/><td colspan="2">concat: z(64) ∼ Norm(0, 1) concat: orientation, z(64) ∼ Norm(0, 1)</td></tr><tr><td/><td>4× fc: 256, 256, 256, 2</td><td>3× fc: 128, 128, 2</td></tr><tr><td>T</td><td>4× fc: 256, 256, 256, 4</td><td>action noise: z(1) ∼ Norm(0, 1)</td></tr><tr><td/><td/><td>e = 3× fc(z): 128, 128, 1</td></tr><tr><td/><td/><td>concat: (state, action + e)</td></tr><tr><td/><td/><td>3×fc: 128, 128, 128, 5; add to state</td></tr><tr><td>Q</td><td>3× fc: 256, 256, 1</td><td>3× fc: 128, 128, 1</td></tr><tr><td>π</td><td>3× fc: 256, 256, 4</td><td>fc (on top-M): 64; fc (on mean belief): 64; concat;</td></tr><tr><td/><td/><td>2 × fc: 128, 1</td></tr></table>

Peer review (iclr2020):
This paper generalizes the differentiable particle filtering (DPF) to POMDP setting. Moreover, it introduces an extra differentiable SMC component as planner into the model, which together with the DPF will be learned together. The authors demonstrated the effectiveness of the proposed algorithm on 2D floor positioning, 3D navigation with image input, and Reacher in MuJoCo. Although the learning through the planner in an end-to-end fashion has been investigated in RL community extensively, e.g., [1, 2], extending it to POMDP is relative novel. However, there are several issues need to be addressed: 1, The optimization objective (1) seems come from nowhere. The motivation of such objective is very unclear. Why this objective is selected? Why not to adapt other alternative objectives, e.g., maximum log-likelihood? Without such justification, the learning target is not convincing. Meanwhile, whether is learned or not. If it is learned, what is the learning objective. 2, The authors claim that by twisting two SMCs, for planning and state prediction, respectively, the algorithm will lead to better performance. However, after twisting the planner into the learning, the proposed method is only applicable for on-policy setting, while the major advantage of model-based RL is for off-policy. This effect has not been discussed explicitly. I am not sure whether the benefit of the proposed algorithm comes from the coupling of two SMCs or on-policy learning. 3, The empirical experiments should be improved to be more convincing. One of the major contribution of this paper is using POMDP, rather than traditional MDP. To justify the benefits of the complicated model, a direct comparison between model-free (DQN) model-based (DeepMDP [4] and PlaNet [3]) algorithm upon MDP with same inputs as the inputs of the proposed model is necessary. Moreover, just one MuJoCo is not sufficient to demonstrate the advantages. Please evaluate the proposed algorithm on more environments in standard benchmarks, e.g., MuJoCo and Atari. Minor: The terminology *dual* has rigorous definition in math. Please use the terminology following the common knowledge. 1, Okada, Masashi, Luca Rigazio, and Takenobu Aoshima. *Path integral networks: End-to-end differentiable optimal control.* arXiv preprint arXiv:1706.09597 (2017). 2, Pereira, Marcus, et al. *Mpc-inspired neural network policies for sequential decision making.* arXiv preprint arXiv:1802.05803 (2018). 3, Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018). Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551. 4, Gelada, C., Kumar, S., Buckman, J., Nachum, O., & Bellemare, M. G. (2019). DeepMDP: Learning Continuous Latent Space Models for Representation Learning. arXiv preprint arXiv:1906.02736.