Discriminative Particle Filter Reinforcement Learning for Complex Partial observations
Authors: Xiao Ma
Peter Karkus
David Hsu
Wee Sun Lee
Nan Ye
Venue:  (2020)
Keywords: Reinforcement Learning
Partial Observability
Differentiable Particle Filtering

Abstract: Deep reinforcement learning is successful in decision making for sophisticated games, such as Atari, Go, etc. 
However, real-world decision making often requires reasoning with partial information extracted from complex visual observations. This paper presents  Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations. DPFRL encodes a differentiable particle filter in the neural network policy for explicit reasoning with partial observations over time. The particle filter maintains a belief using learned discriminative update, which is trained end-to-end for decision making. We show that using the discriminative update instead of standard generative models results in significantly improved performance, especially for tasks with complex visual observations, because they circumvent the difficulty of modeling complex observations that are irrelevant to decision making.
In addition, to extract features from the particle belief, we propose a new type of belief feature based on the moment generating function. DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this paper.  Further,  DPFRL performs well for visual navigation with real-world data in the Habitat environment.

Published as a conference paper at ICLR 2020
DISCRIMINATIVE PARTICLE FILTER REINFORCEMENT LEARNING FOR COMPLEX PARTIAL OBSERVATIONS
Xiao Ma', Peter Karkus', David Hsu' and Wee Sun Lee', Bee',,iu Nan Ye National Unviersity of Singapore, "he University of Queesland Ixiao-ma karkus, dyhsu, leews] (ecomp. nus edu.sg, nan ye@uq. edu. edu au
ABSTRACT
Deep reinforcement learning is successful in decision making for sophisticated games, such Atari, Go, etc. However, real-world decision making often requires reasoning with partial information extracted from complex visual observations. This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), new reinforcement learning framework for complex partial observations. DPFRL encodes differentiable particle filter in the neural network policy for explicit reasoning with partial observations over time. The particle filter maintains a belief using learned discriminative update, which trained end-to-end for decision making. We show that using the discriminative update instead of standard generative models results in significantly improved performance, especially for tasks with complex visual observations, because they circumvent the difficulty of modeling complex observations that are irrelevant to decision making. In addition, to extract features from the particle belief, we propose new type of belief feature based on the moment generating function. DPFRL outperforms sate--oftheearne POMDP RL models in Flickering Atari Games, existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this paper. Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment. The code is available online
INTRODUCTION
Deep Reinforcement Learning (DRL) has attracted significant interest, with applications ranging from game playing (Mnih et al. 2013 Silver et al. 2017) to robot control and visual navigation (Levine et al. 2016 Kahn et al. 20188 Savva al. 2019). However, natural real-world environments remain challenging for current DRL methods (Arulkumaran et al. (017). in part because they require (i) reasoning in partially observable environment and (ii) reasoning with complex observations, such as visually rich images. Consider, for example, a robot, navigating in an indoor environment, with camera for visual perception. To determine its own location and traversable path, the robot. must extract from image pixels relevant geometric features, which often coexist with irrelevant visual features, such as wall textures, shadows, etc. Further, the task is partially observable; single image at the current time does not provide sufficient features for localization, and the robot must integrate information from the history of visual inputs received.
The partially observable Markov decision process (POMDP) provides a principled general framework for decision making under partial observability. Solving POMDPs requires tracking sufficient statistic of the actioonobservation history, e.g., the posterior distribution of the states, called the belief.
Most POMDP reinforcement learning (RL.) methods summarize the history into a vector using recurrent neural network (RNN) (Hausknecht & Stone 2015, Zhu et al. 2018). RNNs are model-free generic function approximators. Without appropriate structural priors, they need large amounts of training data to learn to track complex belief well.
Model-based DRL methods aim to reduce the sample complexity by learning a model together with policy. In particular, to deal with partial observability, [gl al. (2018) recently proposed DVRL, which learns a generative observation model embedded into the policy through Bayes filter. Since

---

Published as a conference paper at ICLR 2020
b2 xf))
Sooiineddhe trining
DPFRL
(bb,)
Vib)
RL
J0g
Genorallio
Function
plo
ha)
h1
0: Function
Compariiltty
fate (Oe, he)
(a) DPFRL Overview
(b) Generative Model
(c) (Discriminative) Compatibility Model
Figure I: (a) DPFRL tracks a learned latent belicf with differentiable particle filter and learns policy conditioned on the partiele belief. (b) Generative models learn p(o he), the distribution of observations o given the latent state ht. The particle filter then evaluates the probability of an observed de to get a compatibility measure of of and he (c) The compatibility function directly predicts compatibility measure of o, and ht using a learned neural network fobe he).
the Bayes filter tracks the belief explicitly, DVRL performs much better than generic RNNs under partial observability. However, a Bayes filter normally assumes a generative observation model, that defines the probability p(o he) of receiving an observation = given the latent state he (Fig. [D). Learning this model can be very challenging since requires modeling all observation features, including features irrelevant for RL. When is an image, p(o ht) is a distribution over all possible images. This means, e.g., to navigate in a previously unseen environment, we need to learn the distribution of all possible environments with their visual appearance, lighting condition, etc. much harder task than learning to extract features relevant to navigation, e.g., the traversable space.
We introduce the Discriminative Particle Filter Reinforcement Learning (DPFRL), POMDP RL method that learns to explicitly track a belief over the latent state without a generative observation model, and make decisions based features of the belief (Fig. (Fig.TT). DPFRL approximates the belief by a set of weighted learnable latent particles {(ht w))) and it tracks this particle belief by a non- parametric Bayes filter algorithm, an importance weighted particle filter, encoded as a differentiable computational graph in the neural network architecture. The importance weighted particle filter applies discriminative update to the belief with an inoervationcoondiiioned transition model and discriminative oobtrvrbbevvaoon compatibility function (serving as the importance weights), both of which are learnable neural networks trained end-to and-to-end. By using these update functions instead of the transition and observation models of the standard particle filter, DPFRL sidesteps the difficulty of learning a generative observation model (Fig. [D). The model is discriminative in the sense that the compatibility function, (obs h2), as shown in Fig. DC. while playing an analogue role p(o, he). is not required to directly represent a normalized distribution over observations; and through end-to- end training it only needs to model observation features relevant for the RL task. Finally, summarize the particle belief for the policy, we introduce novel learnable features based on Geenentinnn Functions (MGFs) (Bulmer 1979). MGF features are computationally efficient and permutation invariant, and they can be directly optimized to provide useful higher-order moment information for learning policy. MGF features could be also used as learned features of any empirical distribution in applications beyond RL.
We evaluate DPFRL on a range of POMDP RL domains: continuous control task from Igl et al. (0188. Flickering Atari Games (Hausknecht & Stone 2013), Natural Flickering Atari Games, new domain with more complex observations that we introduce, and the Habitat visual navigation domain using real-world data (Savva et al.) (2019). DPFRL. outperforms state-of-thee-at POMDP RL methods in most cases. Results show that belief tracking with a particle filter is effective for handling partial observability, and the discriminative update and MGF-based belief features allow for complex observations.
2 RELATED WORK
Real-world decision-making problems are often formulated POMDPs. POMDPs are notori- ously hard to solve; in the worst case, they are computationally intractable (Papadimitriou & Tsit- siklis 1987). Approximate POMDP solvers have made dramatic progress in solving large-scale POMDPs Kurniawati et al. 2008). Particle filters have been widely adopted as a belief tracker for POMDP solvers (Silver & Veness 2010 Somani et al. 2013) having the flexibility to model complex and multi-modal distributions, unlike Gaussian and Kalman filters. However, predefined model and state representations required for these methods (see e.g. Bai al. 2015
2

---

Published as a conference paper at ICLR 2020
(AC u. MS
M
perside update
16 o0 M4
15
File
Figure 2: DPFRL Network. In DPFRL, latent particles [(it. = maintained by differentiable particle filter algorithm, which includes inseraatio-conndiionn transition function firans, discriminative compatibility function fols. and soft-resampling The policy and value function is conditioned on the belief, which is summarized by the mean particle ht, and m moment generating function features, M,
Given the advances in generative neural network models, various neural models have been proposed for belief tracking (Chung et al., 2015 Maddison et al.) 2017 Le et al. 2018 Naesseth al. 2018). DVRL (Igl et al. (0118) uses Vanational Sequential Monte- Carte-Carl method (Naesseth et al. 2018). similar the particle filter we use, for belief tracking in RL. This gives better belicf tracking capabilities, but as we demonstrate in our experiments, generative modeling is not robust in complex observation spaces with high dimensnnnn irrelevant observation. More powerful generative models, c.g., DRAW (Gregor et al. 2015), could be considered improve generative observation modeling; however, evaluating a complex generative model for each particle would significantly increase the computational cost and optimization difficulty.
Learning a robust latent representation and avoiding reconstructing observations are of great interest for RL (Oord et al., 2018 Guo et al. 2018 Hung et al.) 2018 Gregor et al., 2019 Gelada et al.) 2019). Discriminative NNNs have also been widely used for belief approximation in partially observable domains (Bakker 2002 Wierstra et al. 2007: Foerster al., 2016). The latent representation is
directly optimized for the policy pl ht)) that skips observation modeling. For example, Hausknecht
Stone (2015) and Zhu et al. (2018) tackle partially observable Flickering Atari Games by extending DQN (Mnih et al. 2013) with an LSTM memory. Our experiments demonstrate that the additional structure for belief tracking provided by particle filter can give improved performance in RL.
Embedding algorithms into neural networks to allow end-to-end discriminative training has gained attention recently. For belief tracking, the idea has been used in the differentiable histogram filter (Jon- schkowski & Brock 2016). Kalman filter (Haarnoja et al. 2016) and particle filter (Karkus et al. Dosssoonschkowski et al. 2018). Further, Karkuu et al. (2017) combined a learnable histogram filter with the Value Iteration Network (Tamar al.) 2016) and introduced a learnable POMDP planner, QMDP-net. However, these methods require predefined state representation and are limited to relatively small state spaces. Ma et al. (2019) integrated the particle filter with standard RNNs, e.g.. the LSTM, and introduced PF-RNNs for sequence prediction. We build on the work of Ma et al. (2019) and demonstrate its advantages for RL with complex partial observations, and extend it with MGF features for improved decision making from particle beliefs. Note that our framework is not specific PF-RNNs, and could be applied other differentiable particle filters as well.
3 DISCRIMINATIVE PARTICLE FILTER REINFORCEMENT LEARNING
We introduce DPFRL for reinforcement learning under partial and complex observations. The DPFRL architecture is shown in Fig. 2 It has two main components, discriminatively trained partiele filter that tracks a latent belief by, and an actor network that learns a policy p(a b,) given the helief b,.
3.1 PARTICLE FILTER FOR LATENT BELIEF TRACKING
the semantics of states s is typically defined explicitly. State variables may correspond the position of robot, configuration of obstacles, etc. In DPFRL, we do not require explicit specification of the state variables, but implicitly represent the state as a vector h of latent variables, that is, the semantics of the state variables are learned instead of being pre-specified. We use a fully differentiable particle filter algorithm to maintain a belief over h. More specifically, we approximate the belief with a set of weighted latent particles b, {(ht. w " where (hi) (=1 K latent states learned by poricy-oriented training, and (w]) represents the
Latent State Representation. In POMDPs
3

---

Published as a conference paper at ICLR 2020
corresponding weights. Each latent state hf stands for hypothesis in the belief; the set of latent particles provide provide an approximate representation for the belief.
Belief Update. In basic particle filter, there are two key steps to update particle belief { wi-1)== to a new particle belief {h₁, w upon receiving an observation o after ex- ecuting action dz.
~ p(h h))1,
w - npoo: h))uf-1 7 1/1.//1/p/ (p(o h))af
(2)
The first step, Eq. T takes the transition dynamics into account to update each particle. The second step, Eq. 2 takes the observation into account to reweigh the new partieles.
Our belief update has similar structure the standard particle filter, but replace the transition model and the observation model with richer functions to make the update more suitable for learning policy in partially observable domain. Specifically, the update equations are follows.
hig fuaas (b2-1, az, oe),
= 1//ooobbs (hi, oc)wr(1: 1) 1/E// (big 0z) 01) ((M) w(() ] = Soft-Resampling( ((1)), )
(4)
(5)
Below, we first explain the intuition behind the above updates and the roles of firans and fobs as compared to the standard transition and observation models. We then derive that above rules from an importance weighed particle filter in Sect. 3.2
inservation-condiiooned transition update. Eq. 3 takes form more general than that in Eq. 2 instead of using the transition dynamics p(h h;) to evolve particle, we use a more general observation-condiiione transition firans(h [h] h at, or). Incorporating the observation allows alle- viating the problem of sampling unlikely particles. In fact, if we take furens to be p(h fi-1 ae oz), then this allows to skip Eq. 2 and completely avoids sampling particles that are likely consid- ering a only, but unlikely considering both a, and og. Of course, in RL we do not have access to p(h hi- at oz), and instead firans is learned. In our implementation, network first extracts features from o,, they are fed to a gated function following the PF-GRU of Ma et al. 2019), which outputs the mean and variance of normal distribution. Details are in the Appendix.
Importance weighting via a compatibility function. Eis a relaxed version of Eq. instead of using the observation model p(o; h)) to adjust the partiele weights based on their compatibility with the observation, we use a general non-negative compatibility function fobe (hi, or). If the compatibility function is required to satisfy the normalization constraint that E. (h. o) is a constant for all h, then it is equivalent to a conditional distribution of o given h. We do not require this, and thus the update loses the probabilistic interpretation in Eq. 2 However, eliminating the need for the normalization constraint allows the compatibility function to be efficiently trained, as we can avoid computing the normalization constant. addition, since the observation has already been incorporated in Eq. we actually expect that the weights need to be adjusted in way different from the standard particle filter. In our implementation, fora (hi, or) is a neural network with a single fully connected layer that takes in concatenation of h; and features extracted from or. The output of the network is interpreted as the log of fobs: and for numerical stability we perform the weight updates of Eq. in the log-space well. Note that more complex network architectures could improve the capability of fohs. which we leave to future work.
Soft-resampling To avoid particle degeneracy, i.e., most of the particles having a near-zero weight, particle filters typically resample particles. We adopt the soft-resampling strategy of Karkus et al. (2018); Ma et al. (2019), that provides approximate gradients for the non-differentiable resampling step. Instead of sampling from (i) = wi, we sample particles {hr Ji=1 from softened proposal K
distribution qii) = aw, + (1 a)1/K, where an trade-off parameter. The new weights are
wf derived using importance sampling: w" own (1-a)1/KK* We can have the final particle belief as {(M', w Soft-Resampling [(hj w()) ). As result, fobs be optimized with global belief information and model shared useful features across multiple time steps.
Another related concern is that the partiele distribution may collapse to particles with the same latent state. This can be avoided by ensuring that the stochastic transition function ftrans has a non-zen zero variance, e.g., by adding small constant the learned variance.
4

---

Published as a conference paper at ICLR 2020
End-to-end training. In DPFRL the observation--onnditonee transition function firass and the compatibility function fobs are learned. Instead of training for a modeling objective, they are trained end-to-end for the final RL objective, backpropagating gradients through the belief-conditional policy p(a by) and the update steps of the particle filter algorithm, Eq. 3
3.2 CONNECTION TO IMPORTANCE WEIGHTED PARTICLE FILTER
Our belief update can be motivated from the following importance weighted particle filter. Learning directly p(h' h, a, o) is generally difficult, but if we have a distribution q(h' h, a, o) that easy to lear, then use importance sampling to update a particle belief.
h4 qhh a1, 0z):
(6)
wf = (h), 14-1, ar 0, o "l = -=-1/>m 1/2 11 or)ui-1
(7)
where / = p/q is the importance weight.
Consider the case that q(h' | h, a, o) is the conditional distribution of joint distribution q(N', , h, a, o) of the form p(h' h, a)q(( h'). That is, p and share the same transition dynamics p(h' h a).
Then the importance weight is function of h' and o only, because
p(h' h, a. q(h' h,, a,, p(h' h, a)g(( ht) g(o hr)
p(h' | h, a)po h²)
pío hn)
f(h', h, a, o)
This simpler form is exactly the form that we used for fobs in our belief update.
3.3 DISCRIMINATIVE vs. GENERATIVE MODELING
We expect the discriminative compatibility function to be more effective than generative model for the following reasons. A generative model aims to approximate p(o h) by learning a function that takes h input and outputs parameterized distribution over When is, e.g., an image, this requires approximations, e.g., using pixel-wise Gaussians with learned mean and variance. This model is also agnostic to the RL task and considers all observation features equally, including features irrelevant for filtering and decision making.
In contrast, fabs takes and h inputs, and estimates the compatibility of o and h for particle filtering directly. This function avoids forming a parametric distribution over o, and the function can be easier to learn. The same functional form is used for the energy-function of energy- based models (LeCun et al.) (206)) and in contrastive predictive coding (Oord et al., 2018), with similar benefits. For example, Jobss may learn unnormalized likelihoods that are only proportionate to p(o h) up to o-dependent value, because after the normalization in Eq. 4 they would give the same belief update as the normalized p(o h). Further, because Sobe trained for the final RL objective instead of modeling objective, it may learn a compatibility function that is useful for decision making, but that does not model all observation features and has no proper probabilistic interpretation.
While the task-oriented training of discriminative models may improve policy performance for the reasons above, it cannot take advantage of an auxiliary learning signal like the reconstruction objective of generative model. An interesting line of future work may combine generative models with compatibility function to simultaneously benefit from both formulations.
3.4 CELLIT-CCNNIIIOONA ACTOR NETWORK
Conditioning policy directly onto particle belief is non-trivial. To feed it to the networks, we need to summarize into a single vector.
We introduce novel feature extraction method for empirical distributions based on Moment- Generating Functions (MGFs). The MGF an n-dimensional random variable X given by Mx(v)= =Ee X E RR. In statistics, MGF is an alternative specification of its probability dis- tribution (uulmer 1979). Since particle belief b, is an empirical distribution, the moment generating function of 6, can be denoted as Mc. (v) Σ wie b. A more detailed background on MGFs is in Appendix A.2
5

---

Published as a conference paper at ICLR 2020
In DPFRL, we use the values of the MGF at learned locations v as the feature vector of the MGF. The j-th MGF feature is given by M3, (v"). For a clean notation, we use M/ in place of M2, (v'). We use [he M, as features for belief by, where h2 = Σ wihh is the mean particle. The mean particle he, as the first-order moment, and m additional MGF features, give a summary of the belief characteristics. The number of MGF features, m, controls how much additional information we extract from the belief. We empirically study the influence of MGF features in ablation studies.
Compared Ma al. (019)) that uses the mean the belief estimate, MGF features provide additional features from the empirical distribution. Compared to DVRL (Tg et al. 2018) that treats the Monte-Carle samples as sequence and merges them by an RNN, MGF features are permutation- invariant, computationally efficient and easy to optimize, especially when the partiele set is large.
Given the features [h,, M for by, we compute the policy p(a b.) with a policy network π T(be). We trained with an actor-critic RL algorithm, A2C (Mnih et al. 2016), where value network (b,,) is introduced to assist learning. We use small fully-connected networks for π (Tbb) and V V(b)) that share the same input b,.
4 EXPERIMENTS
We evaluate DPFRL in a range of POMDP RL domains with increasing belief tracking and observation modeling complexity. We first use benchmark domains from the literature, Mountain Hike, and 10 different Flickering Atari Games. We then introduce a new, more challenging domain, Natural Flickering Atari Games, that uses a random video stream as the background. Finally apply DPFRL to a challenging visual navigation domain with RGB-D observations rendered from real-world data.
We compare DPFRI with GRU network, state-of-the-art POMDP RL method, DVRL., and ablations of the DPFRL architecture. As brief conclusion, we show that: 1) DPFRL significantly outperforms GRU in most cases because of its explicit structure for belief tracking: 2) DPFRL outperforms the srate-of-the--re art DVRL in most cases even with simple observations, and its benefit increases dramatically with more complex observations because of DPFRL's diseriminative update; 3) MGF features are more effective for summarizing the latent particle belief than alternatives.
4.1 EXPERIMENTAL SETUP
We train DPFRL and baselines with the same A2C algorithm, and use a similar network architecture and hyperparameters as the orig- inal DVRL implementation. DPFRL and DVRL differ in the particle belief update structure, but they use the same latent particle size dim(h) and the same number of particles K as in the DVRL paper (dim(h) = 128 and K = 30 for Mountain Hike, dim(h) = 256 and K = 15 for Atari games and visual navigation). The effect of the number of particles is discussed in Sect. 4.5 We train all models for the same number of iterations using the RMSProp optimizer TTele man & Hinton 2012). Learning rates and gradient clipping values are chosen based on a search the BeamRider Atari game inde- pendently for each model. Further details are in the Appendix. We have not performed additional searches for the network architecture and other prrper-prameters. nor tried other RI. algorithm, such as PPO (Schulman et al. 2017). which may all improve our results.
Figure 3: Mountain Hike Task. An agent navigates on the map from the start position (white dot) to the goal (green dot with the shaded area as the threshold). Par- tial observation is introduced by
Gaussian noise and appended with long noise vector of length I. The reward r(2, y) for position (z, y) is given by the heat map.
All reported results are averages over 3 different random seeds. We plot rewards accumulated in an episode, same as DVRL (Igl et al. 2018). The curves smoothed over time and averaged over parallel environment executions.
4.2 MOUNTAIN HIKE
Mountain Hike was introduced by Igl et al. (2018) to demonstrate the benefit of belief tracking for POMDP RL. It is a continuous control problem where an agent navigates fixed 20 20 map.

---

Published as a conference paper at ICLR 2020
-10
-39
-30
175
-00
-80
D4:
DV%
10h
150
-15
-89
-00
Effect of varying I
1=
50
[ 100
Figure 4: Results for Mountain Hike where useful observations are concatenated with a noise vector of length I.
005300
Flickering Atari Games
Natural Flickering Atari Games
Figure 5: Partially Observable Atari Games. In Flickering Atari Games frames randomly dropped and replaced with blank frame. In Natural Flickering Atari Games the background is replaced with random vidco stream and the Atari components of the image are randomly dropped.
In the original task, partial observability is introduced by disturbing the agent observation with an additive Gaussian noise. To illustrate the effect of observation complexity in natural environments, concatenate the original observation vector with random noise vector. The complexity of the optimal policy remains unchanged, but the relevant information is now coupled with irrelevant observation features. More specifically, the state space and action space Mountain Hike are defined as S = AA = IR2 where with an additive Gaussian noise: 81+1 = 8, + at + Ca. where Ca (00 0.25). The observation space is O IR 2+h, where is predefined constant and [ = O corresponds to the original setting. Observations are O2 of of]] where of 8t + cs ~ (O, 1), and of E RR' is sampled from uniform distribution 24( (4(-10. 10). The reward for each step is given by Ti r( Mt) 0.01|/a, where r 3/3) is shown in Fig. Episodes end after 75 steps. We train models for different settings of the noise vector length I, from i 0 to 100. Results are shown in Fig. 4 We observe that DPFRI. learns faster than the DVRL and GRU in all cases, including the original setting 1 O. Importantly, the noise vector length increases, the performance of DVRL and GRU degrades, while DPFRL is unaffected. This demonstrates the ability of DPFRL to track a latent belief without having explicitly model complex observations.
[ra, Ve] and a; = [8xr,, 5y.]. Transitions of the agent are stochastic
4.3 ATARI GAMES WITH PARTIAL OBSERVABILITY
Atari games are one of the most popular benchmark domains for RL methods (Mnih et al. 2013) Their partially observable variants. Flickering Atari Games, have been used to benehmark POMDP RL methods (Hausknecht & Stone, 2015 Zhu et al. [2018] Igl et al. 2018). Here image observations single frames randomly replaced by blank frame with a probability of 0.5. The flickering obser- vations introduce simple form of partial observability. Another variant, Natural Atari Games (Zhang al. 2018), replaces the simple black background of the frames of an Atari game with randomly sampled video stream. This modification brings the Atari domain one step closer to the visually rich real-world, in that the relevant information is now encoded in complex observations. As shown by Zhang et al. (2018), this poses significant challenge for RL.
We propose a new RL domain, Natural Flickering Atari Games, that involves both challenges: partial observability simulated by flickering frames, and complex observations simulated by random background videos. The background videos increase observation complexity without affecting the decision making complexity, making this suitable domain for evaluating RL methods with complex observations. We sample the background video from the ILSVRC dataset (Russakovsky et al. 2015). Examples for the BeamRider game shown in Fig. Details in Appendix
7

---

Published as a conference paper at ICLR 2020
Table Results Flickering Atari Games and Natural Flickering Atari Games
Flickoring Atari Games
Natural Flickering Atari Games
DPFRL
DVRL era
GRU [el
DPFRL
DVRL
GRU
Pong
05.40:40.76
18.17422.67
6.33-43.03
15.68-1.99
09.78:40.006
262-00.93
ChopperCommand
8,086=159.1
6,602-4449
5,150+ 488.1
1,566=66.03
1,068=1283
1.418=55008
MsPacian
545.3
2,221
2,312-4358
2,106=1223.9
1,358 155.4
1,833=45.45
Centipede
4,849=291.4
1,240+1116
4,395=224
4.164=23.0
3.154=335.9
1.679=116.4
BeamRider
3,940=107,4
1,663:1:13
1,801 155
682.91:7.422
437.3
$25.6=225.25
Frusthite
297.011..85
254.010.455
260.2:14.60
252.4:16.48
254.3-19.20
Bowling
31.8910.34
20.23
30.0:10.18
29.45-10.13
24.80=0.311
27.13:10.41
loeHockey
-4.06-+0.02
4.88
-7.10=0..0
-6.08-4-0188
-8.79-40012
-5,30:20.66
DDunk
-11.25:+:1255
-5.95::1.25
-15 15.8340.34
-15.36:::0.96
-17,62++0.16
-14.31+0,37
Asteroids
1,948=202.66
1,539:773
1,545451
1,489 15.76
1,406 132.3
1,67==571.5
We evaluate DPFRL for both Flickering Atari Games and Natural Flickering Atari Games. We use the same set of games as [gl et al. (2018). To ensure fair comparison, we take the GRU and DVRL results from the paper for Flickering Atari Games, use the same training iterations as Igl et al. (2018), and we use the official DVRL open source code to train for Natural Flickering Atari Games. Results are summarized in Table T We highlight the best performance in bold where the difference is statistically significant (p- 0.05). Detailed training curves are in Appendix E
We observe that DPFRL significantly outperforms GRU in almost all games, which indicates the importance of explicit belief tracking, and shows that DPFRL can learn useful latent belief repre- sentation. Despite the simpler observations, DPFRL significatyy outperforms DVRL and achieves state-of-the-art results on out of 10 standard Flickering Atari Games (ChopperCommand, MsPac- man, BeamRider, Bowling, Asteroids), and it performs comparably in 3 other games (Centipede, Frostbite, IceHockey). The strength of DFPRL shows even more clearly in the Natural Flickering Atari Games, where it significantly outperforms DVRL on 7 out of 10 games and performs similarly in the rest. In some games, e.g. in Pong. DPFRL performs similarly with and without videos in the background (15.65 vs. 15.40), while the DVRL performacce degrades substantially (-19.78 vs. 17). These results show that while the architecture of DPFRL and DVRL. are similar, the policy-oriented discriminative update of DPFRL is much more effective for handling complex observations, and the MGF features provide a more powerful summary of the particle belief for decision making. However, on some games, e.g. ChopperCommant adding background videos. This shows that irrelevant features can make a task much harder, even for discriminative approach, also observed by Zhang et al. (2018).
even DPFRL performance drops significantly when
4.4
VISUAL NAVIGATION
Table 2: Visual Navigation Results
Success Rate
DPERL
0.79
0.88
12.82:15.82
DVRL. GRU
0.09
0.11
5.22-12.24
0.63
0.74
10.14:22.82
PPO Sanva at 2019
0.70
0.80
Figure 6: RGB-D Habitat Observations
Visual navigation poses great challenge for deep RL (Mirowski et al. 2016 Zhu et al. 2017 Lample
Chaplot 2017). We evaluate DPFRL for visual navigation the Habitat Environment (Savaa et al. 2019). using the real-world Gibson dataset (Xia et al. 2018). In this domain, robot needs to navigate to goals in previously unseen environments. In each time step, receives a first-person RGB-D camera image and its distance and relative orientation to the goal. The main challenge lies in the partial and complex observations: first-person view images only provide partial information about the unknown environment; and the relevant information for navigation, traversability, is encoded in rich RGB-D observations along with many irrelevant features, e.g., the texture of the wall. We use the Gibson dataset with the training and validation split provided by the Habitat challenge.
We train models with the same architecture as for the Atari games, except for the observation encoder that accounts for the different observation format. We evaluate models in unseen environments from
8

---

Published as a conference paper at ICLR 2020
Table Ablation Study Results for the Natural Flickering Atari Games
Envs
DPFRL
DPFRI-generative
DPFRL-PI
DPFRL-mean
DPFRL-GRUmerge
Pong
15.65+11999
-0.02
-1..00
-5.33=14.35
13.1444.001
ChopperCommand
1,566-167.03
1,027 12.94
1,287::255.0
1091
1,530=129.31
MsPacman
123.9:1123.9
182.3
2,233110,47
6,878:163.86
1,930.148.54
Contipede
4,164#23.0
3,194=339.4
3.557=3998.1
3.599=-439.8
4.093+76.4
BeamRider
682.91137.42
498.11.8.3
570.71172.3
645,511:27.4
603,8:140.25
Frostbite
260.2:t:4.00
255.9+22.8
178.2481.5
178.4481.70
252. 0,2.1
Bowling
29.45=#0.13
24.68+0.113
25.94=00.55
26,0+0.81
29,50=
Icel lockey
-0.08
-7.88.10.30
-6.02.11.03
16.95
-5,8510,30
DDunk
-15.36+00.6
-15.59+00.06
-1.228+00.6
-1442=0.18
-14,39:0024
Asteroids
1,406:+132.3
1,41545.33
1,18:16444
14333440.73
1,39=11144
the validation split and compute the same metries in the literature: SPL, success rate, and average rewards. Results are shown in Table 2 Further details and results are in Appendix [ anddd
DPFRL significantly outperforms both DVRL and GRU in this challenging domain. DVRL performs especially poorly. demonstrating the difficulty of learning a generative observation model in realistic, visually rich domains. DPFRL also outperforms the PPO baseline from Savva al. (0199).
We note that submissions to the recently organized Habitat Challenge 2019 (Savva et al. 2019), such as (Chaplot al., 22199, have demonstrated better performance than the PPO baseline (while our results are not directly comparable because of the closed test set of the competition). However, these approaches rely on highly specialized structures, such as mapping and 2D path planning, while we use the same generic network as for Atari games. Future work may further improve our results by adding a task-specific structure to DPFRL or training with PPO instead of A2C.
4.5 ABLATION STUDY
We conduct extensive ablation study on the Natural Flickering Atari Games to understand the influence of each DPFRL component. The results are presented in Table
The discriminative compatibility function is effective than a generative observation function. DPFRL--gnerratve replaces the discriminative compatibility function of DPFRL with a generative observation function, where grayscale image observations modeled by pixel-wise Gaussian distributions with learned mean and variance. Unlike DVRL, DPFRL DPFRL-generative only differs from DPFRL the parameterization of the observation function, the rest of the architecture and training loss remains the same. In most cases, the performance for DPFRL- OPFRL-generative degrades significantly compared DPFRL. These results are aligned with our earlier observations, and indicate that the compatibility function is capable of extracting the relevant information from complex observations without having to learn a more complex generative model.
More particles perform better. DPFRL with particle performs poorly on most of the tasks (DPFRLLL- PI). This indicates that a single latent state is insufficient to represent a complex latent distribution that is required for the task, and that more particles may improve performance.
MGF features are useful. We compare DPFRL using MGF features with DPFRL-mean, that only uses the mean particle; and with similar to DVRL. Results show that DPFRL-mean does not work as well as the standard DPFRL, especially for tasks that may need complex belief tracking, e.g., Pong. This can be attributed the more rich belief statistics provided by MGF features, and that they do not constrain the learned belief representation to be always meaningful when averaged. Comparing to DPFRL-GRUmerge shows that MGF features generally perform better. While an RNN may learn to extract useful features from the latent belief, optimizing the RNN parameters is harder, because they are permutation invariant to the set of particles and they result in a long backpropagation chain.
that
separate RNN to summarize the belief,
5 CONCLUSION
We have introduced DPERL, a framework for POMDP RL in natural environments. DPFRL combines the strength of Bayesian filtering and end-to-end RL: it performs explicit belief tracking with

---

Published as a conference paper at ICLR 2020
learnable particle filters optimized directly for the RL policy. DPFRL achieved taae-of-the-an art results on POMDP RL. benchmarks from prior work, Mountain Hike and number of Flickering Atari Games. Further, it significantly outperformed altermative methods in new, more challenging domain, Natural Flickering Atari Games, as well as for visual navigation using real-world data. We have proposed novel MGF feature for extracting statistics from empirical distribution. MGF feature extraction could be applied beyond RL, c.g., for general sequence prediction.
DPFRL does not perform well in some particular cases, e.g., DoubleDunk. While our task-oriented discriminative update are less susceptible to complex and noisy observations than a generative model, they do not benefit from an additional learning signal that could improve sample efficiency, e.g.. through reconstruction loss. Future work may combine generative observation model with the discriminative update in the DPFRI. framework.
ACKNOWLEDGEMENT
This research is partially supported by ONR Global and AFRL grant N62909- 1199909-18-1-0000 1--223.. We want to thank Maximilian Igl for suggesting to add videos to the background of Atari games.
REFERENCES
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning. arXiv preprint
2017.
Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, and Wee Sun Lee. Intention aware online pomdp planning for autonomous driving in a crowd. In 2015 ieee international conference on roboties and automation (icra), pp. 454-460. IEEE, 2015.
Bram Bakker. Reinforcement learning with long short-term memory. In Advances in neural information processing systems, 2002.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Michael George Bulmer. Principles of statistics. Courier Corporation, 1979.
Devendra Singh Chaplot, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Modular visual navigation using active neural mapping, Winner CVPR 2019 Habitat Navigation Challenge, 2019.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. recurrent latent variable model for sequential data. In Advances Neural Information Processing Systems, pp. 2980-2988, 2015.
Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communi- cate to solve riddles with deep distributed recurrent genetworks. arXiv preprint arXiv: 1602.066.. 2016.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. arXiv preprint arXiv: 2019.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: recurrent neural network for image generation. arXiv preprint arXiv: 1502.0463, 2015.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping belief states with generative environment models for rl. arXiv preprint 1rXiv:7906.002337. 2019.
Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A Pires, Toby Pohlen, and Rémi Munos. Neural predictive belief representations. arXiv preprint arXiv:1811.0644 2018.
Tuomas Haarnoja, Anurag Ajay, Sergey Levine, and Pieter Abbeel. Backprop KF: Learning discrimi- native deterministic state estimators. In Advances in Neural Information Processing Systems, pp. 4376-4388 2016.
10

---

Published as a conference paper at ICLR 2020
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. CoRR, abs/1507.06627,, 7(1), 2015.
Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. arXiv preprint arXiv:1810.0672 2018.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for POMDPs. In Proceedings of the International Conference Machine Learning, pp. 2117-2122 2018.
Rico Jonschkowski and Oliver Brock. End-to-end learnable histogram filters. In NeurlPS Workshop Deep Learning for Action and Interaction, 2016.
Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 1-8. IEEE, 2018.
Peter Karkus, David Hsu, and Wee Sun Lee. QMDP-net: Deep learning for planning under partial observability. In Advances in Neural Information Processing Systems, pp. 4694 2017.
Peter Karkus, David Hsu, and Wee Sun Lee. Particle filter networks with application to visual localization. In Proceedings of the Conference on Robot Learning, pp. 169-178, 2018.
Hanna Kurniawati, David Hsu, and Wee Sun Lee. Sarsop: Efficient point-based pomdp planning by
approximating optimally reachable belief spaces. In Robotics: Science and systems, 2008.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Tuan Anh Le, Maximilian Igl, Tom Rainforth, Tom Jin, and Frank Wood. Auto-encoding sequential monte carlo. 2018
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy-based learning. Predicting structured data, 1(0), 2006.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbecl. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):13341333, 2016.
Xiao Ma, Peter Karkus, David Hsu, and Wee Sun Lee. Particle filter recurrent neural networks. arXiv preprint arXiv: 1905. 12885, 2019.
Chris Maddison, John Lawson George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Amaud Doucet, and Yee Teh. Filtering variational objectives. In Advances in Neural Information Processing Systems, pp. 6573-6583, 2017.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.0366 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pp. 1928-1937, 2016.
Christian Naesseth, Scott Linderman, Rajesh Ranganath, and David Blei. Variational sequential monte carlo. In International Conference on Artificial Intelligence and Statistics, pp. 968-977, 2018.
11

---

Published as a conference paper at ICLR 2020
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1xXiv:187.003745 2018.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of markov decision processes.
Mathematics of operations research, 1987.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NewrIPS Autodiff Workshop, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 13::2112552, 2015. doi: 1260110011113-11-00000000
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans. Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied AI research. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver and Joel Veness. Monte-carlo planning in large pomdps. In Advances in neural information processing systems, pp. 2164-2172, 2010.
David Silver, Julian Schrittweser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 2017.
Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. DESPOT: Online POMDP planning with regularization. In Advances in neural information processing systems, pp. 1772-1780, 2013.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In Advances in Neural Information Processing Systems, 2016.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. Solving deep memory pomdps with recurrent policy gradients. In International Conference on Artificial Neural Networks. Springer, 2007.
Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint urXiv:181 11..1///.000 1.06032, 2018.
Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learning 2018.
for pomdps. arXiv preprint
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364 IEEE, 2017.
12

---

Published as a conference paper at ICLR 2020
A BACKGROUND
A.1 PARTICLE FILTER ALGORITHM
Particle filter is approximate Bayes filter algorithm for belief tracking. Bayes filters estimate the belief b, i.e., a posterior distribution of the state &t, given the history of actions 01 and observations 01:-- Instead of explicitly modeling the posterior distribution, particle filter approximates the posterior with set of weighted particles, b, R {(s w and update the particles in a Bayesian manner. Importantly, the particle set could approximate arbitrary distributions, e.g., Gaussians, continuous multi-modal distributions, etc. The mean state can be estimated as the mean particle & Σ w/st- The particle updates include three steps: transition update, measurement update, and resampling.
Transition update. We first update the particles by a given motion model. More specifically, we sample the next state 4141 from generative transition function
$1+1 p(s 5 ae)
(8)
where p(s st uz) is the transition function.
Measurement update. The particle weights are updated again using the observation likelihoods
K
w = mp(o, 1 = 1/
w +1
(9)
where is a normalization factor and p(or 11+1) is the observation likelihood computed by evaluating observation or in a generative observation function p(o s(1))
Resampling. The particle filter algorithm suffer from particle degeneracy, where after some update steps only a few particles have non-zero weights. This would prevent particle filter to approxi- mate the posterior distribution effectively. Particle degeneracy is typically addressed by performing resampling, where new particles are sampled with repetition proportional to its weight. Specifically, we sample particles from a categorical distribution p parameterized by the particle weights {w[) Tu
( w
(10)
where p(t) is the probability for the i-th category, i.e., the i-th particle. The new particles approximate the same distribution, but they assign more representation capacity to the relevant regions of the state space.
A.2 MOMENT-GENERATING FUNCTIONS
In probability theory, the moment- geneenttenn function (MGF) is an alternative specification of the probability distribution of real-valued random variable (Bulmer 1979). As its name suggests, MGF of a random variable could be used to generate any order of its moments, which characterize its probability distribution.
Mathematicallyy the MGF of a random variable X with dimension m is defined by
Mx (v)
EE e x
(11)
where E R and we could consider the MGF of random variable X is the expectation of the random variable ex X
Consider the series expansion of e X
x
X
X)x
x x1=
(12)
2!
n!
This leads to the well-known fact that the j-th order moment Mj (j-way tensor) is the j-th order derivative of the MGF at v = 0.
d'Mx
M,
lv=0
(13)
dvi
In DPFRL, MGFs additional features to provide moment information of the particle distri- bution. DPFRL learns to extract useful moment features for decision making by directly optimizing for policy p(a b.).
13

---

Published as a conference paper at ICLR 2020
B EXPERIMENT DETAILS
B.1 IMPLEMENTATION DETAILS
For the observation encoders, we used the same structure with DVRL. Igl et a 2018) for a fair comparison. For Mountain Hike, we use two fully connected layers with batch normalization and ReLU activation as the encoder. The dimension for both layers 64. For the rest of the domains, we first down-sample the image size to 84x84, then we process images with 3 20-oonvouttion layers with channel number (32, 64, 32), kernel sizes (8. 4, 3) and stride (4, 2, 1), without padding. The compass and goal information are a vector of length 2; they are appended after the image encoding as the input.
Observation Encoders:
Both DVRL and PFGRU- -eGRU-ggeeeative need observation decoders. For the Mountain Hike, we use the same structure as the encoder with a reversed order. The transposed 2D-comvolutional network of the decoder has a reversed structure. The decoder is processed by an additional fully connected layer which outputs the required dimension (1568 for Atari and Habitat Navigation, both of which have 84 x 84 observations).
Observation Decoders:
Observation-conndiiooe transition network: We directly use the transition function in PF- GRU Ma et al. 2019) for furass (hf Oc, which is a stochastic function with GRU gated structure. Action ar 18 first encoded by a fully connected layer with batch normalization and ReLU activation. The encoding dimension for Mountain Hike is 64 and 128 for all the rest tasks. The mean and variance of the normal distribution are learned again by two additional fully connected layers; for the variance, we use Softplus as the activation function.
State-observation compatibility network:
fohs is implemented by a single fully connected layer without activation. In DVRL, the observation function is parameterized over the full observation space and p(o h a.)) is assumed as a multivariate independent Bernoulli distribution whose parameters are again determined by a neural network (Igl et al., 2018). For numerical stability, all the probabilities are stored and computed in the log space and the particle weights are always normalized after each weight update.
Soft-resampling: The soft-resampling hyperparameter a is set to be 0.9 for Mountain Hike and 0.5 for the rest of the domains. Note that the soft noft-resampling is used only for DPFRL, not including DVRL. DVRL averages the particle weights to //K after each resampling step, which makes the resampling step cannot trained by the RI..
The GRU used in DVRL and DPFRL-GRUmes dimension equals the dimension of the latent vector plus 1, which is the corresponding particle weight. The dimension of this GRU is exactly the dimension of the latent vector. For the MGF features, we use fully connected layers with feature dimensions as the number of MGF features. The activation function used is the exponential function. We could potentially explore the other activation functions to test the generalized-Mer features, e.g., ReLU.
Belief Summary:
is single layer GRU with input
The actor network and policy network are two fully connected layers, which take in the belief summary by = [h,, M.lmm] as input. The output dimension of these two networks are chosen according to the RL tasks.
Actor Network and Policy Network:
Model Learning: For RL, we use an A2C algorithm with 16 parallel environments for both Mountain Hike and Atari games; for Habitat Navigation, we only use 6 parallel environments due to the GPU memory constraints. The loss function for DPFRL and GRU-based policy is just the standard A2C loss, L = -A + v CV + X C where C, the policy loss, CV the value loss, C the entropy loss for encouraging exploration, and A and l are two hyperparameters. For all experiments, we use Av = 0.5 and H 0.01. For DVRL, an additional encoding loss I.f is used to train the sequential VAE, which gives a loss function I. 7.DVRL IAA2C + X CF We follow the default setting provided by Igl al. 2 and set _ = 0.1. The rest the hyperparameters, including learning rate, gradient clipping value and in soft-resaplling are tuned according to the BeamRider and directly applied to all domains due to the highly expensive experiment setups. The learning rate for all the networks searched among the following values: (3 10 5 x 10 1 10 4 22 x 10-4 4.3 x 10 10-*): the gradient clipping value are searched among {0.5, 1.0}; the soft-resampling a is searched among (0.5, 0.9). The best performing learning rates were 1 for DPFRI. and GRU, and 2 for DVRL:
14

---

Published as a conference paper at ICLR 2020
the gradient clipping value for all models was 0.5; the soft-resampling is set to be 0.9 for Mountain
Hike and 0.5 for Atari games.
B.2 EXPERIMENTAL SETUP
Natural Flickering Atari games We follow the setting of the prior works (Zhu et al. 2018 Igl
et al. 2018): 1) 50% of the frames are randomly dropped 2) a frameskip of 4 is used 3) there is a 0.25 chance of repeating an action twice. our experiments, we sample background videos from the ILSVRC dataset (Russakovsky et al. 2015). Only the videos with the length longer than 500 frames are sampled to make sure the video length is long enough to introduce variabilityy. For each new episode, we first sample a new video from the dataset, and random starting pointer is sampled in this video. Once the video finishes, the pointer is reset the first frame (not the starting pointer we sampled) and continues from there. Experiment platform: We implement all the models using PyTorch (Paszke et al. 2017) with CUDA 9.2 and CuDNN 7.1.2. Flickering Atari environments modified based on OpenAl Gym (Brockman et al. 2016) and we directly use Habitat APIs for visual navigation. Collecting experience and performing gradient updates are done on single computation node with access to one GPU. For Mountain Hike and Atari games we use NVidia GTX 1080Ti GPUs. For Habitat visual navigation we use NVidia RTX2080Ti GPUs.
C PF-GRU NETWORK ARCHITECTURE
We implement DPFRL with gated transition and observation functions for particle filtering similar to PF-GRU (Ma et al. 2019).
In standard GRU, the memory update is implemented by a gated function:
h4 (1 24) tanh(n:) 2, ht-1, The [n(rr 24,1, bn.
(14)
where Wa and b, are the corresponding weights and biases, and 2 are the learned gates.
PF-GRU introduces stochastic cell update by assuming the update to the memory, ni. follows parameterized Gaussian distribution
ni WnIr Iri h;_1, bn d. N(O,E))) E Wz
by
(15)
With di [ (ot), feneeaaaaaae (a:)). implement the transition function hit1 Strans (hi, or, oc), where fene is the encoding network for observation and f is the encoding network for the actions.
For the observation function, we directly use fully connected layer fobs (ht oz) W. [ht o.] + bo. where Wo and 6, are the corresponding weights and biases.
D DPFRL ALGORITHM
Algorithm 1: DPFRL
Input:
Previous belief bi-1 R ((h/_1, wi_ 1R observation o,, action 0₁
xf <- Encoder(o:)
(encode the raw observation)
(rans(hi (ii, We (x²))
44,
(transition update)
K
wi + 1ofobel2 (e],,,t) hi)wi-1 1/
Σ
wi
(observation update)
[=1
[(h w() f Soft-Resampling((AM) ((
(soft-reaampling)
he wii
(compute the mean)
for j = 1: do
K
MP. - Σ with exp(v
(compute MGF features)
end
p(a bi) t- T(he, M(he, V(b.) f (he, M.
(compute the policy) (compute the value)
Output: Updated belief by 8S ((h) w policy p(a| b.) and value V (b.))
15

---

Published as a conference paper at ICLR 2020
E ADDITIONAL RESULTS
E. FLICKEIRNG ATARI GAMES PLOTS
We provide the accumulated reward curves for Atari experiments in this section.
Standard Flickering Atari Games. For the standard Flickering Atari Games we provide the training curves below. Results for DVRL and GRU are directly taken from Igl al. 2018).
DPTRL
DFTAL
DIVLL
20 GRU
0000
2500
2000
1520
2000
1000
20
ls7
Fromes
Frames
(a) Flickering Pong
(b) Flickering ChopperCommand
Flickering MsPacman
DFFAL
4000
DIFAIL
DFFRI
DVRL
DVRL
DVRL
40000
GRU
GRU
GRU
25:0
260
2000
3500
5:0
25:0
1+7
Fromes
1
Rames
(d) Flickering Centipede
Flickering BeamRider
Flickering Frostbite
36
DFFAI
DEFAIL
34
32
-10.0
wocu
-12.5
-15.0
28
-17.5
26
-20.0
24
-22.5
(g) Flickering Bowling
(h) Flickering IceHockey
(i) Flickering DoubleDunk
DPTAL
DVAL
GRU
3800
2000
Fro 19
Flickering Asteroids
16

---

Published as a conference paper at ICLR 2020
Natural Flickering Atari Games, For Natural Flickering Atari Games report results for separate validation set, where the background videos are different from the training set. The validation environment steps once after every 100 training iterations.
1600
2000
3750
740m
1200
1250
-10
1030
-20
Frames
147
(a) Natural Flickeirng Pong
(b) Natural Flickeirng ChopperCommand
(c) Natural Flickeinng MsPacman
DPPAIL
DPHAL
DPP RL DVRL CRU
DVAL
650
DVRL
250
225
8:0
5:0
200
500
175
430
150
430
2500
2250
Prames
107
Framres
Prames
167
(d) Natural Flickeirng Centipede
(e) Natural Flickeimg BeamRider
(f) Natural Flickeirng Frostbite
30
DPTAL DVAL
DPFIL
DHNI DVRL
DWRL
GAUI
GRU
26-
-14
-17
-10
22
-20
127
107
(g) Natural Flickeirng Bowling
(h) Natural Flickeirng IceHockey
ii) Natural Flickeirng DoubleDunk
2200
2000
800
(i) Natural Flickeinng Asteroids
17

---

Published as a conference paper at ICLR 2020
E.2 VISUAL NAVIGATION
We present the reward curve for the Habitat visual navigation task below. DPFRL outperforms both GRU-based policy and DVRL given the same training time. DVRL struggles with training the observation model and fails during the first half of the training time. GRU based policy learns fast; given only the model-free belief tracker, it struggles to achieve higher reward after a certain point.
We only provide the reward curve here as SPL and success rate are only evaluated after the training is finished.
DPFAL
DVRL
GRU
roward
167
Figure 7: Habitat Visual Navigation Reward
E.3 PARTICLE VISUALIZATION WITH PCA
We further visualize the latent particles by principal component analysis (PCA) and choose the first components. We choose trajectory in the Habitat Visual Navigation experiment, where 15 particles are used. We observe that particles initially spread across the space (f 0). As the robot only receive partial information in the visual navigation task, particles gradually form a distribution with two clusters (1 56), which represent two major hypotheses of current state. After more information is incorporated into the belief, they begin to converge and finally become single cluster (t = 81). did not observe particle depletion and posterior collapse our experiment. This could be better avoided by adding an entropy loss to the learned variance of frras and we will leave it for future study.
(a)
(b) 22
(c) 30
(d) / 56
(e) (=)==7 74
(1 1 1 81
18


Reviews:
[{"review_id": "SkepmDSCYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: my concerns have been addressed and I have updated the score to 8\n****\n\nThis paper introduces 3 neat ideas for training deep reinforcement learning (DRL) agents with state variables so that they can handle partially observed environments:\n1) model the latent state variable as a belief distribution, using a collection of weighted hidden states, like in the particle filter (PF), with an explicit belief update of each particle, calculation of the weight using an observation function, and a differentiable re-weighting function to get the new belief distribution,\n2) base the policy on the whole set of particles, by quantifying that set using its mean as well as a collection of K learnable moments (specifically, K Moment Generating Functions, each one corresponding to a dot product between the moment variable and the hidden state of the particle),\n3) instead of generating the observations, take again the idea from PF which is to measure the agreement between the current observation o_t and the i-th particle state variable h_t^i, via a learnable discriminative function.\nFrom what I understand, the only gradients in the model come from the usual 3 RL losses, and the observation functions in the discriminative PF are trained because they weigh the particles.\n\nThe model, trained using Advantage Actor Critic (A2C) works well on the (contrived, more on that later) \"flickering Natural\" Atari RL environment as well as on the Habitat navigation challenge, outperforming both the GRU-based deep RL agent and the Deep Variational RL based agent that uses variational sequence auto-encoders (and extra gradients from the observation function...). The ablation analysis confirms the advantages of the 3 ideas introduced in the paper.\n\nThe paper is a very well written and the experiments are very well executed. I believe that the idea is novel. I gave this paper only a weak accept because of unclear explanation and of several missed opportunities:\n\n* The observation function f_{obs}(h_t^i, o_t) is insufficiently explained. I understood it was trained using discriminative training. Does it mean that different observations o_t are used, and if so, how many? Or is the observation o_t the current observation of the agent, but only the h_t^i change? In which case, what makes it discriminative? Isn't there a posterior collapse, with all particles ending up bearing the same state? Does the function f_{obs} input o_t or u(o_t), where u is the convolutional network?\n\n* These questions could be easily answered with pseudocode in the appendix.\n\n* In section 3.1, what is the relationship between p_t(i) and f_{obs}(h_t^i, o_t)?\n\n* Particle filters in navigation enable to store the history of the observations of the mobile robot, accumulating the laser range scans and matching them to the observations. At the end, one can visualise the map stored in a given particle, as well as visualise the point cloud of the particle coordinates and show the trajectories of these particles. Here the particles contain the hidden states of the agent. Could you similarly to traditional PF, visualise the position of the agent by matching the point cloud {{h_t^i}_i}_t to a set of observations o_k taken from the whole environment, and plotting a 2D map of weights coming from function f_{obs}(h_t, o_k) evaluated over all k?\n\n* In the discussion, can you comment on the relationship between Monte-Carlo Tree Search in RL agents (sampling different trajectories) vs. here (sampling different states)?\n\n* While I understand the need to use that environment for the sake of comparison to DVRL, the Atari + flickering + natural images dataset is very artificial and contrived. I would be interested in seeing more analysis of the discriminative PF RL algorithm on navigation tasks, given that that's what PF were designed for.\n\nSome missing references:\n* Early references on DRL for navigation:\nZhu et al (2016) \"Target-driven visual navigation in indoor scenes using deep reinforcement learning\"\nLample & Chaplot (2016) \"Playing FPS games with deep reinforcement learning\"\nMirowski et al (2016) \"Learning to navigate in complex environments\"\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1l7IWx99B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the specific question/problem tackled by the paper?\n\nRepresentation learning in POMDPs in order to ignore spurious information in observations.\n\n\nIs the approach well motivated, including being well-placed in the literature?\n\nSome comparisons to related work are missing; while the comparisons would enrich the paper, their absence is not fundamentally limiting to the conclusions.\n\nThere's an additional PSR-related work that can be seen as learning representations for POMDPs (Guo et al., Neural predictive belief representations, arXiv:1811.06407). This work is in line with the work of Gregor et al., 2019, and both provide suitable representation learning techniques for POMDPs. \n\nThese representation learning in the paper is based on action-conditional predictions of future quantities, which is complementary to the approach proposed in the paper. That is, one could conceive adding action-conditional predictions of the future with the particles as the RNN states.\n\n\nDoes the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.\n\nI think the support is somewhat adequate.\n\nThe claim that the proposed method handles spurious information is well supported by the experiment in mountain hike, but not quite so by the Atari experiments. The performance (upon introduction of the \"natural\" on top of flickering) takes a big hit for both DPFRL and DVRL. Still, the performance improvement of DPFRL over DVRL is still an encouraging result.\n\n\nSummarize what the paper claims to do/contribute. Be positive and generous.\n\nThe paper proposes a neural implementation of particle filters, by treating samples of RNN states as particles. The particles are used to estimate moment-generating functions evaluated at trained vectors, which in turn are supposed to provide more information for the policy's decision making. The paper uses a discriminator to shape the representation.\n\nThe ablation study suggests that all three components (particles, MGFs & discrimination) are necessary. However, the third component has been shown not to be exclusively helpful for representation learning (Gregor et al., Guo et al.) I would suggest a study in comparison to Gregor et al.'s method (DRAW) instead.\n\n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\n\nI vote for acceptance.\n\n\nProvide supporting arguments for the reasons for the decision.\n\nI think the algorithmic idea in this paper is a step in the right direction and can be of interest for the community. I would hope for the benchmarks to be more like the Habitat, and less like Atari with background videos. The conclusions in the latter benchmark seem less likely to apply to tasks in physically structured environments.\n\n\nProvide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n\nI think it is important for the paper to qualify the kind of POMDPs being considered. The defining features of most of the environments being used is that the state is observed through a noisy channel. Many POMDPs are of interest because the observations are really providing partial information about the state, even if it is noiseless. This is the case for the Habitat setting.\n\nBecause the paper's claims about the adequacy of the method for POMDPs rests on the choice of environments, I think it's important to quality what kind of POMDPs are being considered here. I would also caution against stating that the environment is closer to the real world. It would perhaps be better to say that the natural flickering is more interesting than the natural and the flickering because it benchmarks robustness to irrelevant information in observations, provided almost in tandem with state information; with intermittently missing observations.\n\nPlease add some explanation about how the negative examples are sampled for the contrastive estimation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJeKIBfg9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a well written paper. It introduces a principled method for POMDP RL:  Discriminative Particle Filter Reinforcement Learning (DPFRL).\nIt combines the strength of Bayesian filtering and policy-oriented discriminative modeling. DPFRL encodes a differentiable particle filter with learned transition & observation models in a neural network, allowing for reasoning with partial observations over multiple time steps. It performs explicit belief tracking with discriminative learnable particle filters optimized directly for the RL policy. \n\nExperimental results show that DPFRL achieves state-of-the-art on POMDP RL benchmarks. I especially like the paper covers a diverse set of applications, including Mountain Hike, the classic Atari games, and visual navigation (Habitat). Improved performance is reported. Results show that the particle filter structure is effective for handling partial observations, and the discriminative parameterization allows for complex observations.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce an RL algorithm / architecture for partially observable                                                      
environments.                                                                                                                      
At the heart of it is a filtering algorithm based on a differentiable version of                                                   
sequential Monte Carlo inference.                                                                                                  
The inferred particles are fed into a policy head and the whole architecture is                                                    
trained by RL.                                                                                                                     
The proposed methods was evaluated on multiple environments and ablations                                                          
establish that all moving parts are necessary for the observed performance.                                                        
                                                                                                                                   
All reviewers agree that this is an interesting contribution for addressing the                                                    
important problem of acting in POMDPs.                                                                                             
                                                                                                                                   
I think this paper is well above acceptance threshold. However, I have a few points that I                                         
would quibble with:                                                                                                                
1) I don't see how the proposed trampling is fully differentiable; as far as I                                                     
understand it, no credit is assigned to the discrete decision which particle to                                                    
reuse. Adding a uniform component to the resampling distribution does not                                                          
make it fully differentiable, see eg [Filtering Variational Objectives. Maddison                                                   
et al]. I think the authors might use a form of straight-through gradient approximation.                                           
2) Just stating that unsupervised losses might incentivise the filter to learn                                                     
the wrong things, and just going back to plain RL loss is not in itself a novel                                                    
contribution; in extremely sparse reward settings, this will not be                                                                
satisfactory.           
Decision: Accept (Poster)