Critic Sequential Monte Carlo
Authors: Vasileios Lioutas
Jonathan Wilder Lavington
Justice Sefas
Matthew Niedoba
Yunpeng Liu
Berend Zwartsenberg
Setareh Dabiri
Frank Wood
Adam Scibior
Venue: ICLR 2023 poster (2023)
Keywords: sequential monte carlo
reinforcement learning as inference
soft Q-learning
heuristic factors
driving behavior models

Abstract: We introduce CriticSMC, a new algorithm for planning as inference built from a composition of sequential Monte Carlo with learned Soft-Q function heuristic factors. These heuristic factors, obtained from parametric approximations of the marginal likelihood ahead, more effectively guide SMC towards the desired target distribution, which is particularly helpful for planning in environments with hard constraints placed sparsely in time. Compared with previous work, we modify the placement of such heuristic factors, which allows us to cheaply propose and evaluate large numbers of putative action particles, greatly increasing inference and planning efficiency. CriticSMC is compatible with informative priors, whose density function need not be known, and can be used as a model-free control algorithm. Our experiments on collision avoidance in a high-dimensional simulated driving task show that CriticSMC significantly reduces collision rates at a low computational cost while maintaining realism and diversity of driving behaviors across vehicles and environment scenarios.

Published as a conference paper at ICLR 2023
CRITIC SEQUENTIAL MONTE CARLO
Vasileios Li J. Wilder Lavington 1:2 Justice Sefas Matthew Niedoba 1.2 Yunpeng Liut.2, Berend Zwartsenherg', Setareh Dabiri', Frank Wood1.23, Adam Šcibior' Inverted AI, University of British Columbia, Mila; "vasi lei iootootsste nnverte.
ABSTRACT
We introduce CriticSMC, new algorithm for planning as inference built from a composition of sequential Monte Carlo with learned Soft-Q function heuristic factors. These heuristic factors, obtained from parametric approximations of the marginal likelihood ahead, more effectively guide SMC towards the desired target distribution, which particularly helpful for planning environments with hard constraints placed sparsely in time. Compared with previous work, we modify the placement of such heuristic factors, which allows us to cheaply propose and evaluate large numbers of putative action particles, greatly increasing inference and planning efficiency. CriticSMC compatible with informative priors, whose density function need not be known, and can be used model-free control algorithm. Our experiments on collision avoidance in a high-dimensional simulated driving task show that CriticSMC significantly reduces collision rates at a low computational cost while maintaining realism and diversity of driving behaviors across vehicles and environment scenarios.
INTRODUCTION
1993) is popular, highly customizable inference algorithm that is well suited posterior inference in state-space models (Arulampalam al. 2002] Andrieu et al. 2004 Cappe et al. (007). SMC is a form of importance sampling, that breaks down a high-dimensional sampling problem into sequence of low-dimensional ones, making them tractable through repeated application of resampling. SMC in practice requires informative observations at each time step be efficient when a finite number of particles is used. When observations are sparse, SMC loses its typical advantages and needs to be augmented with particle smoothing and backward messages to retain good performance (Kitagawa 1994 Moral et al. 2009 Douc et al. 2011).
Sequential Monte Carlo (SMC) (Gordon et al.
SMC can be applied to planning problems using the
inference framework (Ziebart Neumann] [001 Rawlik et al., 2012, Kappen al. 2012) [eeinee] 2018 Abdolmaleki
al. 2010:
et al. with sparse, hard constraints, such as avoiding collisions while driving.
2018 Lavington al. 2021). In this paper we are interested in solving planning problems
this setting, such a
constraint is not violated until the collision occurs, but braking needs to occur well in advance to avoid it. Figure [ demonstrates on a toy example how SMC requires an excessive number of particles to solve such problems. In the language of optimal control (ÓC) and reinforcement learning (RL), collision avoidance is a sparse reward problem. In this setting, parametric estimators of future rewards (Nair et al.) 2018 Riedmiller et al.) 2018) are learned in order alleviate the credit assignment problem (Sutton& & Barto 2018 Dulac Arnold et al. 2021) and facilitate efficient learning.
In this paper we propose novel formulation of SMC, called CriticSMC, where a learned critic, inspired by Q-functions in RL (Sutton & Barto 2018). is used as a heuristic factor (Stuhlmüller al. 2015) in SMC to ameliorate the problem of sparse observations. We borrow from the recent advances in deep-RL (Haarnoja et al.) 80, Hessel et al. 2018) to learn a critic which approximates future likelihoods in a parametric form. While similar ideas have been proposed in the past (Rawlik et al. 2012 Piché et al. 2019). in this paper we instead suggest (1) using soft Q-functions (Rawlik ct al. 2012 Chan et al. 202 Lavington et al., 2021) as heuristic factors, and (2) choosing the placement of such factors to allow for efficient exploration of action-space through the use of putative particles (Fearnhead, 2004). Additionally, we design CriticSMC to be compatible with informative prior distributions, which may not include associated (known) log-density function. In planning contexts, such priors can specify additional requirements that may be difficult to define via rewards, such maintaining human-like driving behavior.

---

Published as a conference paper at ICLR 2023
(a) SMC (10 particles)
(b) SMC (10k particles)
(c) CritieSMC (10 particles)
Figure I: Illustration of the difference between CriticSMC and SMC a toy environment which a green ego agent is trying to reach the red goal without being hit by any of the three chasing adversaries. All plots show overlaid samples of environment trajectories conditioned on the ego agent achieving its goal. While SMC will asymptotically explore the whole space of environment trajectories, CriticSMC's method of using the critic as a heuristic within SMC encourages computationally efficient discovery of diverse high reward trajectories. SMC with a small number of particles fails here because the reward is sparse and the ego agent's prior behavioral policy assigns low probability trajectories that avoid the barrier and the other agents.
We show experimentally that CriticSMC is able to refine the policy of a foundation (Bommasani et al. (02T) autonomous-driving behavior model to take actions that produce significantly fewer collisions while retaining key behavioral distribution characteristics of the foundation model. This is important not only for the eventual goal of learning complete autonomous driving policies (Jain et al.
2021 Hawke et al. 2021). but also immediately for constructing realistic infraction-fin free simulations to be employed by autonomous vehicle controllers (Suo et al. 2021 Bergamini et al. 2021 Scibior al. 2021]] Lioutas et al. 2022) for training and testing. Planning, either in simulation real world, requires a model of the world (Ha & Schmidhuber] (2018). While CriticSMC can act as a planner in this context, we show that it can just easily be used for model-free online control without world model. This is done by densely sampling putative action particles and using the critic to select amongst these sampled actions. We also provide ablation studies which demonstrate that the two key components of CriticSMC, namely the use of the soft Q-functions and putative action particles, significantly improve performance over relevant baselines with similar computational resources.
2 PRELIMINARIES
Since primarily concerned with planning problems, work within the framework of Markov decision processes (MDPs). An MDP M (S. A, R. IT] is defined by a set states E S, actions a E A, reward function r(s, s') R. deterministic transition dynamics function f(s, a). initial state distribution (s) E Po. and policy distribution (() E II. Trajectories are generated by first sampling from the initial state distribution 51 ~ Po. then sequentially sampling from the policy ~ (aa)s)) and then the transition dynamics 8111 + f(s,, ae) for T-1 time steps. Execution of this stochastic process produces a trajectory which is then scored using the reward function The goal in RL and OC is to produce policy n arg max, Ep |E=1 r(st, ae 1)]. We now relate this stochastic process to inference.
f(s1, a1 ....................... (sT, ar)} p².
2.1 REINFORCEMENT LEARNING AS INFERENCE
RL-as-inference (RLAI) considers the relationship between RL and approximate posterior inference to produce class of divergence minimization algorithms able to estimate the optimal RL policy. The posterior we target is defined by set of observed random variables 01:1 and latent random variables TI:7- Here, O defines "optimality" random variables which Bernoulli distributed with probability proportional to exponentiated reward values (Ziebart et al.) 2010 Neumann [ 2018).
They determine whether individual tuple 7, (8e; 0t St+1 is optimal (O2 1) sub-optimal
2

---

Published as a conference paper at ICLR 2023
1: algorithm STANDARD
algorithm H-FACTORS
1: algorithm CRITICSMC
2: of x(aols))
2: aj a(aalss))
2: of x(ad))))
3: 6/+1 a(
3: ii+1 a()
3:
4: "i + - ((
4: ú; e
4: of t -F
5: aj RESAMPLE(W))
5: of ~ RESAMPLE(W)!))
5: of ~ (ESAMPLE(10)o))
6. 5/41 F $141
6: s
6: 6:41 [+ ;
7: 10 + #
7: कi + t
7: ut + N p (4
8: end algorithm
8: end algorithm
8: end algorithm
No heuristic factors
(b) With heuristic factors
(c) CriticMMC version
Figure 2: Main loop of SMC without heuristic factors (left), with naive heuristic factors he (middle) and with the placement we use in CritieSMC (right). We use i for pre-resampling weights and w for post-resampling weights and we elide the normalizing factor We = Σ e for clarity. The placement of he in CritieSMC crucially enables using putative action particles in Section 3.3
(O, = 0). We replace O₂ = 1 with O in the remainder of the paper for conciseness. While we can rarely compute the posterior p($1:T,a1:7 (1:r)
(:)) in closed form, we assume the joint distribution
T
P(8ITT, aI., 0 = (o())) II [(Or) Se, ax, Se+ 18 (T
(I)
t=1
where 8 (()....) is Dirac measure centered on [(st,a)) This joint distribution can be used following standard procedures from variational inference to learn or estimate the posterior distribution of interest (Kingma& & Welling 2014). How close the estimated policy is to the optimal policy often depends upon the chosen reward surface, the prior distribution over actions, and chosen policy distribution class. Generally, the prior is chosen to contain minimal information in order to maximize the entropy of the resulting approximate posterior distribution (Ziebart et al. 2010 Haarnoja et al.) 2018a). Contrary to classical RL, are interested in using informative priors whose attributes want to preserve while maximizing the expected reward ahead. In order to manage this trade-off, we now consider more general inference algorithms for state-space models.
2.2 SEQUENTIAL MONTE-CARI.O
SMC (Gordon et al. 1993 is popular algorithm that can be used to sample from the posterior distribution in non- linerrn state-space models and HMMs. In RLAI, SMC sequentially approximates the filtering distributions (s,, a, O1::) for E 1 T using a collection of weighted samples called particles. The crucial resampling step adaptively focuses computation on the most promising particles while still producing unbiased estimation of the marginal likelihood (Moral) 2004 Chopin et al., 2012, Pitt et al., 2012: Naesseth et al.) 2014 Le 2017). The primary sampling loop for SMC in a Markov decision process provided in Figure 2al and proceeds by sampling action a, given state 8t, generating the next state st+1 using the environment or model of the world, computing a weight vi, using the reward function r, and resampling from this particle population. The post-resampling weights Wj are assumed to be uniform for simplicity but non-uniform resampling schemes exist (Fearnhead & Clifford 2003). Here, each timestep only performs simple importance sampling linking the posterior p( dg O.:)) to p( 1, a++1 (++1|01=++1) When the observed likelihood information is insufficient, the particles may fail cover the portion of the space required to approximate the next posterior timestep. For example, if all current particles have the vehicle moving at high speed towards the obstacle, may be too late brake and causing SMC to erroneously conclude that collision was inevitable, while in fact it just did not explore braking actions earlier on in time.
As shown by Stuhlmüller et al. (2015), we can introduce arbitrary heuristic factors h4 into SMC before resampling, as shown in Figure 2b mitigating the insufficient observed likelihood information. he can be function of anything sampled up to the point where it is introduced, does not alter the asymptotic behavior of SMC, and can dramatically improve finite sample efficiency if chosen carefully, at).
In this setting, the optimal choice for h2 is the marginal log-likelihood ahead E log p (00rr (1)))
3

---

Published as a conference paper at ICLR 2023
which is typically intractable to compute but be approximated. In the context of avoiding collisions, this term estimates the likelihood of future collisions from a given state. A typical application of such heuristic factors in RLAI, as given Pichéé et al. (2019). is shown in Figure 2b
3 CRITICSMC
Historically, heuristic factors in SMC are placed alongside the reward, which is computed by taking a single step in the environment (Figure 2bp. The crucial issue with this methodology is that updating weights requires computing the next state (Line complex simulators, and would prevent the in online control without world model. In order to avoid this issue while maintaining the advantages of SMC with heuristic factors, we propose to score particles using only the heuristic factor, resample, then compute the next state and the reward, as shown in Figure 2c We choose he which only depends on the previous state and actions observed and not the future state, so that we can sample and score significantly larger number of so-called putative action particles, thereby increasing the likelihood of sampling particles with a large h4. In this section we first show how to construct such h, then how to learn an approximation to and finally how to take full advantage of this sampling procedure using putative action particles.
Figure 2b). which can both be expensive in
3.1 FUTURE LIKELIHOODS AS HEURISTIC FACTORS
We consider environments where planning needed to satisfy certain hard constraints C(st) and define the violations of such constraints as infractions. This makes the reward function (and thus the log-likelihood) defined in Section 2 sparse.
log p( St, at, 1 = r(t,, ae sc+1) = -000 Spen: otherwise
if C(1+1) is satisfied
(2)
where 8pen > 0 is a penalty coefficient. At every time-step, the agent receives a reward signal indicating if an infraction occurred (e.g. there was collision). To guide SMC particles towards states that are more likely to avoid infractions likelihoods (Kim al., 2020) defined as h R log p(Oerlss, az). Such heuristic factors up-weight particles proportionally to how likely they are to avoid infractions in the future but can be difficult to accurately estimate in practice.
the future, we use h4 which approximate future
As has been shown in previous work (Rawlik et al. 2012 Levine 2018 Piché et al. 2019 Lavington
et al. 2021). ). log p( (Ourlst, az) corresponds to the "soft" version of the state-action value function (st Ge) used in RL, often called the critic. Following Levine (018). we use the same symbol Q for the soft-critic. Under deterministic state transitions St+1 f- [st, ar), the soft function satisfies the following equation, which follows from the exponential definition of the reward given in Equation (a proof provided in Section A.2 of the Appendix).
Q(st, ax) == log (O.ppOOOTss...... a.) r(5t, az 81+1) + log
E Ret 1 ( (
è Q(s44 (a411)
(3)
CritieSMC sets the heuristic factor h4 = Q( a.), as shown in Figure 2c
We note that alternatively
one could use the state value function for the next state V($+41) = log E Eorzn [exp (( $1+1))]
as shown in Figure 2b This would be equivalent to the SMC algorithm of Piché et al. (2019) (see
Section A.2 of the Appendix), which was originally derived using the two-filter formula (Bresler 1986
Kitagawa| 1994) instead of heuristic factors. The primary advantage of the CriticSMC formulation that the heuristic factor can be computed before the next state, thus allowing the application of putative action particles.
3.2 LEARNING CRITIC MODELS WITH SOFT Q-LEARNING
Because do not have direct access Q, estimate it parametrically with Qo. Equation suggests the following training objective for learning the state-action critic Lavington et al. 2021
((
(oolst, ar) T(Se, azs S141) 1- log E
(TD(O)= =
E Se, are (0,, melaoo
rost (ac+1 1)441)
E
E
1: - 8411, 0741
(
Qua(ss,
27
(4)
81 8e.0t S0 desso
att aaleesl (s442)
4

---

Published as a conference paper at ICLR 2023
where dsao is the state-action occupancy (SAO) induced by CritieSMC, 1 is the stop-gradient operator (Foerster et al. 2018) indicating that the gradient of the enclosed term is discarded, and the approximate target value QTA defined as
K Σ
(sa at, St+1, 2111 r(st, ae s1+1) + y log
1 K
e (
(5)
The discount factor y is introduced reduce variance and improve the convergence of Sofi-Q iteration (Bertsekas) 2019; Chan et al. 2021). For stability, we replace the bootstrap term Q1()) with -avveraging target network Qu (Lilicrap et al. 2016), and use prioritized experience replay (Schaul et al. 2016), a non-uniform sampling procedure. These modifications are standard in deep RL, and help improve stability and convergence of the trained critic (Hessel al. 2018). We note that unlike Piché et al. (2019), we learn the soft-Q function for the (static) prior policy, dramatically simplifying the training process, and allowing faster sampling at inference time.
3.3 PUTATIVE ACTION PARTICLES
Sampling actions given states is often compu- lationally cheap when compared to generat- ing states following transition dynamics. Even when a large model is used to define the prior policy, it is typically structured such that the bulk of the computation is spent processing the state information and then a relatively small probabilistic head can be used to sample many actions. To take advantage porarily increase the particle population size K-fold when sampling actions and then reduce it by resampling before the state is com- puted. This is enabled by the placement of heuristic factors between sampling the action and computing the next state, as highlighted in Figure 2c Specifically, each time step for each particle we sample K actions ā. re- sulting in N-K putative action particles (Fearn) head, 2004). The critic is then applied as heuristic factor to each putative particle, and population of size re-sampled following the next time-step using these weighted examples. The full algorithm is given in Algorithm !
Algorithm Critic Sequential Monte Carlo
procedure CRITICSMC(po f. m. f. Q N, K, T)
Sample s ~ -o((s)
Set 00 I
for t E 11.....7 do
for 1...N do
for k 1... do
Sample aj Set wt N+
π ra (s]) K 16 1 907,070
this, we tem-
end for end for W.
ii
Sample a;
I:N
RESAMPLE
GLNNKK
for
1...N
N do
Set [al] [K| +
Set <- (a, mod K) +
Set al <- a;
Set ab+1 + f(s),
Set wi w (ojaaa -
end for end for return $1:7 a.r WI end procedure
I:N I:N
For low dimensional action spaces, it is possi- ble to sample actions densely under the prior, eliminating the need for a separate proposal distribution. This is particularly beneficial in settings where the prior policy is only defined implicitly by a sampler and its log-density cannot be quickly evaluated everywhere. In the autonomous driving context, the decision leading to certain actions dimensional. Using CriticSMC, a prior generating human-like actions can be provided as a sampler without the need for density function. Lastly, CriticSMC can be used for model-free online control through sampling putative actions from the current state, applying the critic, and then selecting single action through resampling. This can be regarded as a prior-awe aware approach selecting actions similar to algorithms proposed by AAboomaaleki et al. (2018); Song et al. (2019).
be complex, but the action space is only two- or three-
4 EXPERIMENTS
We demonstrate the effectivennes of CriticSMC for probabilistic planning where multiple future possible rollouts are simulated from a given initial state using CriticSMC using two environments: a multi-agent point-masy toy environment and deth-imnnnsional driving simulator. In both environ- ments infractions are defined as collisions with either other agents or the walls. Since the environment dynamics are known and deterministic, do not lear state transition model of the world and there
5

---

Published as a conference paper at ICLR 2023
is no need to re-plan actions in subsequent time steps. We also show that CriticSMC successfully avoids collisions in the driving environment when deployed in a model-free fashion in which the proposed optimal actions are executed directly in the environment at every timestep during the CriticSMC process. Finally, we show that both the use of putative particles and the Soft-Q function instead of the standard Hard-Q result in significant improvement in terms of reducing infractions and maintaining behavior close to the prior.
4.1 TOY ENVIRONMENT
In the toy environment, depicted Figure ! the prior policy is Gaussian random walk towards the goal position without any information about the position of the other agents and the barrier. All external agents are randomly placed and move adversarially and deterministically toward the ego agent. The ego agent commits an infraction if any of the following is true: 1) colliding with any of the other agents, 2) hitting a wall, 3) moving outside the perimeter of the environment. Details of this environment can be found in the Appendix.
We compare Cr using 50 particles and 1024 putative action particles on the planning task against several baselines, namely the prior policy, rejection sampling with 1000 maximum trials, and the SMC method of Piché et al. (2019) with 50 particles. We randomly select 500 episodes with different initial conditions and perform 6 independent rollouts for each episode. The prior policy has an infraction rate of 0.84, rejection sampling achieves 0.78 and SMC of Piché et al. (2019) yields an infraction rate of 0.14. CriticSMC reduces infraction rate to 0.02.
4.2 HUMAN-LIKE DRIVING BEHAVIOR MODELING
Human-like driving behavior models are increasingly used to build realistic simulation for training self-driving vehicles (Suo et al. 2021 Bergamini et al. 2021 Scibior et al. 2021), but they tend to suffer from excessive numbers of infractions, in particular collisions. In this experiment we take an existing model of human driving behavior, ITRA (Scibior et al. 2021 J. as the prior policy of predictions as much as possible. The environment features non-ego agents, for which we replay actions as recorded in the INTERACTION dataset (Zhan et al. 2019). The critic receives a stack of the last two ego-centric ego-rotated birdview images (Figure 3) of size 256x256x as partial observations of the full state. This constitutes challenging, image-based, high-dimensional continuous control environment, in contrast to Piché et al. (2019). who apply their algorithm to low dimensional vector-state spaces in the Mujoco simulator Toodoro Todorov al. (2012); Brockman et al. (016)). The key performance metric in this experiment is the average collision rate, but we also report the average displacement егrоr (ADE,) using the minimum error across six samples for each prediction, which serves as a measure of human- likeness. Finally, the maximum final distance (MFD) metric is reported to measure the diversity of the predictions. The evaluation is performed using the validation split of the INTERACTION dataset, which neither ITRA nor the critic saw during training.
and attempt to avoid collisions, while maintaining the
We evaluate CritieSMC on a model-based planning task against the following baselines: the prior policy (ITRA), rejection sampling with 5 maximum trials and the SMC incremental weight update rule proposed by PPiché et al. (2019) using particles. CriticSMC uses 5 particles and 128 putative particles, noting the computational cost of using the putative particles is negligible. We perform separate evaluation in each of four locations from the INTERACTION dataset, and for each example in the validation set we execute each method six times independently to compute the performance metrics. Table [ shows that CriticSMC reduces the collision rate substantially more than any of the baselines and that it suffers smaller decrease in predictive error than the SMC Piché et al. (2019). All methods are able to maintain diversity of sampled trajectories on par with the prior policy.
Next, we test Cr as a model-free control method, not allowing it to interact with the environ- ment until an action for a given time step is selected, which is equivalent to using single particle in Cr. Specifically, at each step we sample 128 putative action particles and resample one of them based on critic evaluation as a heuristic factor. We use Soft Aetor-Critic (SAC) (Haarnoja jtt al.) 2a) as model-free baseline, noting that other SMC variants are not applicable in this context, since they require inspecting the next state to perform resampling. We show in Table 2 that CriticSMC is able to reduce collisions without sacrificing realism and diversity the predictions. Here SAC does notably worse in terms of both collision rate well as ADE. This is unsurprising as

---

Published as a conference paper at ICLR 2023
Figure 3: Collision avoidance arising from using CritieSMC for control the red ego agent in a scenario from the INTERACTION dataset. There are three rows: the top shows the sequence of states leading to a collision aris- ing from choosing actions from the prior policy, the middle shows that control by CriticSMC's implicit policy avoids the collision, and the third is contour plot illustrating the relative values of the critic (brighter sponds to higher expected reward) evaluated the current state over the entire action space acceleration (verti- cal axis) and steering (horizontal axis). The black dots are 128 actions sampled from the prior policy. The white dot indicates the selected action. Best viewed zoomed onscreen. For more examples see Figureffin the Appendix.
Table I: Infraction rates for different inference methods performing model- predictive planning tested on four locations from the INTERACTION dataset (Zhan et al. 2019).
Collision
Location
Method
MFD
ADEş
Infraction Rate
Prior Rejection Sampling SMC by Piché et al. 2019
0,02522
2,2038
0.3024
0.01758
2.3578
0.3071
DR_DEU DR.DEUUM MT
0.02191
0.4817
CriteMMC
0.01032
2,2009
0,3448
Prior Rejection Sampling SMC by Piché et 2019
0.00874
3.1369
0,3969
0.00218
3.2100
0.3908
DR USA Intersection MA
0.00351
2.8490
0.4622
CriticMC
0.00085
2.8713
0,4479
Prior Rejection Sampling SMC hy Piché 2019
0.00583
3,1004
0,4080
0.00133
3,0211
0.4046
DR USA Roundahout FT
0.00166
3.0086
0.4814
CAuceMCC
0.00066
2,9736
0,4439
Prior
0.00583
3,5536
0.4389
Rejection Sampling SMC by Piché al. 2019
0.00216
3,4992
0.4287
DR_DEU Roundabout.OF
0.00342
3.2836
0.5701
CritiMMC
0.00083
3.4248
0,4450
CriticSMC takes advantage of samples from the prior, which is already performant in both metries,
while SAC must be trained from scratch. This example highlights how CriticSMC utilizes prior information more efficiently than black-box RL algorithms like SAC.
4.3 METHOD ABLATION
Effeet of Using Putative Action Particles We evaluate the importance of putative action particles, via an ablation study varying the number of particles and putative particles in CriticSMC and standard SMC. Table contains results that show both increasing the number of particles and putative articles have a significant impact on performance. Putative particles particularly important since large number them can typically be generated with a small computational overhead.
Comparison of Training the Critic With the Soft-Q and Hard-Q Objective We compare the fitted Q iteration (Watkins & Dayan 1992). which uses the maximum over Q at the next stage to update the critic (i.e., maX Q (441)) with the fitted soft-Q iteration used by CritieSMC (Eq. 4.. The results, displayed in Table show that the Hard-Q heuristic factor leads to significant reduction in collision rate over the prior, but produces significantly higher ADEg score. We attribute this to the risk-avoiding behavior induced by hard-Q.
7

---

Published as a conference paper at ICLR 2023
Table Infraction rates for performing model-free online control against the prior and SAC policies tested on four locations from the INTERACTION dataset Zhan et al. 2019).
Collision Infraction Rate
Location
Method
MFD
ADE6
Prior
0.02522
2.2038
0,3024
DR. Merginggiing..... MT
SAC
0.03899
0.0
1.1548
CriticSMC
0.01376
2.1985
0,3665
Prior
0.00874
3.1369
0,3969
UML_USAA_Itervvection.MM
SAC
0.02700
0.0
4.1141
CriticSMC
0.00285
2.9595
0.4641
Prior
0.00583
3.1004
0,4080
DR.USA Roundabout
SAC
0.04501
0.0
1.7987
CriticSMC
0.00183
3.0125
0.4567
Prior
0.00583
3.5536
0,4389
SAC
0.06400
0.0
3,4583
DEU.RRondabout.OF
CriticSMC
0.00233
3.5173
0.4459
Table 3: Infraction rates for SMC and CriticSMC with varying number of particles and putative particles, tested on 500 random initial states using the proposed toy environment.
Particles
Method
Putative Particles
10
SMC
0.774
0.488
0.383
0.288
0.183
CriticSMC
0.774
0.368
0.298
0.162
0.072
SMC
1024 1024
0.772
0.415
0.281
0.179
0.119
CriticSMC
0.094
0.031
0.021
0.016
0.008
Execution Time Comparison Figure shows the average execu- tion time it takes to predict seconds into the future given second of historical observations for the driving behavior modeling exper- iment. This shows that the run-time of all algorithms of the same order, while the collision rate of CriticSMC is significantly lower, demonstrating the low overhead of using putative action particles.
Madaigreddictvee
3010
3300
SMC Pihdeet (2014)
3330
CrRisSCc
2n
Ecocaian
Modal-free Online
Colistnn Ma
SAC
2.5
RELATED WORK
3.22
SWC
SMC methods (Gordon et al. 1993 Kitagawa 1996 Liu & Chen 1998), also known as particle filters, a well-estabe essbblhhtel family of inference methods for generating samples from posterior distribu- tions. Their basic formulations perform well on the filtering task, but poorly on smoothing (Godsill et al.) 2004) due to particle degeneracy.
06
Execution time comparison between the baseline methods and CriticSMC for both model-predictive planning and model-frec online control. The collision infraction rate averaged across the 4 INTERAC- TION locations.
Figure
These issues are usually addressed using backward simulation (Lind- sten & Schön 2013) or rejuvenation moves (Gilks & Berzuini 2001
Andrieu et al. 2010). These solutions improve sample diversity but are not sufficient in our context, where normal SMC often fails to find
even single infraction-fres sample. [Lzarri et al. (2007) used SMC for learning actor-critic agents with continuous action environments. jet al. (2019) propose using the value function as a backward message SMC for planning. Their method is equivalent what is obtained using the equations from Figure 2b with hr = V($++1)
Similarly to CriticSMC, Piché
log Em... (exp(Q/ee (a 1))] (see proof in Section A.22 of the Appendix). This formulation cannot
accommodate putative action particles and learns parametric policy alongside V(s++1), instead of
applying the soft Bellman update (Asadi & Littman 2017 Chan et al. 2021 to fixed prior.
In experiments used the bootstrap proposal (Grdoon et al. 1993). which samples from the
prior model, but in cases where the prior density can efficiently computed, using a better proposal distribution bring significant improvements. Such proposals can be obtained in variety of
8

---

Published as a conference paper at ICLR 2023
Table 4: Infraction rates for the Hard-Q and the Soft-Q objectives tested on the location DR REUDEU Merging MT from the INTERACTION dataset (Zhan et al. 2019).
Collision Infraction Rate
Method
Critic Objective
MFD
ADEg
Progress
Prior
0.02522
2.2038
0.3024
16.43
Hard-Q Soft-Q
0.01911 0.01376
0,9383
1.0385
14.98
CriticSMC
2.1985
0.3665
15.91
ways, including using unscented Kalman filters (van der Merwe et al. 2000) or neural networks minimizing the forward Kullback-Leible divergence (Gu et al. 2015)) CritiCMC can accommodate proposal distributions, but even when the exact smoothing distribution is used as a proposal, backward messages are still needed to avoid the particle populations that focus on the filtering distribution.
As we show in this work, CriticSMC can be used for planning as well as model-free online control. The policy it defines in the latter case is not parameterized explicitly, but rather obtained by combining the prior and the critic. This is similar to classical Q-learning (Watkins & Dayan 1992), which obtains the implicit policy by taking the maximum over all actions of the Q function in discrete action spaces. This approach has been extended to continuous domains using ensembles (Deisenroth & Rasmussen 2 Ryu et al. 2020: Lee et al. 202I] and quantile networks (Bellemare et al. (017).
The model-free version of CriticMMC is also very similar to soft Q-learning described as described by Haarnoja et al. (2017) Abdolmaleki et al. (2018), and analyzed by Chan al. (202T).
Imitating human driving behavior has been successful in learning control policies for autonomous vehicles (Bojarski al. 2016 Hawke al., 2019) and to generate realistic simulations (Bergamini
ct al. 2021 Scibior et al., 2021. In both cases, minimizing collisions, continues to present one of the most important issues in autonomous vehicle research. Following data-driven approach, [Suo et al. (2021) proposed auxiliary losses for collision avoidance, while Igl et al. (2022) used adversarially trained discriminators to prune predictions that are likely to result in intractions. To the best of our knowledge, ours the first work to apply a critic targeting the backward message this context.
6 DISCUSSION
CritieSMC increases the efficiency of SMC for planning in scenarios with hard constraints, when the actions sampled must be adjusted long before the infraction takes place. achieves this efficiency through the use of a learned critic which approximates the future likelihood using putative particles that densely sample the action space. The performance of CritieSMC relies heavily on the quality of the critic and in this work we display how to take advantage of recent advances in deep RL to obtain one. One avenue for future work is devising more efficient algorithms for learning the soft Q function such as proximal updates (Schulman et al. 2017) the inclusion of regularization which guards against deterioration of performance late in training (Kumar et al.) 2020).
The design of CriticSMC is motivated by the desire to accommodate implicit priors defined as samplers, such as the ITRA model (Šcibior et al. 2021 we used in our self-driving experiments. For this reason, we avoided learning explicit policies to use as proposal distributions since maintaining similarity with the prior can be extremely complicated. Where the prior density can be computed, learned policies could be successfully accommodated. This is particularly important when the action space is high-dimensional and it is difficult to sample it densely using putative partieles.
In this work, we focused on environments with deterministic transition dynamics but CritieSMC could also be applied when dynamics are stochastic (i.e. 8t+1 p(* 88c, az)). In these settings, the planning as inference framework suffers from optimism bias (Levine) 2018 CCan et al. 2021 l).
even when exact posterior can be computed, which is usually mitigated by carefully constructing the
variational family. For applications in real-world planning, CriticSMC relies on having a model of transition dynamics and the fidelity of that model is crucial for achieving good performance. Learning such models from observations is an active area of research (Ha & Schmidhuberi 2018 Chua jet al. 2018 Nagahandi al. 2020). Finally, we focused on avoiding infractions, but CritieSMC applicable to planning with any reward surfaces and to sequential inference problems more generally.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs Program, and the Intel Parallel Computing Centers program. Additional support was provided by UBC's Composites Research Network (CRN), and Data Science Institute (DSI). This research was enabled in part by technical support and computational resources provided by WestGrid www.wreigddddca) Compute Canada www.conpptecanada.com and Advanced Research Computing at the University of British Columbia (arc.ubc.ca).
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL htw.t:tttps/.... http:p:s////peeeeviiee wtwpenrrwieeeeeeeeoorreeeee form/ www.rrrrmmmidSSAAAa 2id=$1ANxQWWO)
Christophe Andrieu, A. Doucet, Sumeetpal S. Singh, and Vladislav Z. B. Tadić. Particle methods for change detection, system identification, and control. Proceedings of the IEEE, 92(3):423-438, 2004. doi: 10.1 10.1109//PROC.223..333
Christophe Andrieu, Arnaud Doucet, and Roman Holenstein. Particle Markov chain Monte Carlo methods. Journal of the Royal Statistical Society:
Series (Statisti-
cal Methodology). 72(3):269 342, 2010. ISSN 1467-9868. 2009.007366 URL wtwes///araeeeebrrrr............ wttp::/wwwww.coooo///aa///////// htt.t///o/o////a/aa////////////r 1467- 68 2009. 00736.x 010.822000000736..
doi: 10.1111/j1166-98668
.eprint: http://aaiiiillaaaayyw../////iii
M. Sanjeev Arulampalam, Simon Maskell, Neil Gordon, and Tim Clapp. A tutorial on particle filters for online nonlinear/non gaussian bayesian tracking. IEEE Transactions on Signal Processing, 50 (2):174-188, 2002. doi: 1.111099789787777
Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedingy of Machine Learning Reseurch, pp. 243-252. PMLR, html
06-11 Aug 2017. URL wwt.hhttppp.. //tp:see//r/ceeeee//e ngs www.mllrpp.r... wwt..prees/////aaada..
Marc G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 449-4594888 458. PMLR, 06-11 Aug 2017. URL http:::/prooeeiiii.eeee .wr...m.rr..... www.mm..p....semmm bell mellemarel7a.. html
Luca Bergamini, Yawei Ye, Oliver Scheel, Long Chen, Chih Hu, Luca Del Pero, Blazej Osinski, Hugo Grimmett, and Peter Ondruska. SimNet: Learning Reactive Self-driving Simulations from Real-world Observations. arXiv:2105.. 12332 [cs). May 2021. URL httpp://axxvv.org
abs/ / 2105 2332 arXiv: 2233..
Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End End Learning for Self-Driving Cars. Technical Report arXiv:1604.0733 07.661166... arXiv, April 2016. URL http:://axx...rggg//////o// wwg.//iw..oor/aaaooooo abt.s///.woobbaa/...//. 07316 arXiv:1604.07316 [cs] type: article.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte
10

---

Published as a conference paper at ICLR 2023
Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir P. Mirchandami, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e. Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108.0722 2021.
Yoram Bresler. Two-filter formulae for discrete- time non-linear bayesian smoothing. International Journal of Control, 43(2):629 641, 1986. doi: 10. 100000000777886603333333303 URL https / /ttp://.111000002211118858833330
//doi.org//i
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. 1606 01540 [cs], June 2016. URL htttp://axxi>
org/abs/1600 6006 01540 arXiv: 1606.01540.
Olivier Cappe, Simon J. Godsill, and Eric Moulines. An overview of existing methods and recent advances in sequential monte carlo. Proceedings of the IEEE, 95(5):899 924, 2007. doi: 10.1 10.1109/ JPROC.2007.8932550
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. arXiv preprint arXiv:2107.0822 07.8088...8....... 2021.
Nicolas Chopin, Pierre E. Jacob, and Omiros Papaspiliopoulos. Sme2: an efficient algorithm for sequential analysis of state space models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 750)):97-426,, Oct 2012. ISSN 3412-7122 doi: 10. 1/j. 1467-9868 httt::://xx.oi.oor/o/ 2012.01046.x. StLhtttp:10 dx doi wwt.ooggg.ooo.o...o 1ttp///0..111111111 1467 9868 2012 01046.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-II), pp. 465-472 Citeseer, 2011.
R. Douc and O. Cappe. Comparison of resampling schemes for particle filtering. ISPA 2005. Proceedings of the 4th International Symposium Image and Signal Processing and Analysis, 2005., 2005. ISSN 1845-5921. doi: 10. 1091100p..2005..19555.
URL http::/dx.. doi. org/
10. 10.1109/ SPA. 2005. 195385
Randal Douc, Aurélien Garivier, Eric Moulines, and Jimmy Olsson. Sequential Monte Carlo smoothing for general state space hidden Markov models. The Annals of Applied Probability, 21(6):2109 2145, 2011. doi: 0.1211410-AAP735, I0-AAP735
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110())2419 22688848888 2021.
Paul Fearnhead. Particle filters for mixture models with an unknown number of components. Statisties and Computing, 14(1):11-21, January 2004, ISSN 0960-3174. doi: DLL.22/BBSTTO..000000.. 0.621.cd.
11

---

Published as a conference paper at ICLR 2023
Paul Fearnhead and Peter Clifford. On-line inference for hidden markov models via particle filters. Journal of the Royal Statistical Sociery: Series B (Stttistical Methodology), 65(())887- 899, 2003. ISSN 13697412, 14679868. URLhttp::/wwe //w.: ]stor. org/ wtrg/stablee
Jakob N. Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric P. Xing, and Shimon Whiteson. DiCE: The Infinitely Differentiable Monte Carlo Estimator. In ICML, pp. 1524-
1533, 2018. /tthhttp://grreeeeiinng ml http:////.ooreeeein......ee/eees wtwrs//eoeeeeeeeeeeee 8a. html
Walter R. Gilks and Carlo Berzuini. Following a moving target- Carlo inference for dy-
namic Bayesian models. Journal of the Royal Statistical Society: Series B (Statistical Method-
ology), 63(1): 2001. ISSN 1467-9868. doi:
URL https .eprint:
//orlinelibrary.w/ wi www.w..ww...m................... com dww.://ooi/aae http://auttiiiinn......ooooooooo
111...111111111111111111 00280
Simon
Godsill, Amaud Doucet, and Mike West. Monte carlo smoothing for nonlinear time series. Journal of the American Statistical Association, 99(465):156-166 2004. ISSN 01621459, URL
http: / www jstor.
N.J. Gordon, D.J. Salmond, and A.F.M. Smith.
Novel approach to nonlinearrnon-gguusian bayesian state estimation. IEE Proceedings F (Radar and Signal Processing). 140:107-113(6), April 1993. ISSN 0956-375X. URL httttt:///tgtttttat journa 1 8/ 0. 049/ip-f-2.... 993. 0015
Shixiang Gu, Zoubin Ghahramani, and Richard E. Turner. Neural Adaptive Sequential Monte Carlo. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS 15, pp.
Cambridge, MA, USA, 2015. MIT Press.
David Ha and Jürgen Schmidhuber. World models. 2018. doi: 15221//ZENODO.12000 URL https: zenodo.
htops::/zzeodd..o wttp://rr/reeeooeeeeeeeee/o
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1362-1361. PMLR, 06-11 Aug 2017. URL https: htttp:://proeedings. mlr press//70/haarojjal
html
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018a. URL
https::/pproeeednngg www.inngssm.......mmmm mlr press wtteee/s/////aaaajjaaaaa... html
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05500
2018b.
Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, and Alex Kendall. Urban Driving with Conditional Imitation Learning. Technical Report arXiv:1912.00177 arXiv, De- cember 2019. URL article.
00177 arXiv:1912.0017 [cs] type:
Jeffrey Hawke, E Haibo, Vijay Badrinarayanan, and Alex Kendall. Reimagining an autonomous vehicle. ArXiv, abs/2108. 18.005805, 2021.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018
Maximilian Igl. Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulation. Technical Report arXiv:2205.033 195. arXiv, May 2022. URL http::////// http:////xxi.../ wttps///.rrrabb///e/////////// 03195 arXiv:2205.0m .22550033311 195 [cs] type: article.
12

---

Published as a conference paper at ICLR 2023
Ashesh Jain, Luca Del Pero, Hugo Grimmett, and Peter Ondruska. Autonomy 2.0: Why is self-driving always 5 years away? ArXiv, abs/2107 142, 2021.
Hilbert Kappen, Vicenç Gómez, and Manfred Opper: Optimal control as a graphical model inference
problem. Machine learning, 87(2): 2012.
Geon-Hyeong Kim, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Variational infer- ence for sequential data with future likelihood estimates. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5296-5305. PMLR, 13-18 Jul 2020. URL https: http:://proceednng.
www.. www.ppressvilll. Dttess//119/kim20.. wttp//imm00....mmmmm html
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//arxiv.org/ abs/ 1312.612222 6114
Genshiro Kitagawa. The two-filter formula for smoothing and an implementation of the Gaussian-sum smoother. Annals of the Institute of Statistical Mathematics, 46(4):605- 623, 1994. URL
repec. retpc..org/eP.ccsrrrr.rr ststt:::::t:::
1994:1: 4::: 623
Genshiro Kitagawa. Monte carlo filter and smoother for non-gaussian nonlinear state space models. Journal of Computational and Graphical Statistics, 5(1):1-25, 1996. doi: 10. 1080/000011110 8600. 1996. 10474692. URL wtttt:://///.. //wp:::::/www....pp.. tandfon ine..... com/ /aas//bs/////// 100.0000001061600
1996. 10474692
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine, Implicit paramerarrattttiizaatonn inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 14498, 2020.
Jonathan Wilder Lavington, Michael Teng, Mark Schmidt, and Frank Wood. A closer look at gradient estimators with reinforcement learning as inference. In Deep RL Workshop NeurlPS 2021, 2021.
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Reinforcement learning in continuous action spaces through sequential monte carlo methods. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran As- sociates, Inc., 2007. URL https: http:///cccccccce.eeeeee 0f 0be 0FF40000222004447556222222225523
newp:eeeeeeeeen.ee.nee.eee..eeee neurips http:ccee/po000.777..///./ /000///e///0///1l/// file/
pdff
Tuan Anh Le. Unbiasedness of the Sequential Monte Carlo Based Normalizing Constant Esti- mator. hctps::/ww// StE.::://ww.tuna.nne tuananhle co. www.kkktteeeee wtwp:///eeeeeeeeeeeeeeeebeebeeee
html 2017. Accessed: 2022-04-29.
Kimin Lee. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pp. 6131-6141. PMLR, 2021.
Sergey Levine. Reinforcement Learning and Control Probabilistic Inference: Tutorial and Review. [cs, star], May 2018. URL http://aarxiv..rrr/bbb//855..0.0
arXiv: 1805.00909.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico May 2-4, 2016, Conference Track Proceedings, 2016. URL
http://arxiv>: org/ abs 15099 02971
Fredrik Lindsten and Thomans B. Schön. Backward Simulation Methods for Monte Carlo Statistical Inference. Foundations and Trends in Machine Learning. 2013. URL
ore
Soeee.zzg/dccumentt //tttt////18888
Vasileios Lioutas, Adam Scibior, and Frank Wood. TITRATED: Learned human driving behavior
without infractions via amortized inference. Transactions on Machine Learning Research, 2022.
URL /tt.httttps:://perreviii net. twtt:formmmiiiom (ot.forrmmmmmm05552250
13

---

Published as a conference paper at ICLR 2023
Jun S. Liu and Rong Chen. Sequential monte carlo methods for dynamic systems. Journal of the American Statistical Association, 93(443):1032 1044, 1998. doi: 10.1080001621459.1999 10473765.
//tc///oiio wwo. 100.010000/0000 0ww.000001112115100 1998 10473765
P.D. Moral, Doucet, and S.S. Singh. Forward smoothing using sequential Monte Carlo. CUED/F-
INFENG/TR. University of Cambridge, Department of Engineering, 2009. URL https://
books. . gww.le.oooooooooooooooo wwwbooks/bboo.ccocoooooo..... 2xC.
Pierre Del Moral. Feynman-Ken Kac Formulae: Genealogical and Interacting Particle Systems with Applications. Probability and Its Applications. Springer- Verlag, New York, 2004. ISBN 978-0- 387-20268-6. doi: 10. 1007/978- 1-77788--4684--- 1393--.3--.. URL httph::///wwtt springerie com/ gp/
fook//780387202665
Christian A. Naesseth, Fredrik Lindsten, and Thomas B. Schön. Sequential monte carlo for graphical models. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp. 1862-1870, Cambridge, MA, USA, 2014. MIT Press.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR, 2020.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations, In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 6292 6299, 2018. doi: 10.1109/ICRA.211
8463162.
Gerhard Neumann. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pp. 817-824, 2011.
Alexandre Piché, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal. Probabilistic planning with sequential monte carlo methods. In International Conference on Learning Represen- tations, 2019. URL
net wtw.e/wuneee/eeemmteemm Gnney
Michael K. Pitt, Ralph dos Santos Silva, Paolo Giordani, and Robert Kohn. On some proper- ties of markov chain monte carlo simulation methods based on the particle filter. Journal of Econometrics, (2):11334-13)) 2012. ISSN 0304-4076. doi: https://doi.or//00.1111 (ccomom.2012..66.m URL https: // //www. sciencedizect. com scieecc/ecccttccee////////////// pii/ p11/503040761201150 Bayesian Models, Methods and Applications.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
Daniele Reda, Tianxin Tao, and Michiel van de Panne. Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning. In Proc. ACM SIGGRAPH Conference Motion, Interaction and Games, 2020.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4344 4353. PMIR, 10-15 Jul 2018. URL TtI.https: httttp:://proceedingg mlr.press/v8D
1t html
Moonkyung Ryu, Yinlam Chow, Ross Anderson, Christian Tjandraatmadja, and Craig Boutilier. CAQL: Continuous Action Q-Learning Technical Report arXiv: 1909. 12397, arXiv, February 2020.
URL httphhttp///xii..rrrtt/ absa 12397 1909 [cs, stat) type: article.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, Puerto Rico, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
14

---

Published as a conference paper at ICLR 2023
Adam Scibior, Vasileios Lioutas, Daniele Reda, Peyman Bateni, and Frank Wood. Imagining The Road Ahead: Ageti-Agent Trajectory Prediction via Differentiable Simulation. 2021 IEEE 24rd International Conference Intelligent Transportation Systems (ITSC), 2021.
Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posterioni policy optimization for discrete and continuous control. arXiv preprint wxXi::909.112335 2019.
Andreas Stuhlmüller, Robert X. D. Hawkins, N. Siddharth, and Noah D. Goodman. Coarse-to-Fine Sequential Monte Carlo for Probabilistic Programs. arXiv preprint M.XXvx:559.002000 2015. URL
http://arxi:: http/r/gg/////////
abs wttp:1509...0 02962
Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. TrafficSim: Learning to
Simulate Realistic Multi-Agent Behaviors. 1rXiv:2101.0655: January 2021. htttt
//arxiv..rg/ab///2221. 06557
06557
arXiv: 2101
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033.
2012. doi:
Rudolph van der Merwe, Arnaud Doucet, Nando de Freitas, and Eric Wan. The Un- scented Particle Filter. In Advances in Neural Information Processing Systems, vol- ume 13. MIT Press, 2000. URL https: httppppppppsps...... wtt.ipps.ccpppaeee httpp//s....pppee//////aata Abstract.html Abbsseeeet...t.ttttt..........
Christopher C. Warkins and Peter Dayan. Q-learning. Machine Learning, 83):279-292, May 1992. ISSN 1573-0565. BFC0992698
doi: 10. 10.1007/BF009992698
/htttps://do..or//
Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kümmerle, Hendrik Königshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTER- ACTION Dataset: An INTERnational, Adversarial and Cooperative moTIOON Dataset in Interactive Driving Scenarios with Semantic Maps. 1910vv191.003000 [cs, eess), 2019.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICMI., 2010.
15

---

assume the state Published as a conference paper at ICLR 2023
A APPENDIX
A. SEQUENTIAL MONTE CARLO
Here we briefly give an overview of the sequential Monte Carlo (SMC) algorithm adapted for Markov decision processes (MDPs). We borrow the notation from Section 2 in the main paper. Obtaining state-action pairs (81:7; 01:t) that maximize the expected sum of rewards corresponds to sampling state-action pairs from the posterior P(51.7, p($1.T,01.T(O (0.))..))) The SMC inference algorithm (Gordon et al. 1993) approximate the filtering distributions p(st, 0e) (1:)). In general. SMC assuming the existence of proposal distribution g($++1, Qt-1/St, a₂) but for simplicity
instead use bootstrap proposals that use the prior policy. The algorithm samples N independent particles from the initial distribution sf (o($1) where each particle has uniform weights wi 1//V. At each iteration, the algorithm advances each particle one step forward by sampling actions from a. ~ I T(a (s)) and then compute the next states by 86+1 p(s++1/ ss, af) accumulating the optimality likelihoods p(O] , . ((+1) = "( .6. .8+1) in the corresponding particle weight, saving the sum weights and normalizing them before proceeding to the next time step. In our setting,
dynamics 0($ az) of the environment to be deterministic 8+41 < f(,, az). SMC suffers from weight disparity which can lead reduced effective sample size of particles. This is mitigated by introducing a resampling step RESAMPLES i. I:N] at every iteration to help SMC select promising particles with high weights that have higher chance of surviving whereas particles with low weights most likely will get discarded. See Douc & Cappe (2005) for an extensive overview of different resampling schemes. Algorithm summarizes the SMC process for MPDs using bootstrap proposals.
Algorithm 2 Sequential Monte Carlo
procedure SMC(po. f. f. N, T)
Sample !-N Po((1)
Set wo 1:N
for E 1......... do
for E 1...N do
Sample 0i ~ T(a st)
Set 8 t [( a.)
Set op + 1o ee (c a7 (
end for
Sct W.
N wi
Sample at 1:N RESAMPLE
1:V W.
for n E 1. 1...N do
Set at +- 2.46
Set sh 4- 2
Set of < We
end for end for
I.N I:N I:N return 81 012 W1T end procedure
A.2 DERIVATIONS
Soft-Q function Below is the derivation of the soft-Q function defined the log probability of the backward message.
(str de) log logp(Or.r(st,. az)
= log DOgp(Orrss, ar) + log p(0 ac),
(6)
where
log pa (())(st;
E
(r(st, ae, 8t-1)];
(7)
16

---

Published as a conference paper at ICLR 2023
and
=log / [
1175)))))))) =
log
p(8t+118t, (x) (8+
si41 Jayt1
log
E 1 poplp 1 (ssa))) (0
E
(8)
If we assume the dynamics p(4 Stc ar) of the environment to be deterministic 8(+1 + f(s), we can simplify Equation
Q(sr, az) r(se, ae, 8(41) log
E (a4
(9)
SMC using value function as heuristic factors Piché et al. (2019) proposed using state values (sx) as backward messages in SMC for planning. Based on the two-filter formula (Bresler 1986
Kitagawa 1994). they derive the following weight update rule
exp (r(st, S(+1) + V($++1) log E lexp (V(s)))
w. W1-1
E 4141 p ol))
(10)
We omit the term - log my va(ayls (ag)))) since we assume to use bootstrap proposals instead of learning them. Thus, the current action sampled as a (aa(st)) our framework, for simplicity, we assume the use of deterministic state transition dynamics p(st+1|s2,,, az) which simplifies the update rule
Wt Wt-1 1 (((tt, at, 81+1) + V(8++1) - V(si).
(11)
Piché al. (2019) in practice trained an SAC-based policy and used the learned state-action value functions Q (st dz) to approximate state values (s²)). Following a similar experimentation setting, we use soft approximation of the value function terms using state-action value functions Q(st, az) as described by Levine (2018) using
(sx) = log E
(exp(Q(s;, a)))
(12)
This results in the following partiele weight update rule
W/ = wt-1 exp r(si, at, S1+1) + log E (1110(012151111
[exp(Q(s411 ((11))
log E (exp(Q(st, &d]])
â ()==(a())))
(13)
We can then define the heuristic factor used in Figure 2b of the main paper
hg = log E (exp(Q(s(+1, ()11 log E [exp(Q(st,, @
(14)
d 1 (
which utilizes soft approximation of the value function. It is worth emphasising that next state sample 8141 from the environment model is required which makes the use putative action particles (see Section 3.3 of the main paper) inefficient and expensive contrary to the proposed Cri method.
A.3 TOY ENVIRONMENT EXPERIMENT DETAILS
In this environment. the ego agent is described by e, (zf,z6, where x, y is the position in the square coordinate system [0, 1]² and rd is the radius. We randomly position other agents of (= yot po') where i [0, 5]. In addition, there is partial barrier in the middle with gates gh = (rg* yo* w9* where r yo* are the coordinates of the center of the gate k, w is the width of the opening and h E [1,3]. Finally, goal position G (x y ,) is positioned the other side of the barrier. The ego and the other agents are moving by displacement actions
a, Ar, A..).
The state representation consists of relative distances between the ego agent and the other agents, the center of the gates and the goal position. A two-layer fully connected neural network with ReLU
17

---

Published as a conference paper at ICLR 2023
activation function and size of 64 takes input this representation and produces state encoding. A similar network takes as input the wo-diiensiooal displacement actions and produces the action encoding. Finally, another two-layer network takes as input the concatenation of the state and actions encodings and produce the Q values.
We train the model using single Nvidia RTX 2080Ti GPU. The prioritized experience replay buffer has a size of million stored experiences. The discount factor is set to 0.99, the batch size to 256 and the learning rate to 0.001. Finally, we sample 1024 actions during running CriticSMC while training the critic model.
A.4 DRIVING BEHAVIOR MODEL EXPERIMENT DETAILS
The prior model we picked for this experiment is ITRA (Scibior 2021 but any other probabilistic behavior model can be used. We follow the same architecture and training procedure as described in Seibior et al. 2222)). The prior model is trained on the INTERACTE ACTION (Zhan et al. 2019) dataset and the task is that given 10 timesteps of observed behavior, predict the next 30 timesteps of future trajectories. For the critic, we used the same convolution neural network architecture as the prior model. The critic takes as input the last two observed birdviews images and encodes them separately. The concatenation of the two representations along with the action encoding is processed by a final layer that produces the Q value. The architecture for these layers is the same as in Section [A.3]
We train the critic model using single Nvidia RTX 2080Ti GPU. The prioritized experience replay buffer has size of 1.5 million stored experiences. The discount factor is set to 0.99, the batch size to 256 and the learning rate to 0.001. Finally, we sample 128 actions during running CriticSMC while training the critic model.
A.4.1 REINFORCEMEN LEARNING ENVIRONMENT
The environment used to train the RL agents takes as input a location from the ACTIIOCTON dataset and trains a single-agent policy where all non-ego actors rollout according to ground truth. Because the CriticSMC algorithm rolls out every agent according to ground truth for the first ten frames of each trajectory before prediction, we simply remove these frames and begin executing the policy on frame eleven. At time step t, the policy takes the previous and current birdview images (b,-1, bt) where each image has a size 256 x 256x1. 1. The stacked birdview images make the total input for the policy and value function to be 2x 2x266x26- 1. The policy produces an action a, E 2[-1,11 2 which corresponds to the bicycle kinematic model's relative action space (see Sciiior
ct al. 2021 for more details). The differentiable simulator (Scibior et al., 2021 then uses a, to update its state and returns the next birdview image br+1. In this setting the policy distribution that is learned follows a squashed normal distribution (Haarnoja et al., 2018b), as is standard for the SAC implementations (Haaroja et al. 2018b). The stochastic policy learned by SAC is tailored towards exploration and thus behaves poorly. For this reason we only report its deterministic behavior (e.g. the mode of the policy) in Table 2 of the main paper. For each of the four locations that were evaluated, the RL agents were run over three different learning rate schedules and three different reward structures for minimum of 150k time steps. The policy uses the same convolutional neural network architecture as in CriticSMC and is updated according to the soft actor-critic algorithm in
(Brockman et al. 2016). Table 5 shows the hyper-paue parameter settings used for
training.
Reward Surfaces In the three rewards settings which we tested, there were a number of different feedback mechanisms which were used to produce the desired behavior (i.e. low-collision probability and low ADE). The first, was a score based reward upon an estimate of the log log-probability under ITRA. To compute this "score reward", the environment passes the pair of birdview images (b,, bt+1) to ITRA, which generates the hypothetical action af a!IRA that ITRA would have taken to make the state transition from by to bz Then, the environment sets the reward to be a monotonic function of the likelihood of a under a normal distribution centred around az ITRA, r++1 tanh (log p(a;; at where p(: a.TTAA,
ITRA E))
ITRA, Σ) = Aa(aTTRAA Σ) for some covariance Σ.
Next, we include five simpler reward surfaces which have been shown to improve performance in the literature (Reda et al.] (2020). First, the "action reward" is a linear function of the absolute difference between the action output by the policy and the action which ITRA would have taken time step t:
18

---

Published as a conference paper at ICLR 2023
Reinforcement Learning Baseline Parameters
Parameter Name
Parameter Value(s)
Parameter Description
1
The covariance matrix of the multivariate
normal distribution centred around hypo- thetical ITRA action a,
01
0.15
Coefficient for score reward, this parameter scales how closely the agent should track estimated log-likelihood of actions under the ITRA model.
0/2
2
Coefficient for action reward, this incen- tives the policy be as close to the mode of ITRA as possible without access to score function over those actions.
03
0.05
Coefficient for action difference reward, this incentivizes the agent to produce se- quences of actions which are smoother, and therefore often more human-like.
Qr4,5,6
0 or
Boolean coefficients selecting whether in- fraction, survival, or ground truth rewards are used.
Y
0.99
Discount factor, set encourage lower vari- ance gradient estimates, but greedier policy behavior (Sutton & Barto 2018).
Learning Rate
0.0002,
0,00012,
Learning rate for optimization (in this case
0.00008
the Adam Optimizer).
Batch Size
256
Number of examples used in each gradient decent update for both the critic and policy networks.
Buffer Size
500000
Size of SAC experience buffer (equivalent maximum number of steps which can be taken within the environment).
Learning Starts
1000
Number exploration steps used (e.g. a uniform distribution over actions) before learned stochastic policy is used to gather interactions.
0.005
Polyak parameter averaging coefficient which improves convergence deep Q learning algorithms (Haarnoja et al., 2018b).
Latent-Features
256
Number of neurons used in the output of the feature encoder, and which is fed the standard two layer multi-layer per- ceptron defined by standard SAC algo- rithms (Haarnoja et al., 2018b).
Table 5: Hyper- pepeerpprrametes for the reinforcement learning baseline used in Section 4.2. All hyper- parameters which were not listed above, use the default values provided by the SAC implementation of stable-baselines3 (Brockman et al. 2016).
(2- a [T) Second, the action difference reward is the scaled absolute difference between the current and previous actions: ||u, 0t-1l11- Third, the environment computes the "ground-truth reward" T by evaluating 8; against the ground truth data from the INTERACTION dataset. In particular, the environment sets the reward to be linear function of the negative Euclidean distance time + 1 between the xy-coordinate the geo-vch according the simulator, $1+1, and that according to ground truth, 8 100 - || SGT /l2. Fourth, we include a "survival reward" of the agent does not commit an infraction step f. Lastly, the infraction reward is -5 if the agent commits any type of infraction at step and 5 otherwise.
19

---

Published as a conference paper at ICLR 2023
Using these five feedback mechanisms, we consider three different reward surfaces. Each of which are defined following reward calculation:
T = 011++1 1+1
SCORE
+
ACTION
[+1
+
013'++1 i+1
ACTION DIFF
0.4 1+1
INFRACTION + 2+1 + 06G
SURVIVE
GROUND TRUTH
(15)
In the first reward setting which was considered, we set all coefficients og to zero except the SURVIVE reward, and thus refer to this reward type as the survival reward setting, Next we consider setting where we set all to zero except the GROUND TRUTH reward, and refer to this setting as the ground-trune setting. Lastly, we considered setting where: an 0.15, 02 2.0, Q3 0.05, and the remaining o; are all set to zero. We refer to this setting as the ITRA setting, as it includes the most information about the ITRA model. arrive at the final result, models where trained under all three of these settings, evaluated, and then chosen based upon the lowest collision infraction rate.
A.5 CRITICSMC AS AN EFFICIENT SMC INFERENCE ALGORITHM
We include in the supplementary material a demo code implementation of CriticSMC applied to the following linear Gaussian state-space model (LGSSM) with well-defined critic function
fst, az) $t +
(16)
p(50) = N\0,1)
(17)
p(ae(s:) .O..5 St, 1)
(18)
p(s 1 az) = ) ($++1)
(19)
0,
if - 10 otherwise
S 8141 51 10-2
log,(O,p (0lpp ue, 8141)
(20)
100000
(8); az)
(21)
2 1000 + azl +
(22)
where the state transition function f(s,, az) assumed to be computationally expensive. The
conditional posterior samples from p p($1:1, (1:1/O1:r) are defined as states that are within the range We use T = 10 our experiments.
defined in Equation 20
Figure 5 demonstrates the performance of CriticSMC compared to SMC for estimating the (negative) mog-maaginal likelihood p(O1.T) relative to the computational time needed to execute the inference algorithm.
20

---

Published as a conference paper at ICLR 2023
Number of Particles:
CricsMC SMC
criscsMC
SMC
Negaiive Log.-aooooaaaaaage Manginal Liilbood
1000
Number Action Patticass
Number Action
Number of Particles:
CHRRsSMC SMC
CritisMCC
SMC
Neqalive Log.taaginall Liaiiteod
10"
Meeurnc Compu aicanall Time
10'
10-
Action Particies
Number Fartictes
Number of Particles: 10
CHRtSSMC SMC
CritisMC SMC
Regaive Margnall Lielhood
10'
Compo tatiraa Tim
10-1
Figure 5: Given a well-defined linear Gaussian state-space model, we evaluate the performance of SMC and CritieSMC estimating the (negative) log-marginal likelihood (bar plots on the left column) relative to the speed of inference measured wall-clock time (bar plots on the right column). We pick the number of particles as N {1, 5, 10) and the number of putative action particles as K (1, 100, 1000). CriticSMC is able to better estimate the marginal likelihood significantly faster than SMC by taking advantage of a large population of putative action particles and computationally efficient critic function used as heuristic factor to guide inference.
21

---

Published as a conference paper at ICLR 2023
A.6 NOTATIONS AND ABBREVIATIONS
$1.7
sequence of states
d1.7
sequence of actions prior policy
(t)(st)
p(5,+1|50,00 at)
state transition dynamics density
r(st, ae, ai, Sc+1)
reward value received timestep
p(Oz|st, ae sc+1)
optimality probability defined as the exponentiated reward soft state-action value function referred to the critic parametric approximation of the critic
Q(5l, ai)
Q+
Q
fixed target critic model used for computing the TD error heuristic factor at timestep t pre-resampling particle weight post-resampling particle weight normalizing factor ancestral indices for each particle i
he
it
We
of
Ppen
penalty coefficient
?
discount factor
T
horizon length
1 1
T
timesteps
n N
particle number
k 1 K
putative action particle number
Table 6: Notations
SMC:
Sequential Monte Carlo
MDP:
Markov Decision Process
RL:
Reinforcement Learning
HMM:
Hidden Markov Model
RLAI:
Reinforcement Learning as Inference
TD:
Temporal Difference
SAC:
Soft Actor Critic
MPC:
Model Predictive Control
ADE:
Average Displacement Error
MFD:
Maximum Final Distance
Table 7: Abbreviations
22

---

Published as a conference paper at ICLR 2023
Figure 6: More examples similar to Figure 3 of the main paper.
23


Reviews:
[{"review_id": "ZXR8ZA6VrL3", "reviewer": "Reviewer_9uTg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a sequential monte carlo (SMC) algorithm for planning. The main contribution is to define the particle weights using the Q(s, a)-function, which allows using a large number of actions for the same state without having to evaluate the transition model for each.", "summary_of_the_review": "The paper proposes a simple method which has tangible benefits. The presentation could be improved to make limitations clearer (i.e. the optimism bias) and the experimental evaluation could be strengthened (e.g. by comparing to [1] using parts of their experiments section) but the paper still makes contributions that are worthy of acceptance.\n\n---\n\n[1] Probabilistic Planning with Sequential Monte Carlo Methods, https://openreview.net/forum?id=ByetGn0cYX", "strength_and_weaknesses": "**strengths**\n\n* The paper is proposing a simple modification of regular SMC (in the context of planning) which brings two benefits: a computational one (that we can use a larger number of particles for the same number of transition evaluations) and a conceptual one (that the resulting algorithm can be used in a model-free fashion).\n* The paper contains results on refining a prior policy, which is an interesting scenario that is somewhat underrepresented in current research.\n\n**weaknesses**\n\n* Given the method's proximity to [1], it is unfortunate that the paper doesn't contain a comparison to [1] in the same experimental suite or an overlapping one. That makes it harder to compare the two methods.\n* The baselines considered in the experiments are: ITRA, SAC, SMC and rejection sampling. The only baseline which has previous published results on these problems is ITRA, which the proposed method does outperform, but that is also to be expected since the authors are building on top of ITRA. In other words, the paper is missing a baseline which has previous published results on this experiment suite. In my opinion this is the paper's biggest issue at the moment.\n\n---\n\n[1] Probabilistic Planning with Sequential Monte Carlo Methods, https://openreview.net/forum?id=ByetGn0cYX", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FK92egiBKqu", "reviewer": "Reviewer_toeN", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Critic Sequential Monte-Carlo approach, a simple extension of the Sequential Monte-Carlo method, in the framework of \"planning as inference\".\nThe approach combines the sequential Monte-Carlo technique with heuristic factors computed using a soft q-function.\nThe authors then test the framework in two safety-related tasks, a multi-agent point-mass environment, and a driving simulator.\n", "summary_of_the_review": "While this paper seems to be overall a decent contribution, it is too incremental. To accept incremental work, I would expect clear performance improvements, a strong experimental campaign, vast ablation studies, or interesting insights from the theoretical standpoint.\nThis paper misses all the points above.\n\nFurthermore, the authors are using safety scenarios as a test bed. There is an entire literature on safety in Markov decision processes:\n1) The Constrained MDP framework (CMDP)\n2) The Control barrier function approach\n3) The shielding approach\n4) Reachability analysis\n\nMany of these methods are also model-based and are able to ensure safety while optimizing a policy. It is easy to perform slight modifications to these methods and could fit perfectly the target domains (I suspect these methods will outperform the proposed approach in the given tasks)\n\nI suggest the reviewer either expand the set of tasks to show more generality of their approach (they claim it, but is not shown) or add safe RL baselines to the comparison. For sure, if the focus of the paper is safety, at least a discussion on safe RL methods should be fundamental, explaining why a \"planning as inference\" approach should be better than methods explicitly designed to enforce safety.", "strength_and_weaknesses": "- The paper is sufficiently well written\n- The paper is too incremental\n- The paper lacks a proper experimental analysis on standard and complex benchmarks: the toy example is too simple, and the simulator seems too complicated and convoluted to understand the algorithm's behavior.\n- it is missing a strong theoretical contribution to the method, right now it seems to be mostly a heuristics\n- The paper builds on pretty strong assumptions: known model and deterministic dynamics\n- The paper tries to tackle a typical problem of safe reinforcement learning, without citing any of the approaches and papers on the areas.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EjzifjZOw7", "reviewer": "Reviewer_ViDn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a solution to the planning as inference problem in domains where it is easy to generate samples of actions in the current state but it is expensive to simulate samples of next states. The proposed solution involves learning a parametric approximation of the soft value of each state-action pair by using gradient descent. The paper also shows some examples on a self-driving car datasets and demonstrate better results. There are also some ablation studies.\n\nThe main contribution of the work is the observation that in some domains it is hard to simulate the next state, but very cheap to sample an action in the current state. This implies that the existing methods in the literature of learning a soft value of a state is not as helpful as learning the soft-value of a state-action pair (the so called critic function). The authors demonstrate this point by devising a particle filter-based algorithm that uses hypothetical particles in the training episodes to learn the critic function quickly.", "summary_of_the_review": "Slightly novel contribution in the area of planning as inference using deep learning with some parts unclear, but in general a reasonable contribution.\n", "strength_and_weaknesses": "The paper is written quite clearly and the main point is very easy to grasp. This is a very logical extension of the previous work by Piché et al 2019 which learns the soft-state-value function. The idea of using putative particles to learn the soft-state-action-value function aka the critic is quite well motivated.\n\nWhat I have would have liked to see is some tradeoffs between using a large number of putative particles versus using a few putative states. In the work of Piché only one putative state is used, but one could consider a simple extension of that approach where multiple putative states are used. I understand the paper's contention that putative actions are cheaper than putative states, but putative actions are not entirely free either.\n\nNevertheless, there are clear benefits of doing model-free learning unlike the previous approach of Piché.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-3KnnMJjc9", "reviewer": "Reviewer_qbn3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for planning-as-inference in reinforcement learning problems with sparse rewards. A method based on sequential Monte Carlo is derived, similar to Piché et al. (2019), to sample from the optimal state-action trajectories distribution by applying filtering and smoothing techniques. The algorithm uses estimates of a soft Q-function as a critic to provide heuristic factors when resampling actions according to their optimality within the SMC loop. In contrast to prior work on SMC for planning in RL, CriticSMC uses a fixed prior policy, instead of learned one, and uses the learned Q-function to guide action sampling, which simplifies the training process.", "summary_of_the_review": "The paper's contribution seems novel, but aspects of the methodology are confusing and the experimental comparisons should be expanded to better support the claims.\n\n#### Rebuttal update\nI have raised my score (5 to 6) after reading the authors' response.", "strength_and_weaknesses": "# Strengths\n* The idea of using Q-functions to guide sampling in SMC when observations are sparse is interesting and might be useful beyond the RL and planning contexts.\n* The proposed algorithm shows significant performance improvements in a toy collision-avoidance experiment and a more realistic one using a benchmarking dataset when compared to a baseline human-like driving policies and soft-actor critic.\n\n# Weaknesses\n* **Background.** The background on SMC for planning-as-inference is quite confusing for someone familiar with SMC and control, but unfamiliar with this line of work applying SMC to planning. Questions I had at a first pass: *Is it only simulating states in parallel and \"observing\" the optimality indicator*? *How do you use the sampler with a (physical) agent*? There is no clear distinction between **what's done in simulation and what's applied in the actual environment** by the agent. I had to go and read Piché et al.'s (2019) work on SMC for planning to understand that SMC is playing the role of a sampler from the optimal state-action trajectories distribution. The sampled optimal action trajectories can then either be executed blindly by an agent in the target environment after sampling or be used in a model predictive control (MPC) fashion where only the first action is applied to the environment, and from the resulting next state a new SMC planning procedure unrolls.\n\n* **Sec. 3.1: Deterministic vs. stochastic state transitions.** The soft Q-function approximation in Equation 3 relies on the assumption of deterministic state transitions, but the algorithm descriptions in both Figure 2 and Algorithm 1 use stochastic state transitions. It also gives the impression to me that deterministic transitions could lead to some degenaracy in the SMC scheme. However, this mixed presentation in the methodology is confusing and does not explain how deterministic transitions may affect sampling performance in practice.\n\n* **Sec. 3.2: Data for training.** It is not clear to me what is the used to learn the parametric Q function estimators in Eq. 4. No reference for what *state-action occupancy* (SAO) means is provided. I am not sure if it is based on the experience data (from the target environment) observed up to a given point in time, when executing online, or purely on simulation data using the world model that is applied within SMC.\n\n* **Related work.** It might be worth contrasting the paper with some earlier work on SMC for RL:\n\n    Lazaric, A., Restelli, M., & Bonarini, A. (2007). Reinforcement Learning in Continuous Action Spaces through Sequential Monte Carlo Methods. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in Neural Information Processing Systems (Vol. 20).\n\n* **Experiments.** The method is proposed as a general framework for planning in environments with sparse rewards, but it is only tested on two autonomous driving environments. It would be beneficial to compare its improvements over state-of-the-art on RL methods for sparse rewards in other classic environments used in this line of work, such as Atari games. Otherwise, the claims of the paper should at least be adjusted to reflect the focus on autonomous driving.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster