SPO: Sequential Monte Carlo Policy Optimisation
Authors: Matthew Macfarlane
Edan Toledo
Donal John Byrne
Paul Duckworth
Alexandre Laterre
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Policy Optimisation
Planning
Model-based RL
Primary area: reinforcement_learning

Abstract: Leveraging planning during learning and decision-making is central to the long-term development of intelligent agents. Recent works have successfully combined tree-based search methods and self-play learning mechanisms to this end. However, these methods typically face scaling challenges due to the sequential nature of their search. While practical engineering solutions can partly overcome this, they often result in a negative impact on performance. In this paper, we introduce SPO: Sequential Monte Carlo Policy Optimisation, a model-based reinforcement learning algorithm grounded within the Expectation Maximisation (EM) framework. We show that SPO provides robust policy improvement and efficient scaling properties. The sample-based search makes it directly applicable to both discrete and continuous action spaces without modifications. We demonstrate statistically significant improvements in performance relative to model-free and model-based baselines across both continuous and discrete environments. Furthermore, the parallel nature of SPO’s search enables effective utilisation of hardware accelerators, yielding favourable scaling laws.

SPO: Sequential Monte Carlo Policy Optimisation
Matthew V Macfarlane
Edan Toledo
University of Amsterdam m. ww.macaallaneeuvaa.
InstaDeep
Donal Byrne
Paul Duckworth InstaDeep
InstaDeep
Alexandre Laterre
InstaDeep
Abstract
Leveraging planning during learning and decision-making is central to the long- term development of intelligent agents. Recent works have successfully combined tree-based search methods and self-play learning mechanisms to this end. However, these methods typically face scaling challenges due to the sequential nature of their search. While practical engineering solutions can partly overcome this, they often result in negative impact on performance. In this paper, we introduce SPO: Sequential Monte Carlo Policy Optimisation, a model-based reinforcement learning algorithm grounded within the Expectation Maximisation (EM) framework. We show that SPO provides robust policy improvement and efficient scaling properties. The sample-based search makes it directly applicable to both discrete and continu- ous action spaces without modifications. We demonstrate statistically significant improvements in performance relative to model-free and model-basee baselines across both continuous and discrete environments. Furthermore, the parallel nature SPO's search enables effective utilisation of hardware accelerators, yielding favourable scaling laws.
Introduction
The integration of reinforcement learning (RI.) and neural-guided planning methods has recently achieved considerable success. Such methods effectively leverage planning during training via iterative imitation learning [7, 68, 18, 8). Applying additional computation via planning generates improved policies which is then amortised [4] into neural network policy. Using this policy as part of planning itself creates a powerful self-improvement cycle. They have demonstrated state of the art performance in applications ranging from chess [75] and matrix multiplication [24], to language modelling [61]. However, one of the most commonly-used rearch-based policy improvement operators, MCTS [16]: i) performs poorly low budgets [33], ii) is inherently sequential limiting its scalability [51, 73], and iii) requires modifications to adapt to large or continuous action spaces [57, 41]. These limitations underscore the need for more scalable, efficient and generally applicable algorithms.
In this work, we introduce SPO: Sequential Monte Carlo Policy Optimisation, model-based RL algorithm that utilises scalable sampled-based Sequential Monte Carlo planning for policy improvement. We formalise SPO as approximate policy iteration algorithm, and show that it is formally grounded within the Expectation Maximisation (EM) framework.
Work done during internship at InstaDeep
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SPO is unique by leveraging both breadth and depth of search in order to provide better estimates of the target distribution, derived via EM optimisation, compared to previous works. When viewed relative to popular algorithms it combines the breadth used in MPO [1] and the depth used in V-MPO [76], while enforeing KI. constraints on updates to ensure stable policy improvement. SPO achieves strong performance across range of benchmarks, outperforming AlphaZero, leading model-based expert iteration algorithm. We also benchmark against a leading Sequential Monte Carlo (SMC) algorithm from Piché et al. [65] and show that leveraging posterior estimates from SMC for policy improvement is central to strong performance.
We demonstrate that i) SPO outperforms baselines across both high- -ige-dimensional continuous control and challenging discrete planning tasks without algorithmic modifications, and ii) the sampling- based approach is inherently parallelisable, enabling the use of hardware accelerators to improve training speed. This provides significant advantages over MCTS-based methods [16], whose sequential process can be particularly inefficient during training. 3) The explicit use of KI targeting leads to strong and stable policy improvement, reducing the need for grid search over exploration hyperparameters in search. We find that SPO, empirically has strong scaling behaviours, with improved performance given additional search budget both during training and inference.
Related Work
There is extensive history of prior works that frame control and RL as inference problem [82, 45]. Expectation Maximization [19, 28, 60] is popular approach to directly optimising the evidence lower bound (ELBO) of the probability of optimality of a given policy. A number of model free methods implement EM, including RWR [63]. REPS [64], MPO [1], V-MPO [76). and AWAC [59] each with different approaches to the E-step and M-step (summarised in Furuta al. [29]). Model based algorithms such as MD-GPS [58] and AlphaZero [74] can also be viewed under the EM framework
using LQR [5] and MCTS [9, 16] during the E-step respectively. Grill et al. [33] show MCTS is a form of regularised policy optimisation equivalent to optimising the ELBO. Such model-based algorithms, that iterate between policy improvement using search, and projecting this improvement to the space of parameteriseable policies, are also often referred to as Expert iteration (ExIt) [74, 7] or Dual Policy Iteration [77]. Such approaches have also been applied active inference [27] where MCTS has been used to estimate expected free energy, with this computation then amortised into a neural policy [25]. Our work can also be framed as Exit, but differs by using Neural SMC for planning with an explicit KL regularisation on the improvement.
Sequential Monte Carlo (SMC) [31, 43, 52], also referred to as particle filters [66, 6), is inference method often employed to sample from intractable distributions. Gu et al. [34] explore the parameter- isation of the proposal in SMC using LSTMs [40], showing that posterior estimates from SMC can be used to train a parameterised proposal in non-control based tasks. Li et al. [49] demonstrate the same concept but using an MCMC sampler [30]. SMC can be applied to sample from distributions over trajectories in RL. For example, SMC-Learning [44] updates SMC weights using a Boltzmann exploration strategy, but unlike SPO it does not control policy improvement with KL regularisation or leverage neural approximators. Piché et al. [65] derive an SMC weight update to directly estimate the posterior over optimal trajectories. However this update requires the optimal value function, which they approximate using SAC [37] value function. This approach requires high sample counts and does not leverage SMC posterior estimates for policy improvement, instead utilising a Gaussian distribution or SAC to train the proposal. CriticSMC [50] estimates the same posterior but utilises a soft action-value function to score particles [36], reducing the number of steps performed in the environment, enabling more efficient exploration and improved performance compared with Piché et al. [65]. However their method is slower in wall clock time and uses a static proposal, therefore not performing iterative policy improvement. A useful properly of SMC search methods is that it is inherently parallelisable. Parallelising MCTS with a virtual loss has been explored, however this often leads to performance degradation [51], increasing exploration and leading to
lower SMC budgets
states that are difficult to evaluate Dalal et al. [17].
2

---

3 Background
Sequential decision-making can be formalised under the Markov Decision Process (MDP) frame- work [67]. An MDP is a tuple (S, Tr 7. 7. where, S is the set of possible states, A is the set of actions, T : S x A P(S) is the state transition probability function, r 8 x A IR is the reward function, 7 E [o, 1] is the discount factor, and jı is the initial state distribution. An agent interacts with the MDP using policy S + P(A). that associates every state a distribution over actions. We quantify the quality of a policy by the expected discounted return, that the agent seeks to maximise πt' E arg ma Ex, iEi= yree]. where r1 = r(s,, at) is the reward received at time I and IT is the set of all realisable policies. The value function V (s:) E., [Ei= 7'rx 8s]] maps a state 8, to the expected discounted sum of future rewards when acting according to w. Similarly, the state-action value function Q 0e) E -> 2Tr, | st, 0r] maps a state st to the expected discounted return, when taking the initial action de and following и thereafter.
Control as inference formulates the RL objective an inference problem within a proba- bilistic graphical model [82, 42, 45]. For a horizon T, the distribution over trajectories r =
-T (so, 40, 81, a1, .... ST, QT) is given by p(r) = su(s0) II ((ar)T((st (tc at), which is a func- tion of the initial state distribution, the transition dynamics, and an action prior. Note that this distribution is insufficient for solving control problems, because it has no notion of rewards. We therefore have to introduce an additional optimality variable into this model, which we will denote Or that is defined such that p(O, = 1[T) x exp(rı). Therefore, a high reward at time means a high probability of having taken the optimal action that point. We are then concerned with which is the distribution of trajectories given optimality at every step. We denote Ou. as O going forward. The RL objective is then formulated as finding a policy to maximise log maximising the distribution of optimality all timesteps, given actions sampled according to æ. To optimise this challenging objective, we can derive the evidence lower bound (ELBO) using an auxiliary distribution g [60]:
the target distribution
1) =log [T))p((
1/T ldr, which intuitively can be thought of as
1) log f x x(()p((-= 1/7)dr = log / (((T)
((())(((( (T))p(O 1/7) dr
log Pr
((T)
2 f ((5) [oogp(0= 117) + log
[()]
g(T)) dr E.g
Σ T'
-L(((T))l=(T))-
a
(1)
Since p(O = 1/7) x explE, re) and a is a normalising constant, ensuring a valid probability distribution.
Expectation Maximisation (EM) [20] has been widely applied to solve the optimisation problem max, p = 1) [19, 63, 64, 1, 76). After deriving the lower bound J (g. π) on the objective in eq. (1) using an auxiliary non-parametric distribution g. EM then performs coordinate ascent, iterating between optimizing the bound with respect to g (E-step) and with respect to the parametric policy x (M-step). This generates sequence of policy pairs {(TT, qo), (TI, 91 gi)..... (Tn, 9r)) such that in each step in the EM sequence, I (4++1, T++1) 2 J [[g, T.). Viewed through a traditional policy improvement lens, the E-step corresponds to policy evaluation phase where we perform rollouts, generating states with their associated estimates for q and value estimates. The M-step corresponds to a policy improvement phase where т and V are updated. This step can also be thought of as amortising the probabilistic inference computation performed in the E-step, into neural network forward pass operation. 2
Sequential Monte Carlo (SMC) methods [31] are designed to sample from an intractable distri- bution p referred to as the target distribution by using tractable proposal distribution 3 (we use 3 to prevent overloading of q). Importance Sampling does this by sampling from 3 and weighting samples by p(x)/B(x). The estimate is performed using a set of N particles (x: ) Inmis where (n) represents a sample from the support that the target and proposal distributions are defined over, along with the associated importance weights (w(")) In=1. Each particle uses sample from the proposal distribution to improve the estimation of the target, increasing N naturally improves the estimation of p [10]. Once importance weights are calculated, the target is estimated as Σm w(()) 5_(0s (z),
Many related RL algorithms can be understood by the different approaches to either the E-step or M- M-step.see appendix F.1 for summary of EM approaches.
3

---

where w is the normalised importance sample weight and 6_(=) is the dirac measure located at x(1). Sequential Importance Sampling generalises this for sequential problems, where x (2)...., TT). In this case importance weights can be calculated iteratively according to:
p(ra((1a-1)
we (a1::)) w B( (x11))
Particle filtering methods can be affected by weight degeneracy [56]. This occurs when a few particles dominate the normalised importance weights, rendering the remaining particles negligible. As the variance of particle weights is guaranteed to increase over sequential updates [21], this phenomenon is unavoidable. When the majority of particles contribute little to the estimation of the target distribution, computational resources are wasted. Sequential Importance Resampling (SIR) [53] mitigates this problem by periodically resampling particles according to their current weights, subsequently resetting these weights.
4 SPO Method
In this section, we present a novel method that combines Sequential Monte Carlo (SMC) sampling with the Expectation Maximisation (EM) framework for policy iteration. We begin by formulating the objective function that we aim to optimise. We then outline our iterative approach to maximising this function, alternating between an expectation step (E-step) and a maximisation step (M-step). Within the E-step, we derive the analytical solution for optimising the objective with respect to the auxiliary distribution q. We then demonstrate how SMC can be employed to effectively estimate this target distribution. The M-step can be viewed as projection of the non-parametric policy obtained the E-step back onto the space of feasible policies. A comprehensive algorithmic outline of our proposed approach is provided in appendix D.2.
4.1 Objective
We add an additional assumption to the lower bound objective defined in eq. (1) and assume that the auxiliary distribution g, defined over trajectories т. can be decomposed into individual state dependent distributions, i.c q(T) = p(so) II ( which decomposes in the same way. This enables the objective to be written with respect to individual states instead of full trajectories. Multiplying by a, the objective can then be written as follows:
We parameterise using
T(g = E.g
Σ 1=0
x lr. (KL(g(a)s)) ll x (a|s,, 0)}
+ (().
Previous works have explored this formulation where p(0) is prior over the parameters of [38, 71, l].
4.2 E-step
Within the expectation step of EM, we maximise eq. (3) with respect to q. As in previous work, instead of optimising for the rewards, we consider an objective written according to Q-values [1. 59, 54).
max Bm(a) [Ba(als) (s, a)] - aKL(g(-)s)) (--1s,0)))
q
This aims to maximise the expected Q-value of policy g. with the constraint that it doesn't 't move too far away from म. Framing optimisation with respect to Q-values is useful as it enables the use of function approximators to estimate Q.
Maximising this objective is difficult due to the dependence of both the expectation terms and Q- values on g. Following previous works, we fix the Q-values with respect to fixed policy म. resulting partial E-step optimisation [74, 76). We use the most recent estimate q as the fixed policy we perform maximisation with respect to, similar to AlphaZero's 's approach of acting according to the expert policy. The initial state distribution p(s) is likewise fixed to be distributed according to us. In practice, we use FIFO replay buffer and sample from it to determine ftg.

---

Equation (4) consists of balancing two objectives. To optimize this, we frame this as constrained maximization problem. to avoid scaling problems, see appendix G.3 for further information. This objective limits the KI. between q and ı from exceeding certain threshold:
max E pa (s) [B((as))
[ [s.a
(5)
s.t. p (s) KL (q(als) [(a] 8.)) <.
The following analytic solution to the constrained optimisation problem can then be derived using the Lagrangian multipliers method outlined in appendix G.1. Likewise 17" obtained by minimising the convex dual function eq. (7), and intuitively can be thought of as enforcing the KI. constraint on preventing di from moving to far from n;
Q (s((s))) - (V*
gi s) ((als, exp
exp
(6)
11"
where (s) is an action independent baseline. Q(s, a) V(s) is referred as the advantage, where optimising for Q- Q-values vs advantages is equivalent [59]. Optimising with advantages however, has demonstrated practically outperform optimising Q-values directly [62, 76, 59]. This also natural update to perform as the policy improvement theorem [78] outlines that update by policy iteration can performed if at least one state-action pair has positive advantage and a non-zero probability of reaching such state. Although we have closed form solution for q. we do not have either the value function or action-value function in practice. the next section we outline our approach to estimating this distribution.
Regularised Policy Optimisation: our closed form solution the optimisation eq. (6), we are required to solve for 17' which ensures that the KL is constrained. This can be calculated by minimising the following objective [84], see appendix G.2 for derivation:
(gx(als exp
exp (
AA AA a)
g(n) = nE + / ((s)) log
da ds.
Practically, we estimate eq. (7) using a sample- based estimator according to the distribution of states from the replay buffer and stored values for (als, 0;) and A (s,
4.2.1 Estimating Target Distribution
Building upon previous work using Sequential Monte Carlo for trajectory distribution estimation [44, 65], we propose novel approach that integrates SMC within the Expectation Maximisation
M-step
(7(s, a)
#.(a),,0.)
gi(ass) x (a)s,, exp
'7
a'mu(ttaa
L
S
of m 1i ~
E-step
Figure 1: SPO search: n rollouts, represented by particles ,',..., ,,", each of which represents an SMC trajectory sample, performed in parallel according to (left to right). At each environment step, the weights of the particles are adjusted (indicated in the diagram by circle size). We show two resampling regions where particles are resampled, favouring those with higher weights, and their weights are reset. The target distribution is estimated from the initial actions of the surviving particles (rightmost particles). This target estimate, qi, then used update π in the M-step.
5

---

framework to estimate the target distribution defined in Equation 6. Our SMC-based method enables sampling from qi over trajectories of length h, incorporating multiple predicted future states and rewards for a more accurate distribution estimation. A key property of estimating the target using SMC is that it uses both breadth (we initialise multiple particles to calculate importance weights) but also depth (leveraging future states and rewards) to estimate the distribution. In contrast, MPO evaluates several actions per state, focusing on breadth without depth, while V-MPO calculates advantages using n-Step returns from trajectory sequences, leveraging depth but only for a single action sample. Our method combines both aspects, enhancing the accuracy of target distribution estimation, crucial for both action selection and effective policy improvement updates in the M-step. An outline of the SMC algorithm is provided in algorithm 1, with a visual representation in fig. 1.
SMC estimates the target by maintaining a of particles (a'**) each of which maintains an importance weight for a particular sample. Given calculated importance weights SMC, estimates the target distribution according to (T) Σn= (i()) degn) (T) where (((n) is the normalised importance sample weight for particular sample. We next outline the sequential importance sampling weight update needed to estimate eq. (6). Since we can sample from T(o,, 8t, 0z), we leverage this as our
proposal distribution 3. Given the target distribution q can be decomposed into individual state dependent distributions, we can define (I(TT/Taa-)) and ((TI(Tia-1) as:
Pi(Te(TLE-1) & T(5+41 at) (:) (5,, 0,) exp (i(oe)))) St) Vi
(8)
B(Te(Tia-1) x (5 |8t, ar )T) (acl0r; 0:)
(9)
leading to the following convenient SMC weight update according to eq. (2):
T T(selse-2, (T ( 01-))
exp (A* (A"(ae, (ac. st)/n") T (aeess,, 0s) x(a (r, 0,)
Tmoddl (st w(T1a) oc w(T (T
(10)
where (a(a,,))) - (x())) is simplified to A ((c, 8t), Tie is sequence of state, action pairs from timestep to L, and Tmooee is the environment transition function of the planning model. Note that work assumes the availability of model that accurately represents the transition dynamics of the environment T, and therefore simplify the update to w w(T.:)) x w (TT.a-1) exp (A* (at. st)/"")).
Algorithm I outlines the method for estimating eq. (6) over h planning steps for N particles (line 3) using the advantage based weight update (line 7) eq. (10). Once h steps in the environment model have been performed in parallel, we marginalise all but the first actions as a sample based estimate of % (line 14).
The advantage function A is typically un- known, so we compute 1-step estimate at each iteration using the value function and ob- served environment rewards A(s,,a) = r, + V (s+++) V (st). Practically, we parame- terise the value function V using a neural net- work and train it using GAE [70] on true envi- ronment rollouts collected. After la steps, the im- portance weights leverage li-stess of observed states and rewards during planning and corre-
Algorithm 1 SMC q target estimation (timestep t)
1: Initialize {x] [n)
2: Set (w!") = 1
3: for E [t I,..., 11) do
4: {a!") T-|ss(") (1) 0) 0)
5:
(s(1) Tmade
6: [r(()) ~ Tmodel (5)
(w)))) w expp A(. r a )/1 W
7:
8:
if
1) mod p= 0 then
sponding advantage estimates. Compared to tree-based methods such as MCTS, SMC does
9:
{ (m) NN
Jum1 ~ Mult(n; w/!" w (N)
10:
(w (n) = 1
not require maintaining the full tree in memory.
11: end if
Instead, after each planning step, it only needs to retain the initial action, current state, and current particle weight. It also does not require spend- ing computation sending backward messages to update statistics of previous nodes everytime new node is added to the tree.
12: end for
13: {a. (n) } as the set of first actions of (a((n) a
14: ((als.) = En= m (n) (a)
Resampling Adaptive Search: To mitigate the issue of weight degeneracy [56). SPO conducts periodic resampling (lines 8- 11). This involves generating new set of particles from the existing set

---

by duplicating some trajectories and removing others. based on the current particle weights. This process enhances computational efficiency in estimating the target distribution. By resampling, we avoid updating weights for trajectories with low likelihood under the target, thereby reallocating computational resources to particles with high importance weights [22].
4.3
M-step
After completing the E-step (which generates qi), we proceed with the M-step, which optimises eq. (3) with respect to т, parametrised by 8. By eliminating terms that are independent of π, optimise the following objective, corresponding to a maximum a posteriori estimation with respect to the distribution q::
max J (qs, TT) = max peg (4) [Baa((ls) [log (a)s, 0)]] + log p(0).
(11)
#
This optimisation can be viewed as projecting the non-parametric policy back to the space of parametrisable policies 1lg, as performed in expert iteration style methods such as AlphaZero. p(0) represents a prior over the parameter 0. Previous works find that utilising a prior for to be close to the estimate from the previous iteration 6, leads stable training [1, 76]. Therefore we assume gaussian prior over the current policy parameters, see appendix G.5 where we show utilising such a prior leads to the following constrained objective:
max Eu,, (s) [Eataass) [log (a|s, 0)]]
0
(12)
s.t. ps) (s) [KL (T(aa 0,), (als, 0))] < 6m
4.4 Policy Improvement
Previous approaches to using SMC for RL either do not perform iterative policy improvement using SMC [65], or lack policy improvement constraints needed for iterative improvement [44]. Assuming that SMC provides perfect estimates of the analytic target distribution at each step i, Expectation Maximisation algorithm will guarantee that successive iterations will result in monotonic improvements in our lower bound objective, with details outlined in appendix G.4 [54].
Proposition 1. Given a non-parametric variational distribution qi and parametric policy ne. Given 4.+1, the analytical solution to E-step optimisation eq. (3), and Te the solution to maximisation problem in the M-step eq. (12) then the ELBO J guaranteed to be monotonically increasing: I (4+41, To..1) (o.41 ) 2 I J[s, )
In practice we are unlikely to generate perfect estimates the target through sample based inference and leave derivations regarding the impact of this estimation on overall convergence for future work. We also draw the connection between our EM optimisation method and Mirror Descent Guided Policy Search (MD-GPS) [58]. Our objective can be viewed as a specific instance of MD-GPS (see appendix G.6). Depending on whether dynamics are linear or not, optimising the EM objective can be viewed either as exact or approximate mirror descent [11]. Monotonic improvement guarantees in MD-GPS follow from those of mirror descent.
5 Experiments
In this section, we focus on three main areas of analysis. First, we demonstrate the improved performance of SPO in terms of episode returns, relative both model-free and model-based algorithms. We conduct evaluations across a suite of common environments for both continuous control and discrete action spaces. Secondly, training, showing that asymptotic performance scales with particle count and depth. Finally, we explore the performance-to-spee trade-off at test time by comparing SPO directly to AlphaZero as the search budget increases.
examine the scaling behaviour of SPO during
5.1 Experimental Setup
In order to ensure the robustness of our conclusions, we follow the evaluation methodology proposed by Agarwal al. [3], see appendix H for further details. This evaluation methodology groups perfor- mance across tasks within environment suite, enabling clearer conclusions over the significance of
7

---

10.
MPD
SP0
9.6
125
PPO
5MC
20
0.0
Timeslggs
3el
(a) Discrete: Rubik's Cube 7 and Boxoban Hard. (b) Continuous: Ant, HalfCheetah, Humanoid.
Figure 2: Learning curves for discrete and continuous environments. The Y-axis represents the interquartile mean of min-max normalised scores, with shaded regions indicating 95% confidence intervals, across 5 random seeds.
learning algorithms as a whole. We include individual results in appendix C, along with additional analysis measuring the statistical significance of our results.
Environments: For continuous control we evaluate the Brax [26] benchmark environments of: Ant, HalfCheetah, and Humanoid. For discrete environments, we evaluate on Boxoban [35] (a specific instance of Sokoban), commonly used assess planning methods, and Rubik's Cube, sparse reward environment with a large combinatorial state-action space. See appendix A for further details regarding environments.
Our model-free baselines include PPO [72], MPO [1] and V-MPO [76] for both continuous and discrete environments. For model bbaeed algorithms, we compare performance to AlphaZero (including search improvements from MuZero [68]) and an SMC method introduced by Piché et al. [65] 3 (which refer to SMC-ENT due to its use of maximum entropy RL to train the proposal and value function used within SMC). Our AlphaZero benchmark follows the official open-source implementation' For continuous environments modern extension to MuZero for large and/or continuous action spaces. utilise a true environment model, aligning implementation of Sampled MuZero more closely with AlphaZero. Our core experiments configure SPO with 16 particles and horizon of 4, and AlphaZero with 64 simulations to equalise search budgets. For remaining parameters, see appendix E. and appendix D.3. Each environment and algorithm were evaluated with five random seeds. 5
Baselines:
baseline results to Sampled MuZero [41],
5.2 Results
Policy Improvement: In fig. 2 we show empirical evidence that SPO outperforms all baseline methods across both discrete (left) and continuous (right) benchmarks for policy improvement. This conclusion also holds for the per-environment results reported appendix C. SMC-ENT [65] is strong SMC based algorithm, however, SPO outperforms it for both discrete and continuous environments providing strong evidence of the direct benefit using SMC posterior estimates for policy improvement. We also highlight the importance regularising policy optimisation, which is practically achieved by solving for 17" eq. (7), and re-estimated at each iteration. In appendix B we ablate the impact of this adaptive method by comparing varying fixed temperature schedules. While a good temperature can be found through expensive grid search for each new environment we find the adaptive temperature ensures stable updates leading to strong learning performance across all environments, with minimal overhead to compute 17"
We find that SPO performance is statistically significant compared to AlphaZero across the eval- uated environments, see appendix for detailed results. The substantial variation in AlphaZero performance across these environments underscores the difficulty in tuning it for diverse settings. For instance, while AlphaZero performs well on Boxoban and HalfCheetah, its performance drops significantly other discrete and continuous problems. This inconsistency poses major challenge
'e use MDQN [83], and SAC [37] for discrete and continuous environments. respectively. Note that in the discrete case, SMC-ENT using SAC was very unstable, chose MDQN as better alternative.
'Open-source implementation available at metx.
Inference code and checkpoints available https: wwwp:/s..ocooom/mteeteeeeee hotp:///c...//ttttppp///////

---

Number Partclis
SPO
0.8
80%
SP0 Depth 4
Depth
Normaladd Sloak
0.6
8ilee Rale
70%
3.4
0.2
50%
0.001s
0,0105
0.100s
Stop (4)
Figure (left) Scaling: Mean normalised performance across all continuous environments on 10t environment steps, varying particle numbers N and horizon h for SPO during training. (right) Wall Clock Time Comparison: Performance Rubik's cube plotted against wall-clock time for AlphaZero and 3 versions of SPO (varying by SMC search depth), with total search budget labeled at each point.
for its applicability to real-world problems. In contrast, SPO performs consistently well across all environments (both discrete and continuous), highlighting its general applicability and robustness to various problem settings.
5.3 Scaling SPO during training
In fig. 3 (left) we investigate how SPO performance is impacted by scaling particle counts N and horizon length during training, both of which we find improve the estimation of the target distribu- tion. Our results show that scaling both the particle count and the horizon leads to improvements in the asymptotic performance of SPO. also suggests that both variables should scaled together for maximum benefit having a long horizon with low particle counts actually negatively impact performance. Secondly, we highlight that while for our main results we use a total search budget (particles x depth) of 64 for policy improvement, be achieved by reducing this budget by a factor of four, which demonstrates competitive performance low compute budgets.
scaling results show that competitive results can
5.4 Scaling SPO at test time
We can scale particle counts and horizon during training improve target distribution estimates for the M-step. However, this scaling can also be performed after training when тe, is fixed, enhancing performance, since actions are sampled in the environment according to the improved policy This equates to additional E-step optimization, which can enhance performance due to the representational constraints of the space of parameterised policies. In fig. 3 (right) we demonstrate how the performance of both SPO and AlphaZero search scales with additional search budget at test time using the same high performing checkpoint. We plot the time taken for single actor step (measured on a TPUv3-8) against solve rate for the Rubik's Cube problem with time logarithmic axis. Specifically, we evaluate on 1280 episodes for cubes ten scrambles away from solved. While we recognise that such analysis can be difficult to perform due to implementation discrepancies, we used the official JAX implementation of AlphaZero (MCTX) within our codebase and compare this to SPO. Additionally, exclusively measure inference thus training implementation details affect our measurements.
This provides evidence that AlphaZero has worse scaling when compared to SPO on horizons four and eight, noting that for horizon of two, SPO performance converges early, as low depths can act as bottleneck for further performance improvements. This highlights the benefits of the SPO parallelism. This chart also shows how different compute preferences at test time, a different horizon length preferable. For limited compute, low horizon lengths and relatively higher particle counts provide the best performance, but as compute availability increases, the gains from increasing particle counts decrease and instead horizon length / should be increased.

---

Lastly, the plot illustrates the increase in available search budget. by using SPO, given time restriction rather than a compute restriction. For example. SPO can use times more search budget, when compared to AlphaZero, given a requirement of 0.1 second per step.
6 Conclusion
Wlanning-based policy improvement operators have proven to be powerful methods to enhance the learning of policies for complex environments when compared to model-free methods. Despite the success of these methods, they are still limited in their usefulness, due to the requirement of high planning budgets and the need for algorithmic modifications to adapt to large or continuous action spaces. In this paper, we introduce modern implementation of Sequential Monte Carlo planning as a policy improvement operator, within the Expectation Maximisation framework. This results in a general training methodology with the ability to scale in both discrete and continuous environments.
Our work provides several key contributions. First, we show that SPO is a powerful policy improve- ment operator, outperforming our model- and model-free baselines. Additionally, SPO is shown to be competitive across both continuous and discrete domains, without additional environment- specific alterations. This illustrates the effectiveness of SPO as generic and robust approach, in contrast to prior methods that require domaii--peccfic enhancements. Furthermore, the parallelisable nature of SPO results in efficient scaling behaviour of search. This allows SPO to achieve significant boost in performance at inference time by scaling the search budget. Finally, we demonstrate that scaling SPO results in faster wall-clock inference time compared to previous work utilising tree- tree-based search methods. The presented work culminates in a versatile, neural- guide., sample-baser based planning method that demonstrates superior performance and scalability over baselines.
Limitations & Future Work: This work demonstrates the efficacy of combining probabilistic inference with reinforcement learning and amortisation. While our work considers relatively standard form of Sequential Monte Carlo, future work could investigate making improvements to this inference method, in order to further improve the estimate of the target distribution, while maintaining the scalability benefits demonstrated. We also draw the connection to Active Inference [27] which has been explored in the context of agent behaviour in complex environments, where Monte Carlo Tree Search has been used to scale previous methods along with deep learning [25]. Leveraging Sequential Monte Carlo in such methods along with amortisation could be promising direction of research to further scale such methods. Our work exclusively uses exact world models of the environment in order to isolate the effects of various planning methods performance. Extending our research to include a learned world model would broaden the applicability of SPO to more complex problems that may not have a perfect simulator and are therefore unsuitable for direct planning. Additionally, we apply Sequential Monte Carlo (SMC) only in deterministic settings. Adapting our approach for stochastic environments presents challenges. Specifically, the importance weight update in SMC can lead to the selection of dynamics most advantageous to the agent, potentially fostering risk-seeking behaviour in stochastic settings. However, mitigation strategies exist, discussed in Levine [45].
Acknowledgments and Disclosure of Funding
The authors would like to thank Teodora Pandeva for detailed discussions on SPO theory and for detailed revisions of the manuscript. Clément Bonnet for brainstorming discussions and feedback and David Kuric for revisions of the final manuscript. We also thank members of the InstaDeep team for their support in the preparation of the final manuscript. Lastly we thank the anonymous reviewers for comments and helpful discussions that helped improve the final version of the paper.
Research was supported with Cloud TPUs from Google's 's TPU Research Cloud (TRC). Matthew Macfarlane supported by the LIFT-project 019.011, which is partly financed by the Dutch Research Council (NWO).
References
[1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL https: httpt::::///eeeeeiee...... Set.
10

---

[2] Jacob Adkins, Michael Bowling, and Adam White. A method for evaluating hyperparameter sensitivity reinforcement learning. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks.
[3] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 11-93344-29322 2021.
[4] Brandon Amos al. Tutorial on amortized optimization. Foundations and Trends in Machine Learning, 16($):592-732, 2023.
[5] Brian DO Anderson and John B Moore. Optimal control: linear quadratic methods. Courier Corporation, 2007.
[6] Christophe Andrieu, Arnaud Doucet, Sumeetpal S Singh, and Vladislav B Tadic. Particle methods for change detection, system identification, and control. Proceedings of the IEEE, 92 (3):423-438, 2004.
[7] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[8] Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Planning in stochastic environments with a learned model. In International Conference on Learning Representations, 2021.
[9] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256 2002.
[10] Alan Bain and Dan Crisan. Fundamentals of stochastic filtering, volume 3. Springer, 2009.
[11] Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
[12] Clément Bonnet, Daniel Luo, Donal John Bymee Shikha Surana, Paul Duckworth, Vincent Coyette, Laurence Illing Midgley, Sasha Abramowitz, Elshadai Tegegn, Tristan Kalloniatis, et al. Jumanji: a diverse suite of scalable reinforcement learning enviromments in jax. In The Twelfth International Conference on Learning Representations, 2023.
[13] Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747-769, 2021.
[14] James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miin and Qiao Zhang. JAX: composable transformations of Python+NumPPy programs, 2018. URL http: nttp://github com/ google/jax.
[15] Cédric Colas, Olivier Sigaud, and Pierre Yves Oudeyer. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. International conference on machine learning, pages 1039-1048. PMLR, 2018.
[16] Rémi Coulom. Efficient selectivity and backup operators in Monte-Carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
[17] Gal Dalal, Assaf Hallak, Steven Dalton, Shie Mannor, Gal Chechik, et al. Improve agents with- out retraining: Parallel tree search with off off- -oficy correction. Advances in Neural Information Processing Systems, $4:5518-5555 2021.
[18] Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2021.
[19] Peter Dayan and Geoffrey Hinton. Using expectation-maxiiiiaan for reinforcement learning. Neural Computation, 9(2):271-27) 1997.

---

[20] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39 (1):1-22, 1977.
[21] Arnaud Doucet, Simon Godsill, and Christophe Andrieu. On sequential monte carlo sampling methods for bayesian filtering. Statistics and computing, 10:197-208, 2000.
[22] Arnaud Doucet, Nando De Freitas, Neil James Gordon, et Sequential Monte Carlo methods in practice, volume 1. Springer, 2001.
[23] Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance-hor properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2775-2785, 2019.
[24] Alhussein Fawzi, Matej Balog. Aja Huang. Thomas Hubert, Bernardino Comera-Paredee Mohammadamin Barekatain, Alexander Novikov, Francisco R. Ruiz, Julian Schritwieser, Grzegorz Swirszez, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-5.. 2022. 150044411-8-02222722222
doi:
[25] Zafeirios Fountas, Noor Sajid, Pedro Mediano, and Karl Friston. Deep active inference agents using monte-carlo methods. Advances in neural information processing systems, 33:1 1662- 1 11675, 2020.
[26] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021. URL ttttp://githie com/ gowgle/oooeaee
[27] Karl Friston. The free-energy principle: a unified brain theory? Nature reviews neuroscience, 11(2):177-138, 2010.
[28] Thomas Furmston and David Barber. Variational methods for reinforcement learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 241-248. JMLR Workshop and Conference Proceedings, 2010.
[29] Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Co-adaptation of algorithmic and implementational innovations inference-basee deep reinforcement learning. Advances in neural information processing systems, 34:9828-9844 2021.
[30] Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. Markov chain Monte Carlo in practice. CRC press, 1995.
[31] Neil J Gordon, David Salmond, and Adrian FM Smith. Novel approach to nonlinear/non- gaussian bayesian state estimation. IEE proceedings (radar and signal processing), volume 1 140, pages (37-113. IET, 1993.
[32] Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards a standardised performance evaluation protocol for cooperative marl. Advances in Neurul Information Processing Systems, 35:5510-5521, 2022.
[33] Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-carlo tree search as regularized policy optimization. In International Conference on Machine Learning, pages 3769-3778. PMLR, 2020.
[34] Shixiang Shane Gu, Zoubin Ghahramani, and Richard E Turner. Neural adaptive sequential monte carlo. Advances in neural information processing systems, 28, 2015.
[35] Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, Timothy Lillicrap, and Victor Valdes. An investigation of model-free planning: boxoban levels. https:/github..com/// httss//hthbb.ommmmeemmmmmbbbbbee 2018.
12

---

[36] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
[37] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[38] Hirotaka Hachiya, Jan Peters, and Masashi Sugiyama. Efficient sample reuse in em-based policy search. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2009, Bled, Slovenia, September 7-11, 2009, Proceedings, Purt I 20, pages 669-484 Springer, 2009.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
[41] Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning complex action spaces, In International Conference on Machine Learning, pages 4476-4486. PMLR, 2021.
[42] Hilbert J Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as graphical model inference problem. Machine learning, 87:159-182, 2012.
[43] Genshiro Kitagawa. Monte carlo filter and smoother for non-gaussian nonlinear state space models. Journal of computational and gruphical statistics, 5(1):1-25, 1996.
[44] Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Reinforcement learning in contin- uous action spaces through sequential monte carlo methods. Advances in neural information processing systems, 20, 2007.
[45] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00999 2018.
[46] Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning, pages 1-9. PMLR, 2013.
[47] Haim Levy. Stochastic dominance and expected utility: Survey and analysis. Management Science, pages 555-593, 1992.
[48] Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In First International Conference on Informatics in Control, Automation and Robotics, volume 2, pages 222-229. SciTePress, 2004.
[49] Yingzhen Li, Richard E Turner, and Qiang Liu. Approximate inference with amortised meme. arXiv preprint arXiv: met.:00222..333 2017.
[50] Vasileios Lioutas, Jonathan Wilder Lavington, Justice Sefas, Matthew Niedoba, Yunpeng Liu, Berend Zwartsenberg, Setarch Dabiri, Frank Wood, and Adam Scibior. Critic sequential monte carlo. In The Eleventh International Conference on Learning Representations, 2022.
[51] Anji Liu, Jianshu Chen, Mingze Yu, Yu Zhai, Xuewen Zhou, and Ji Liu. Watch the unobserved: simple approach to parallelizing monte carlo tree search. In International Conference on Learning Representations, 2019.
[52] Jun S Liu and Rong Chen. Sequential monte carlo methods for dynamic systems. Journal of the American statistical association, 1334444 1998.
[53] Jun S Liu, Rong Chen, and Tanya Logvinenko. A theoretical framework for sequential im- portance sampling with resampling. In Sequential Monte Carlo methods in practice, pages 225-246. Springer, 2001.
13

---

[54] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference Machine Learning, pages 13644-13668. PMLR, 2022.
[55] Henry B Mann and Donald R Whitney. On test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50-60, 1947.
[56] Simon Maskell and Neil Gordon. A tutorial particle filters for on-line nonlinear/non gausssian bayesian tracking. IEE Target Tracking: Algorithms and Applications (Ref. No. 2001/174), pages 2-1, 2001.
[57] Thomas M Moerland, Joost Broekens, Aske Plaat, and Catholijn Jonker. A0e: Alpha zero in continuous action space. arXiv preprint arXivv1805.090 2018.
[58] William Montgomery and Sergey Levine. Guided policy search as approximate mirror descent. arXiv preprint arXiv: 2016.
[59] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint wVXi:22006.09359 2020.
[60] Gerhard Neumann et al. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pages 817-824, 2011.
[61] Jing-Cheng Pang, Pengyuan Wang. Kaiyuan Li, Xiong-Hul Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self- eeff-imprveeeet by reinforcement learning contempla- tion. arXiv preprint 23X.::2300.14483, 2023.
[62] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117 2019.
[63] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference Machine learning, pages 745-750, 2007.
[64] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607-1612, 2010.
[65] Alexandre Piché, Valentin Thomas, Cyril Ibrahim, Yoshua Bengio, and Chris Pal. Probabilistic planning with sequential monte carlo methods. In International Conference on Learning Representations, 2018
[66] Michael K Pitt and Neil Shephard. Auxiliary variable based particle filters. Sequential Monte Carlo methods in practice, pages 273-293, 2001.
[67] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[68] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
[69] John Schulman. Trust region policy optimization. arXiv preprint arviv arXiv: 150..05477.77 2015.
[70] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:/ //tp:////rvvv.e//e org. aa//ab///1000 02438.
[71] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft g-leamnin, arXiv preprint
2017.
14

---

[72] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:177..06377 2017.
[73] Richard Segal. On the scalability of parallel uct. In International Conference on Computers and Games, pages 36-47. Springer, 2010.
[74] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3354 359, 2017.
[75] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science, 110164419)1140--1444 144, 2018.
[76] H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2019.
[77] Wen Sun, Geoffrey Gordon, Byron Boots, and Bagnell. Dual policy iteration. Advances in Neural Information Processing Systems, 31, 2018.
[78] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[79] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 10.1109//ROS2011 2012.6386109,
[80] Edan Toledo. Stoix: Distributed single-agent reinforcement learning end-to-end in jax, April 2024. URL https: //github. com/ homm/EddnTooee//toiiiii
[81] Edan Toledo, Laurence Midgley, Donal Byme, Callum Rhys Tilbury, Matthew Macfarlane, Cy- prien Courtot, and Alexandre Laterre. Flashbax: Streamlining experience replay buffers for rein- forcement learning with jax, 2023. URL htthtt/::// h/tgh/tt/////tthuub.. httpg/gttubbccoooooo cot ctwm//nstaadeppi//laaaaaaaaa
[82] Mare Toussaint and Amos Storkey. Probabilistic inference for solving discrete and continuous state markov decision processes. In Proceedings of the 23rd international conference on Machine learning, pages 945-952, 2006.
[83] Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances Neural Information Processing Systems, 33:4235 4246, 2020.
[84] Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-free free preference based reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
15

---

Appendix
Table of Contents
A Environments
17
A.1 Sokoban
A.2 Rubik's Cube
A.3
Brax
B
Ablations
B.1
E-step constraint
B.2
E-step Q-value Optimisation
B.3
SMC Target Estimation Validation
20
C
Expanded Results
21
C.I Detailed Summary Results
................................
21
C.2 Hardware
22
C.3 Individual Environment Results
23
D
SPO
24
D.I Temperature Loss
24
D.2
Approximate Policy Iteration
Aee.............................
24
D.3
Hyperparameters
................................
25
D.4
Intuition for Temperature and Exploration:
26
E
Baselines
26
E.
Hyperparameters
26
F Expectation Maximisation
30
F.I
Overview of methods
30
G Proofs and Discussions
31
G.1 E-step Analytic solution
................................
31
G.2 E-step KL constraining temperature
31
G.3 E step: Constraint Optimisation
32
G.4 Proof of Proposition 1: Monotone Improvement Guarantee
32
G.5 M-step objective
32
G.6 Connection Mirror Descent Guided Policy Search
Statistical Precipice
H.l Overview
34
H.2 Hyperparameters
34
16

---

A Environments
A.1 Sokoban
A.1.1 Overview
Table I: Summary of Boxoban dataset levels
Diffieulty Level
Dataset Size
Unfiltered Train
900k
Unfiltered- Validation
100k
Unfiltered- Test
1k
Medium
450k
Hard
50k
We use the specific instance of Sokoban outlined in Guez et al. [35] illustrated in Figure 4 with the codebase The datasets employed in this study publicly
available https://github. com/instadee attaaaaaaeeaaajumaaaaaa accessible https: / /tt///gtttubbbbt com/ com/googl e wwwpeddnd/booobbaaaeeedoe -eveel.. These datascts are split into different levels of difficulty, which are categorised Table
this research, we always train on the Unfiltered-Theet Train dataset. Evaluations are performed on the Hard datasct, important for differentiating the strongest algorithms.
Figure 4: Example of Boxoban Problem
A.1.2 Network Architecture
Observations represented array of size (10,10) where each entry is integer representing the state of cell. We used deep ResNet [39] architecture for the network to embed the observation. We define block consisting of the following layers: [CNN, ResNet, ResNet]. Four such blocks are stacked, each characterized by specific parameters:
Output Channels: (256, 256, 512)
Kernel Sizes: (3, 3,
Strides: (1, 1,
Additionally, the architecture includes two output heads (policy and value), each comprising two layers of size 128 with ReLU activations. The output heads share the same torso network. We use the same architecture for all algorithms.
A.2
Rubik's Cube
A.2.1
Overview
For our Rubik's Cube experiments, we utilize the implementation in Bonnet el al. [12] available at https: //github. com/ scaling the number of random actions performed in solution state generate problem instances. We always perform training on a uniform distribution sampled from the range [3.7] scrambled states, followed by exclusively evaluating on 7 scrambles. The observation is represented using an array of size (6,3.3). The action space is represented using array of size (6,3) corresponding to each face and the depth at which it can be rotated.
Rubik's cube problems can be be made progressively difficult by
17

---

A.2.2 Network Architecture
We utilise an embedding layer to first embed the face representations to size (6,3.3,4). The embedding is flattened and we use a torso layer consisting of a two layer residual network with layer size 512 and ReI.U activations. Additionally, the architecture includes two output heads (policy and value), each comprising of single layer of size 512 with ReL.U activations. We use the same architecture for all algorithms.
A.3
Brax
A.3.1
Overview
For experimentation on continuous settings, we make use of Brax [26]. Brax is library for rigid body simulation much like the popular MuJoCo setting [79] simulator. However, Brax is written entirely in JAX [14] to fully exploit the parallel processing capabilities of accelerators like GPUs and TPUs.
is important note, that the time of writing, there are different physics back-ends that can used. These back-ends have differing levels computational complexity the results between them are not comparable. The results generated in this paper utilise the Spring backend.
Table 2: Observation and action space for Brax environments
Environment Observation Size
Action Size
Halfcheetah
18
Ant Humanoid 376
27
8
17
Table 2 contains the observation and action specifications of the scenarios used for our experiments. We specify the dimension size of the observation and action vectors.
A.3.2 Network Architecture
In practice, we found the smaller networks used in the original Brax publication limit overall performance. We instead use 4 layer feed forward network to represent both the value and policy network. Aside from the output layers, all layers are size 256 and use SiLUU non-linearities. use the same architecture for all algorithms. Unlike Rubik's s Cube and Sokoban, we do not use shared torso to learn embeddings.
'except for SMC-ENTR,fo which we to stabilise performance.
18

---

B
Ablations
B.1
E-step KL constraint
The KL constraint for the E-step within the Expectation Maximisation framework is an important component of ensuring policy improvement, due to the KL term in the ELBO. We investigate the impact of the adaptive temperature for both discrete and continuous environments. We compare SPO with an adaptive temperature updated every iteration to variety of fixed temperatures.
Tead Sepprttre Sumposttre 5.1 SPO
Shd Feed Nampenstone 11
Flod Namperstaee 20.0 Ineppmotern
Timastops
Figure 5: Brax
Figure 6: Sokoban and Rubik's Cube
This ablation shows that SPO with adaptive temperature is among the top performing hyperparameter settings across all environments. However we also that possible tune temperature that works well when considering wide range of temperatures. This consistent with previous results in Peng et al. [62] that also find practically for specific problems a fixed temperature can be used, Of course in practice having an algorithm that can learn this parameter itself is practically beneficial, removing the need for costly hyperparameter tuning, since the appropriate lemperature is likely problem dependant.
Subsequently, evaluated whether the partial optimisation of the temperature parameter 7) effectively main- tained the desired KI. divergence constraint and how different values of this constraint affected performance.
14000
62
330
Figure (a) The estimated KL. divergence between the prior policy x and the target policy q generated by SMC for different values of during training the Brax Ant task. (b) Evaluation performance during training for different values of
As shown in Figure 7(a), the KL divergence constraint is tightly maintained throughout training for different values of Figure 7(b) demonstrates that, for the Ant task, allowing larger KL divergence between the prior and target policies (g and n, respectively) leads to improved performance.
B.2 E-step Q-value Optimisation
For our analytic solution to the optimisation problem in the E-step we choose to add value baseline such that target distribution re-weights with respect Advantages instead of Q-values. practice there restrictions on utilising Q-values instead the importance weight update of SMC eq. (10). Below we show results on the continuous environments comparing SPO using advantages to SPO using Q-values.
We see in Figure that the use Q values produces a reduction in performance that is relatively constant throughout training. It is possible that this reduction is due to different hyperparameters being required to effec- tively utilise the Q value distribution but in order to accurately judge the effect, kept hyperparameters constant
19

---

Normal Fpante Natturn
0.8
0.4
0.6
Timestops
Figure Q-value ablation on Brax tasks
across the runs. Secondly we note this is consistent with existing results for EM algorithms, demonstrating that Advantages outperform Q-values Peng et al. [62], Song al. [76], Nair et al, [59].
B.3 SMC Target Estimation Validation
Estimating the target distribution accurately is important for policy improvement SPO. Therefore, we aim to validate the ability of SMC search to better approximate the true target distribution than simple 1-step function evaluation is done in algorithms such MPO [1]. To do this, while the true target distribution not directly accessible, approximate it using unbiased Monte Carlo oracle. We 1280 rollouts, to end the episode, for every state and every action. We evaluate the impact varying planning horizons (depth) and particle counts on the KL divergence between the SMC-estimated target policy and the Monte Carlo oracle in the Sokoban environment.
large computational budget of
3.75
123
238
1.2
1.00
0.25
0.00
Sond
Figure 9: Comparison of the KI. divergence to large Monte Carlo simulation of target policy for different planning horizons and particle counts for Sokoban
As shown in Figure 9, increasing the number of particles and the planning depth reduces the KI. divergence to the oracle, indicating improved estimation of the target distribution. This aligns with SMC theory [22]. which suggests more particles deeper planning lead better approximations. These results highlight importance of leveraging both breadth (more particles) and depth (longer planning horizons) SMC for accurate target estimation.
20

---

C Expanded Results
provide more detailed analysis the results presented the main body the paper, including final point aggregate performance of algorithms, the probability of improvement plots and the performance profiles. All plots are generated according to the methodology outlined by Agarwal et al. [3] and Gorsane ct al. [32] by performing the final evaluation using parameters that achieved the best performance throughout intermediate evaluation during training.
The point aggregation plots show the rankings of normalised episode returns of each algorithm when different aggregation metrics are used. The probability of improvement plots illustrate likelihood that algorithm X outperforms algorithm Y on a randomly selected task. It is important to note that a statistically significant result from this metric is probability of improvement greater than 0.5 where the confidence intervals (CIs) do not contain 0.5. The metric utilises the Mann- Whimeyy U-statistic [55]. See [3] for further details. The performance profiles illustrate the fraction of the rums over all training environments from each algorithm that achieved scores higher than specific value (given on the X-axis). Functionally, this serves the same purpose as comparing average episodic returns for each algorithm in tabular form but a simpler format. Additionally, if algorithm's 's curve is strictly greater than equal to another curve, this indicates "stochastic dominance" [47, 23].
C.1 Detailed Summary Results
Median
IQM
Mean
VMPO SPO SMC-ENT PPO MPO AZ-S
0.60
0.75
0.90
0.60
0.75
0.90
0.7
0.8
0.9
Normalised mean episode return
Figure 10: Aggregate point metrics for Brax suite. 95% confidence intervals generated from stratified bootstrapping across tasks and seeds reported.
Median
IQM
Mean
VMPO
SPO
SMC-ENT
PPO
MPO
AZ
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Normalised mean episode return
Figure 11:
Aggregate point metrics for Sokoban and Rubik's Cube. 95% confidence intervals generated from stratified bootstrapping across tasks and seeds are reported.
In Figures 10 and 11, we present the absolute metrics following the recommendations by Agarwal et al. [3], Colas et al. [15] and Gorsane al. [32]. These metrics evaluate the best set of network parameters identified during 20 evenly spaced training evaluation intervals. The absolute evaluations measure performance across 10 times the number of episodes periodically assessed during training, resulting in total of 1280 episodes evaluated. We report point estimates and their 95% confidence intervals, which were calculated using stratified bootstrapping. Agarwal et al. [3] advocate for the Interquartile Mean (IQM) as more robust and valid point estimate metric. Our results indicate that SPO surpasses all other algorithms in every point estimate. Notably, the IQM and mean point estimates demonstrate statistical significance.
21

---

1.00
1.00
Fraction of runs with score
0.75
0.75
0.50
Fraction runs
0.50
AZ-S
AZ
MPO
MPO
0.25
PPO
0.25
SMC-ENT
SPO
SPO
VMPO
VMPO
0.00
0.00
Normalised mean episode return (t)
Normalised mean episode return (1)
(a) Brax.
(b) Sokoban and Rubik's Cube.
Figure 12: Performance profiles. The Y-axis represents the fraction runs that achieved greater than specific normalised score represented the x-axis, with shaded regions indicating 95% confidence intervals generated from stratified bootstrapping across both tasks and random seeds.
In Figure 12, present the performance profiles which visually illustrates the full distribution of scores across all tasks and seeds for cach algorithm. We see that SPO has higher lower bound on performance, addition upper bound, indicating lower variance across tasks seeds. Additionally, we see that SPO is strictly above all other algorithms indicating that SPO is stochastically dominant" to all baselines.
Algorithm X
Algorithm
Algorithm X
Algorithm
SPO
VMPO
SPO
~MMP
SPO
SMC-ENT
SPO
PPO
SPO
AZ-S
SPO
MPO
SPO
PPO
SPO
SMC-ENT
SPO
MPO
SPO
AZ
0.00
0.25
0.50 0.75 1.00 P(X Y)
0.00
0.25
0.50 0.75
1.00
PIX > Y)
(a) Brax.
(b) Sokoban and Rubik's Cube.
Figure 13: Probability of Improvement.
Lastly, present the probability of improvement plots in Figure 13. We see that SPO has high probability of improvement compared all baselines. Additionally, probabilities above 0.5 thus indicating statistical significance as specified by Agarwal et al. [3]. Furthermore, we see all have upper bounds greater than 0.75, thereby indicating that these results are statistically meaningful as specified by Bouthillier et al. [13].
larger than 0.5 and have their CIs
C.2 Hardware
Training was performed using a mixture of Google v4-8 and v3-8 TPUs. Each experiment was run using a single TPU and only v3-8 TPUs were used to compare wall-clock time.
"A random variable X is termed stochastically dominant over another random variable Y if P(X > 2) P(Y > for all and for P(X r) P(Y > r).
22

---

C.3 Individual Environment Results
All individual task results presented in fig. 14. Specifically, present the IQM of returns achieved during the evaluation intervals throughout training.
AR
The kiheee
1200
030
Transags
(a) Ant.
(b) HalfChectah.
200
19989
6.03
400
280
Treater
Treesies
(c) Humanoid.
(d) Sokoban.
Rasko abe
(e) Rubik's Cube scrambles
Figure 14: Performance across different environments.
23

---

D SPO
Below outline the practical policy- based losses used within SPO,
(=( - Σ
(alls) log no, (als)
u(n) me + ((s)log (f T(ols, exp
1 (s) da ds.
(a) = (Ea Re-p(o) (gg (Tom [TT)ll))
(x) [ Esmp(s) (Te m²)]
D.1 Temperature Loss
The temperature loss is calculated using advantages collected during SPO search with rollouts performed according policy If resampling is utilised, after resample the distribution over trajectories shifts towards the target distribution away from æ. Therefore in order calculate the loss g(n), only utilise the advantages up to the resampling period with rollouts performed according to æ.
D.2 Approximate Policy Iteration Algorithm
Below outline the overall SPO algorithm with the main loop split into the E-step and M-step.
Algorithm 2 SPO Algorithm
1: B (
2: Initialize the policy, value function, temperature, and alpha:
3: Initialize 0o, do, 1/0, 00
4: for iteration k = 0, 1, 2,.,.. wwm.nummitte nuum_teraaiins do
5: Expectation Step (E-step):
6:
for agent m = 1.2..... M in parallel do
7:
Initialize state 8g
8:
for timestep i = 1,2, 1......, rollout Jength do
9:
a {a! L.V I==1
(sM Tkk OR, Φk)
B> sample-based estimate of qa b Sample an action to execute in the environment Execute action a, observe state 8
10:
a ~ [a!n) 1==
N
11:
8441 T(s;a))
12:
r(s;, a)
> Observe reward r:
Store (st, Ti, in
14:
end for end for
16:
Maximization Step (M-step):
for batch b 2. num batches do
18:
Sample batch from replay buffer:
19:
Sample batch size N' from replay buffer (s, r, ū) ~ B
20:
Update value function using GAE targets:
Φk+1 + arg min, Es-B (Vose(s, ø') - V(s, o))²
21:
Update parameterized policy using sample-based estimate g:
23:
0x+1, 11+11 < arg ming E.-5 [C. (0, a) + (K))
Update dual temperature:
25:
Mk+1 & arg min, Eg- [g(n)]
o' - poloyakk(o'... (x+1)+)
27:
end for
end for
24

---

D.3 Hyperparameters
Table SPO Hyperparameters for Continuous and Discrete Environments
Parameter
Continuous Environments
Discrete Environments
Actor & Critic Learning Rate
3e
4
Dual Learning Rate
le-3
3e
Discount Factor
0.99
0.99
GAE Lambda
0.95
0.95
Replay Buffer Size
6.5c*
6.5c*
Batch Size
32
64
Batch Sequence Length
32
17
Max Grad Norm
0.5
0.5
Number of Epochs
128
16
Number of Envs
1024
768
Rollout Length
32
21
T (Target Smoothing)
5e -3
33
Number of Particles
16
16
Search Horizon
4
4
Resample Period Initial !
4
4
0.5
Initial a
0.5
Initial CPp,
10
Initial ozg
500
Ey
0.2
0.5
Eo
le
Se- 2
Se 4
Dirichlet Alpha
0.03
Root Exploration Weight
0.25
25

---

D.4 Intuition for Temperature and Exploration:
The temperature parameter Y plays crucial role in balancing exploration and exploitation during the search process SPO. In this work, it is derived via first choosing the desired KL divergence between the target policy q: generated by search and the current policy m. the temperature is too low (i.e., 7) small), exponential weighting exp (A" (s, a)/n) becomes sharply peaked around the highest- vavattage actions. This causes the importance weights to concentrate on few particles, and during resampling, the particles can collapse onto single root action. Such collapse reduces diversity in the search and renders the rest of the planning process ineffective. Conversely, if the temperature is high (i.c., 17 large), the weighting becomes flatter, leading to excessive exploration. While exploration is necessary, too much can prevent the search from effectively focusing on promising paths. This work shows that deriving this value via target KL leads to stable and strong performance across different environments.
Baselines
For the baseline implementations, we expanded and adapted the existing implementations from Stoix [80]. All implementations were conducted using JAX [14], with off-policy algorithms leveraging Flashbax [81].
E.1
Hyperparameters
E.1.1
Continuous Control
Table 4: Hyperparameters for PPO and Sampled AlphaZero
Parameter
Sampled AlphaZero
Actor Learning Rate
0.00069
3c
Critic Learning Rate
0.00054
3e
Rollout Length
16
32
Number Epochs
64
Number of Minibatches
16
Buffer Size
65536
Batch Size
32
Sample Sequence Length
32
Discount Factor
0.99
0.99
GAE Lambda
0,95
0.95
Clip Epsilon
0.1
Entropy Coefficient
0.005
0.005
Value Function Coefficient
Max Grad Norm
0.5
0.5
Decay Learning Rates
True
Standardize Advantages
True
Number of Simulations
64
Dirichlet Alpha
0.03
Dirichlet Exploration Fraction
0.25
Number of Samples
20
Gaussian Noise Exploration Fraction
0.001
'Available http://ggitbbb.coom/EEEanoottttt
26

---

Table 5: Hyperparameters for MPO and VMPO
Parameter
MPO
УMPO
Rollout Length
8
32
Number of Epochs
72
16
Buffer Size
200000
Batch Size
256
Sample Sequence Length
16
Period
1
Actor Learning Rate
le
3c
Critic Learning Rate
3c
Dual Learning Rate
le
le
7 (Target Smoothing)
0.005
0.005
Discount Factor
0.99
0.99
Max Grad Norm
0.5
0.5
Decay Learning Rates
True
Number of Samples
128
Gg
0.05
0.05
Eap
0.05
0.05
0.0005
0.0005
Initial
10,0
10.0
Initial Ctp.
10,0
10.0
Initial as:
500
GAR Lambda
0.95
0.95
Actor Target Period
25
Table 6: Hyperparameters for SMC-Ent
Parameter
SMC-Ent
Rollout Length
Number of Epochs
512
Buffer Size
500000
Batch Size
512
Actor Learning Rate
3e
Q Learning Rate
3e
(Target Smoothing)
0.005
Discount Factor
0.99
Reward Scale
10.0
Number of Particles
Max Depth
Resample Temperature
1.0
Number of Samples for Value Function
27

---

E.1.2
Discrete Control
Table 7: Hyperparameters for AlphaZero and PPO
Parameter
AlphaZero
PPO
Learning Rate
3c-4
3c
Rollout Length
21
85
Number of Epochs
16
4
Buffer Size
65536
Batch Size
64
Sample Sequence Length
17
Discount Factor
0.99
0,99
GAE Lambda
0.95
0.95
Max Grad Norm
0.5
0.5
Number of Simulations
64
Number of Minibatches
64
Clip Epsilon
0.1
Entropy Coefficient
le
Standardize Advantages
True
Table 8: Hyperparameters for MPO and VMPO
Parameter
MPO
УмРО
Actor Learning Rate
2e 4
3e 4
Decay Learning Rates
True
Dual Learning Rate
0.02
0.01
Number Epochs
En
0.00015
0.001
GAE Lambda
0.95
0.95
Discount Factor
0.95
0.99
Initial 17
3.0
3.0
Initial a
3.0
3.0
Max Grad Norm
0.5
0.5
Number of Samples
128
Q Learning Rate
0.001
Rollout Length
Sample Sequence Length
17
(Target Smoothing)
0.005
Batch Size
256
Buffer Size
500000
Actor Target Period
64
28

---

Table 9: Hyperparameters for SMC-Ent
Parameter
SMC-Ent
Rollout Length
Numher of Epochs
Buffer Size
1000000
Batch Size
2048
Q Learning Rate
3e
T (Target Smoothing)
0.005
Discount Factor
0.97
Max Grad Norm
0.5
Huber Loss Parameter
1.0
Entropy Temperature
0.03
Munchausen Coefficient
0.9
Clip Value Min
-1e
Number of Partieles
Search Horizon
4
Resampling Period
4
Resample Temperature
0.1
Reward Scaling
10.0 (Rubik's Cube) / 1.0 (Sokoban)
29

---

Expectation Maximisation
F.1
Overview of methods
EM algorithms differ small number of dimensions through which the algorithm be understood. The first dimension, G. the optimization objective that is maximized the E-step. This includes whether advantages, Q-values, or rewards are used, and with respect to which policy. We then consider whether a trust region constraint is used both in the E-step and M-step.
With defined optimization objective, various methods can be used to estimate it. Most methods derive an analytic solution to the optimization problem and then estimate this distribution using techniques such TD(0) function approximation. For example, AlphaZero does not explicitly derive analytic solution estimates solution to this optimization objective through Monte Carlo Tree Search (MCTS).
Table 10 provides summary some common EM methods, highlighting their core differences.
Table 10: Summary of EM based Algorithms
Method
E-step TR
M-step TR
g estimate
Depth Breadth
MPO
Q
Yes
Yes
Analytic Function Approximation Analytic + n-step TD top-k Adv
M
V-MPO
T
Yes
Yes
AWR
No
No
Analytic n step TD
1
AWAC
TT
No
No
Analytic + Function Approximation
1
POWER
log º
Yes
No
Analytic step TD
T
I
RWR
1) 1logr
No
No
1
REPS
A"
No
Analytic + TD(O)
AlphaZero
Q
Yes
No
MCTS
> 0 >
SPO
1aa
Yes
Yes
Analytic SMC
D
M
Note that I refers to an episode length. In additon we add two important dimensions of the G estimate (breadth and depth). Methods like MPO estimate the analytic solution directly using Q-function. This be leveraged estimate the target distribution for selection of M actions, but without leveraging depth, so rewards and futuree states are not used improve estimates. contrast, V-MPO leverages depth form an n-step estimate, but only for one of the actions, leading to relatively poor estimate of the analytic solution.
30

---

G Proofs and Discussions
G.1 E-step Analytic solution
We outline the analytic solution eq. (5). by writing a Lagrangian equation and solving for g. We can first represent our constrained optimisation exactly with the following optimisation problem and associated constraints. add state dependent baseline (8) the optimisation objective and notate (* (s)
as ( a).
max ()(()) If c(al)) v2) da ds
s.t. / (a(da))KLaaaaal)assss.... <e,
(13)
I (m())) [f o(als)da ds 1.
following Lagrangian can be constructed,
[(g,n.)) = Ma(s) [ ((ols) ds
+ n <- f- (a(()
/ a(cll) log
g(als) x(als,0:)
da da
(14)
+ + 1 ( ()(s)
If [(aal)) da ds
We can then solve for the value of that maximises this expression by taking the derivative with respect to g and setting it to O.
ÖL(g. n.
(3, log (ogg)(((s)) + log (als, (n + 7):
(15)
0q
analytic form the optimal distribution can then calculated as:
g(als) r[als, 0.) exp (s, a)
+ 77
exp
(16)
G.2 E-step KL constraining temperature
We now derive the dual function to minimised order to obtain ry within the analytic solution g. which crucial for enforcing the KL constraint.
The final term ntz acts as normalising constant since is independent of and therefore we can construct the following equality.
f (++>)=/)
17 1 17
(a)(a, da,
exp
raass 0s) exp
(17)
77
Now that we have expressions for Di2 and we can form the dual function g(r)) by substituting these terms back into original Lagrangian. After simplifying we recover the following expression.
((n) = n/c + (s()) log (ʃ (alss exp
(A9(a,)) () da ds
(18)
17
The optimal dual variable be calculated follows
arg min g(1).
(19)
31

---

G.3 E step: Constraint Optimisation
In optimising eq. (4), the first term (y(als) (Q" a)] is dependent the scale of reward in the environment. This make hard choose a, as the first and second mean that for each new environment new a should used requiring costly hyperparameter tuning to find. As explored in previous work [1. 76] we choose to enforce a hard constraint, as opposed to a soft constraint. Instead of choosing a we choose which is the maximum KL divergence between and I which is practically less sensitive to reward scale. We found that choosing all environments explored. This important there has been trend in recent years add hyperparameters Reinforcement Learning algorithms [2] resulting in the need for costly hyperparameter sweeps for algorithms to work on each new problem they are applied to.
arbitrary scales. Practically this
be far easier, resulting in value that generalised across
G.4 Proof of Proposition 1: Monotone Improvement Guarantee
The optimality of policy ia, log Prog (O 1), is lower bounded by the following ELBO objective:
T(g = Erog
(n'r aD (s.,0) + ((0).
We improve Ty by optimizing the ELBO alternatively via EM. We will outline the proof of the monotonic improvement guarantee of the ELBO using the EM procedure, at the i-th iteration of training. We note that this holds under the assumption that we calculate the true value of the closed form solution g, in the E-step, which is practice unlikely be the case.
E-step: By the definition of E-step, we improve the ELBO with respect to 9. We show the E-step update will increase ELBO:
4:+1 = arg max B. [Bane((ss) [A" (s, a]] (aDxxalll))lll(sssss 0.]]
arg max Brog
( (otee-DDxx((/)))tttte)00..
= arg max [4 0i)
= I(4+1, n8.) 2 I ).
M-step: We update 0 by
0:41 = arg max E [alB (112) [o To [a]ss] + log l(g)
arg max E 1-DDx 1944 (-1sx) (-|s. 0)] log
arg max E (awn)1((1) (A (s, a)] ( 0)] p(0)
arg max J (g)41,
Therefore, we have: [4:+1, to. ) 2 [4:+1, T..) Combining these two results we have that after successive applications of the E-step and M-step,
(1++1, me.1) 2 J no,).
G.5 M-step objective
In Reinforcement Learning it can be beneficial constrain policies from moving too far from the current policy, often leading increased stability performance of an algorithm [69, 72]. In this work utilise Gaussian prior around 0c, our current value of 6, optimised from the previous iteration.
Therefore, N(,, 2)) where μ 6. and -ー1 (O.).. where F is the Fisher information matrix. The prior term in eq. (11) can then be written as log p(0) - X 0²)² F(0.) -(0))-1(0 0.) + c. The first term in this expression the quadratic approximation of the KL divergence [69] while the second term e is term that does not depend on and therefore when optimising eq. (11) with respect to 0, can be dropped. Using this approximation we rewrite eq. (11) as:
max (+) [Eowgtals) [log (aas, 0)] A KL (ala)s, 0,) ((ass, 0))]
Like the E-step, choosing À can be non-trivial so we convert it into a hard constraint optimisation eq. (12). Note that e in eq. (12) is different from eq. (5). We note that considering Gaussian prior in the M-step is not strictly required for SPO performance, however practically it adds stability to training.
32

---

G.6 Connection to Mirror Descent Guided Policy Search
We highlight the connection between the use of Expectation Maximisation to improve the evidence lower bound and Mirror Descent Guided Policy Search (MD-GPS). Mirror Descent Guided Policy Search builds upon previous guided policy search work [46]. Rather than only enforcing the constraint between g and at convergence (referred to a the local policy:p; and global policy: Ta respectively), they constrain yi against the current policy R; every iteration. This results in a very similar algorithm to EM optimisation. Below we provide the outline of MD-GPS algorithm, clearly showing the equivalence EM [58]:
Algorithm 3 Mirror descent guided policy search (MDGPS): convex linear variant
1: for iteration & E {l,.... K} do
2: C-step: ps <- arg minp, Ep,(r) Ei ((x,, u.) such that DRL mo(r)) S
3: S-step: Te <- arg ming E; DKL (Pi(T) (T(TT)
4: end for
Comparing MD-GPS EM, the C-step equivalent to the E-step and S-step equivalent the M-step, with the loss being the negative the expected discounted sum of returns over trajectories. MD-GPS assumes that the S-step can perfectly minimise the objective in the case of linear dynamics and quadratic costs [48], resulting in exact mirror descent. In practice, most applications of MD-GPS will not satisly such constraints and so are akin approximate mirror descent. However, long constraint between the local and global policy can be enforced terms of KL, various bounds on cost of the global policy can be constructed, see Montgomery and Levine (58) for further details.
33

---

H
Statistical Precipice
H.1 Overview
Advancements in computational power and algorithmic capabilities have led shift in the evaluation of reinforcement learning algorithms, now typically assessed through extensive suites of tasks. Performance metrics such as the mean or median score per task are commonly used, but these metrics can fail to account for the statistical uncertainties arising from limited number of runs and varying random seeds. The trend towards computationally intensive benchmarks further complicates the issue, as cach run span from hours to weeks, making impractical to conduct numerous runs per task and thereby increasing the uncertainty the reported metrics. To address these challenges, we adopt the evaluation methodologies proposed by Agarwal et al. [3] and Gorsane et al. [32].
Each algorithm is evaluated across M tasks within a specified environment suite, with N independent runs per task m E M. During each run E N, performance is measured over E episodes, each consisting of T timesteps. At each interval i, the mean return G is computed, and the model is checkpointed. The model with the highest mean return across intervals selected for final evaluation.
Following the completion of training run n E N. the best model is further evaluated over 10 x E episodes. We normalise scores Tm,, for each task m = 1, , .M and run n = I,...,N, .N, scaling them based on the minimum and maximum scores observed across all runs. This normalization produces set of normalised scores 21.M.::N per algorithm. These scores are then aggregated into a single scalar estimate, ..
To ensure robust statistical confidence, we employ 95% confidence interval derived from stratified bootstrapping over the M N experiments, treating these as random samples. This method integrates the performance across all tasks and runs, simulating the statistical reliability of multiple runs on single task while considering task diversity. Results are reported for the entire suite.
Metries:
We utilize normalised scores to evaluate algorithm performance, employing metrics that go beyond simple median and mean calculations:
Interquartile Mean (IQM): This metric calculates the mean of the central 50% of runs, excluding the lower and upper 25%. It is more robust outliers than the mean and less biased than the median, offering higher statistical efficiency and detecting improvements with fewer runs [3].
Probability of Improvement: This measures the likelihood that algorithm X will outperform algo- rithm Y on a random task m, using the Mann- Sain-Wh U-statistic. It is defined as:
Pr(X > Y)
Pr(Xm. > Ym)
(20)
M
N K
Pr(Xm > Ym)
S( Bm
(21)
NK
v=
1,
if y <x
S(e.y)
y x
(22)
0, yf
Statistical significance is determined using the Seymann-Pearson criterion, based on the confidence interval bounds [13].
Additionally, performance profiles can be employed to visually compare methods, illustrating the fraction of runs that exceed given threshold. These profiles aid in identifying stochastic dominance and empirical performance bounds. We also plot the interquartile mean score against environment steps to evaluate sample efficiency.
H.2 Hyperparameters
Table 11: Statistical Precipice Evaluation Hyperparameters
Parameter
Value
Number of Games for In- Training Evaluation
128
Number of Games for End-of- -fainnin Evaluation 10 E
1280
Seeds per environment N
Tasks per suite M
Continuous=3, Discrete=2
34

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the
results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiscless settings, model well- -peciffifiitnnnnn
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that influence the performance the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex. text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
a'applicable, the authors should discuss possible limitations of their approach to address problems
of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: not derive any new theoretical results in this paper
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
35

---

Inversely, any informal proof provided in the core of the paper should be complemented by
Tormal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or providc access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g.. with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
Inference code be released at publication
Guidelines:
The answer NA means that paper does include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttps://nips.cm cctpppipcccpppccc////ccc
for more details.
While we encourage the release of code and data, we understand that this might not be possible. "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (c.g., for a new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (httpa: pettpa///nipa cc cc/public/ Peidee/CooeSubminn www.Suuwwissssoooooooooooooooooe isiinooiccy)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed
method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
36

---

At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Detting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them. full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Dnes the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include error bars for the main results of the paper in addition statistical significance tests in the appendix. additional scaling plots not include error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful
show in tables figures symmetric error bars would yield results that are out range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Guidelines:
The answer means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than experiments reported in the paper (c.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics
Answer: [Yes]
37

---

Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation
from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work advances the theoretical understanding and practical implementation of model- based RI. methods, with no explicit socictal impact, while indirectly contributing to future research and technological innovation
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not existing assets.
38

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the is used and, if possible, include URL.
name of license (e.g.. CC-BY 4.0) should be included each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperawi thcode. www.ohhwwod/aae..aaeoo...t...cca com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are alongside assets?
introduced in the paper well documented and is the documentation provided
Answer: [NA]
Justification: [NA]
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/revicw based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, not include any information that would break anonymity applica- ble), such as the institution conducting the review.
39


Reviews:
[{"review_id": "9PPXiIZThX", "reviewer": "Reviewer_R8DZ", "summary": "This paper introduces a novel iterative, particle-based approach to sequential Monte Carlo planning by combining expectation maximization with importance sampling. Their approach uses a model to sample real state-action trajectories in a problem domain then, after importance-weighting said transitions, computes a target policy distribution for the agent. This target policy is used for policy improvement. The paper supplies a theoretical derivation of their approach and provides experimental results against baseline algorithms.", "questions": "Line 204: How exactly is $\\hat{A}$ computed? It isn't very clear on that.\n\nHow much does removing samples based on weight affect exploration? Is this only helpful when you have a perfect model?", "limitations": "They broach the limitations of their work sufficiently.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper introduces a novel combination of previously established algorithms to tackle sequential decision making.\n\n**Significance:** The results of the paper demonstrate that their proposed approach is *easily parallelizable*, *per-iteration faster* and *explicitly tackles a shortcoming of a prior approach (i.e., addresses weight degeneracy)*. All these would make their work interesting to the planning community.\n\n**Clarity:** The paper is well written and easy to understand.\n\n**Quality:** The motivation for the work is strong as they study a problem of general interest to the community. The conceptual rationale for their approach is sound and the theoretical build makes sense. The experimental evaluation focussed on the claims the authors made and back up said claims. They not only show performance improvements against commonly used approaches but also improvements on computational costs.", "weaknesses": "I think that an evaluation with an imperfect model should have been included rather than left as future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their  positive review. \n\nRe: evaluation of SPO with an imperfect model. Our primary goal was to focus on the novel planning aspects of SPO and its use within training. We believe that the results demonstrate that SMC is a valuable planning method in its own right demonstrably outperforming other commonly used planning methods like AlphaZero when the model is known.\n\nHowever, we acknowledge that incorporating a learned model is a crucial next step for demonstrating practical applicability. Recent advancements in model-based methods, such as DreamerV3 [2] and EfficientZeroV2 [3], offer advanced architectures for modelling the underlying MDP. These methods are policy-learning agnostic and could easily integrate with the SPO algorithm. As long as the world model accurately predicts rewards, SPO should perform effectively. Additionally, Piché et al. (2018) [1] demonstrated the feasibility of using a learned model with a SMC-based search for maximum entropy RL. This prior work supports the potential integration of learned models in our approach, which we plan to explore in future work.\n\n**Questions**\n\n*Q: Line 204: How exactly is $\\hat{A}$ computed? It isn't very clear on that.*\n\nA: Practically, at each step in the sequence we estimate the advantage of a single state action pair using a 1-step estimate $\\hat{A}(a_t,s_t) = r(a_t,s_t) + \\gamma V(s_{t+1}, \\theta_i) - V(s_t, \\theta_i)$. As we perform search this we accumulate advantages at each step to calculate the overall importance weights of the sequence according to Eq 8. However, if desired, any advantage estimation technique that can be constructed during the search from a sequence of rewards and values. For example, GAE to allow a trade-off between bias and variance.\n\n\n*Q: How much does removing samples based on weight affect exploration? Is this only helpful when you have a perfect model?*\n\nA: This is a good question. The most important factor is actually the temperature variable which is controlled by the desired KL divergence $\\epsilon$ between the target policy $q_i$ generated by search and the current policy $\\pi_i$. If the temperature is too low, then during resampling, the particles can collapse onto a single root action which would make the rest of the search pointless. But when the temperature is too high, the search can be too exploratory and not effectively travel down promising paths. This does need to be considered when scaling the search method depth-wise and when choosing the resample period. This wouldn't only be helpful when using a perfect model as ultimately as long as a learned model has accurate reward prediction and dynamics prediction, the exploration/exploitation acts the same. We will amend our manuscript to include this explanation in order to provide greater intuition on the exploration of the SPO search.\n\nThank you once again for your valuable feedback.\n\n**References**\n\n[1] Piché, Alexandre, et al. \"Probabilistic planning with sequential monte carlo methods.\" International Conference on Learning Representations. 2018.\n\n[2] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\" arXiv preprint arXiv:2301.04104 (2023).\n\n[3] Wang, Shengjie, et al. \"EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data.\" arXiv preprint arXiv:2403.00564 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xIdjUMtYuG", "reviewer": "Reviewer_xUvG", "summary": "This paper propose SPO:  Sequential Monte Carlo Policy Optimisation, a RL algorithm with the Expectation Maximisation (EM) framework for MDP. Experiments on both discrete and continuous environments show that SPO outperforms baselines in terms of sample efficiency and scalability.", "questions": "See the weakness.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Quality: The paper clearly introduces the background of Sequential decision-making, Control as inference, Expectation Maximisation and Sequential Monte Carlo. \nSignificance: The performance improvement of the proposed algorithm is significant.", "weaknesses": "Originality: The paper propose SPO, which is the combination of Sequential Monte Carlo (SMC) sampling with the EM framework. The novelty is limited.\nQuality: The paper does not discuss what problem the paper aims to tackle. The paper uses SMC to better estimate the target distribution. But the paper does not give any experimental results showing better estimation results. Ablation studies are missing. Theoretical insights of proposition 1 are missing.\nClarity: The paper is hard to follow as it is full of formulas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**On Novelty:** Re: \"limited novelty'', we respectfully disagree with the reviewer. We highlight three contributions (all included in the manuscript lines 27, 39, 38): \n\n1. This is the first instance of Sequential Monte Carlo being used as a policy improvement operator for RL. To date, MCTS has been the most widely adopted search-based policy improvement operator. The paper presents a robust search algorithm, with a theoretical foundation, resulting in an operator that outperforms competing methods on a range of benchmarks.\n2. The introduction of an inherently parallelisable operator is also a strong contribution. We are not aware of any successful search based policy improvement operations that are able to leverage parallelism over search to speed up inference during training. In fact most works note that parallelisable versions of MCTS strongly reduce performance [1,2] and have not successfully implemented parallelisable versions of AlphaZero for policy improvement[3].\n3. Introducing a method that requires no algorithmic modifications between continuous and discrete environments while demonstrating strong performance in both settings, should not be considered as limited.  The generality of popular RL algorithms is often limited with algorithms such as AlphaZero requiring considerable modifications just to be applied to complex continuous domains. This is an important barrier for the applicability of search based policy \n\n**On Quality/Clarity:**\nThe manuscript highlights deficiencies with the current SOTA methods (lines 22-25). After introducing SPO (lines 26-36), we explain how SPO tackles each of the highlighted problems. To clarify, we aim to solve the scalability problem in current search based policy improvement methods, such that search can be executed in parallel without performance deterioration observed in previous works [1,2]. This allows for search methods to leverage hardware accelerators to perform search faster reducing train times (lines 38-39). We explain that SPO is widely applicable to discrete and continuous environments without modifications (lines 37-39). Lastly we explain SPO is theoretically backed  (line 29) and that the method scales for different levels of planning (line 41-42).\n\n \n\n**On Ablations:** \"Ablation studies are missing'' This is an incorrect statement from the reviewer. In the original manuscript we provide two important ablations. We ablate the difference between sampling the target policy in (6) and the alternative formulation without a baseline (line 660), important as previous works differ on these objectives [1,69]. We also provide a second ablation that demonstrates how the appropriate temperature for SMC varies significantly between environments (line 649). This backs up our approach of learning the temperature variable, and demonstrates this is an important contribution to reduce hyperparamter tuning. We add a 3rd ablation to elicit insight into how SPO enforces the KL constraint in policy improvement (Figs 1). We show two charts with varying runs for different maximum KL constraints. We see when viewed on the iteration level that SPO is able to accurately target the KL of choice (fig 1 a). Fig 1 b) shows the impact different KL constraints have on training curves.\n\nRe: \"the paper does not give experimental results showing better estimation results\". In practice we do not have access to the target distribution, therefore it is non-trivial to construct an ablation that directly measures how good methods are at estimating it. However, we add new ablation that leverages Monte Carlo samples to generate an unbiased estimate (given large compute budget). It compares the KL divergence of our SMC policy to a Monte Carlo oracle (Fig 3). We investigate the impact of depth and particle number on the KL divergence to the oracle for SMC on Sokoban. This concludes that scaling particles is particularly important for improving the target estimation, aligning with SMC theory [4], also that leveraging depth improves estimation.  Methods like MPO that limit particles to the number of actions and depth 0 are severely limited in their estimation of the target, likewise V-MPO only uses 1 sample albeit to a depth of N. Lastly comparing MPO to SPO there is a performance gap across all environments. Both methods have similar theoretical foundations the difference being how samples are obtained from the target. Therefore any performance differences can be attributed to changes in target estimation.\n\n**On Theoretical insights:** Similarly, the following statement made by the reviewer is untrue\n\"Theoretical insights of proposition 1 are missing\" We provide full theoretical insights for proposition 1 in Appendix section G.3 (line 749). This is referenced in the main manuscript on line 238.\n\n**On Formulas:** Many of the formulas are either existing work, or modifications to existing work. We believe that the provided theoretical justification is in line with the standards of rigour expected at NeurIPS. It is reasonable to expect that experts in the field would feel comfortable with work that provides such level of detail in order to clearly understand the theoretical foundations and methodology. If you think certain formulas could be re-positioned in the paper, with additional insights provided, we welcome specific feedback in order to increase readability.\n\n**References**\n\n[1] Liu, Anji, et al. \"Watch the unobserved: A simple approach to parallelizing monte carlo tree search\" arXiv preprint (2018)\n\n[2] Segal Richard B. \"On the scalability of parallel UCT.\" International Conference on Computers and Games. Berlin, Heidelberg: Springer Berlin Heidelberg 2010\n\n[3] Seify Arta, and Michael Buro. \"Single-agent optimization through policy iteration using monte-carlo tree search.\" arXiv preprint  (2020)\n\n[4] Del Moral et al Branching and interacting particle systems approximations of Feynman-Kac formulae with applications to non-linear filtering Springer Berlin Heidelberg 2000", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHLFIUIp2X", "reviewer": "Reviewer_X2nr", "summary": "This paper maximizes the expected reward wrt the policy by rewriting it as a log marginal likelihood in a probabilistic model where latents comprise of states, actions, and “optimality” observations.\n\nThe prior over states is the world model and the prior over actions is the policy which being optimized. The likelihood is the exponentiated reward which can be normalized to make it a distribution over auxiliary “optimality” variables.\n\nThe marginal likelihood can be optimized via EM where in the E step, we maximize the evidence lower bound (ELBO) in (3) with respect to the variational distribution q and in the M step, we fix q and maximize the objective with respect to the policy π.\n\nIn the E-step (Sec. 4.2), the paper derives an analytic expression for q via the Lagrange multipliers method (eq. 6).\n\nIn the M-step (Sec. 4.3), the paper uses this q to maximize the objective with respect to π. The key difficulty is that the q derived in the E-step is only known up to a normalizing constant and so the paper proposes to get approximate samples from q using SMC (Sec. 4.2.1).\n\nThe experiments compare the proposed method to baselines on several tasks with discrete and continuous action spaces, and show wins. There are ablations showing the tradeoff between using more particles vs unrolling for longer.\n\nIn addition to performance, the paper argues that SPO is preferable to the popular MCTS method because it parallelizes more easily and performs well across more domains.", "questions": "Major\n- SMC (Sec. 4.2.1): From (6) it seems like there is just one distribution we want to sample from instead of a sequence of them. What’s the sequence of distributions that SMC targets? In other words, in the background section about SMC, it’s unclear what is $p(x)$ and what is $p(x_t | x_{1:t - 1})$; how do terms in (8) relate to this $p$?\n- If I reverse-engineer a sequence of targets distribution from (8), it should be $p(a_{1:t}, s_{1:t}) = \\prod_{j = 1}^t \\mathcal T(s_t | …) \\pi(a_t | …) \\exp(A^{\\bar \\pi}(a_t, s_t) / \\eta^*)$. That is, the final target distribution is the transition times the policy times the exponent of the *sum* of advantage functions. I think this is incorrect since it doesn’t correspond to the sum of per-step rewards. Something that would make more sense to me is to have the incremental weights contain the exponentiated rewards for all time steps instead of the last one, where it’s the exponentiated advantage function. This is obviously a completely different algorithm.\n- I don’t understand why (5) follows from (4). We have an unconstrained optimization of a regularized objective in (4) which is turned into a constrained optimization problem in (5). I also don’t understand how $\\alpha$ turns into $\\epsilon$.\n- Why does (10) rewrite (9) as a constrained objective where it seems like in practice, we’re actually just optimizing (9).\n- How is the value function and the Q-function learned? These seem critical for the method but I couldn’t find a description.\n- E-step: I agree that it is difficult to optimize with respect to $q$ since it’s the distribution of the expectation and the term inside the expectation, $Q^q$ depends on it (eq. 4). The paper chooses to fix $Q^q$. I would have expected that it is the $q$ in $\\mathbb E_q$ that makes optimization difficult since typically we’d have to resort to the reparameterization trick or REINFORCE. If we fix the $q$ in $\\mathbb E_q$, optimizing the expectation is simple: just sample from $q$ and directly optimize $Q^q$.\n\nMinor\n- $\\gamma$ in (3) is not introduced before.\n- Should there be a likelihood term in the incremental weight in (2)? Or is $p(x_t | x_{1:t - 1})$ an unnormalized distribution? $p$ is not defined in this paragraph.\n- Swapping the prior term $\\log p(\\theta)$ in (9) to the KL term in (10) is a little odd. Would it simply not work otherwise?\n- In App. G.1, why is $\\mu_q$ dependent on $q$?\n\nOther relevant work\n- [AESMC](https://arxiv.org/abs/1705.10306), [FIVO](https://arxiv.org/abs/1705.09279), [VSMC](https://arxiv.org/abs/1705.11140)\n- [DVRL](https://arxiv.org/abs/1806.02426)\n- [SIXO](https://arxiv.org/abs/2206.05952), [NAS-X](https://arxiv.org/abs/2308.14864)", "limitations": "-", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Combining SMC and RL is interesting and the experiments show strong empirical gains. However, theoretically, I am not fully satisfied with the method itself—or at the very least, I found it difficult to understand the method fully from the description. See questions / comments below.\n\nTherefore, I’m giving this paper a reject in its current form but I am willing to raise the score if the method or its description is improved.", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review which will be used to clarify details of the paper\n\n1. We aim to sample from the target policy over sequences $\\tau= (s_0, a_0, s_1, a_1, ..., s_t, a_t, s_{t+1})$  similar to [60]. In the following notation we switch to using $\\tau$ from $x$ and give definitions for the density $p$ and proposal $\\beta$ at policy iteration $i$.\n\n* $p_i(\\tau_{1:t})  \\propto \\mu(s_0) \\prod_{j=0}^t \\mathcal{T}(s_{j+1}|s_j,a_j) \\pi_i(a_j|s_j, \\theta_i) \\exp\\left(\\frac{\\bar{A_i}(a_j,s_j)}{\\eta^*_i}\\right)$\n\n\n* $p_i(\\tau_t|\\tau_{1:t-1}) \\propto \\mathcal{T}(s_{t+1}|s_{t},a_{t}) \\pi_i(a_t|s_t, \\theta_i) \\exp\\left(\\frac{\\bar{A_i}(a_t, s_t)}{\\eta_{i}^*}\\right)$\n\n* $\\beta(\\tau_{1:t})  \\propto \\mu(s_0) \\prod_{j=0}^t \\mathcal{T_{\\text{model}}}(s_{j+1}|s_j,a_j) \\pi_i(a_j|s_j, \\theta_i)$\n\n* $\\beta(\\tau_{t}|\\tau_{1:t-1})  \\propto \\mathcal{T_{\\text{model}}}(s_{j+1}|s_j,a_j) \\pi_i(a_j|s_j, \\theta_i)$\n\nThe weight update:\n\n$$w_t(\\tau_{1:t}) \\propto w_{t-1}(\\tau_{1:t-1}) \\cdot \\frac{p(\\tau_t|\\tau_{1:t-1})}{\\beta(\\tau_t|\\tau_{1:t-1})} = w_{t-1}(\\tau_{1:t-1}) \\cdot \\left(\\frac{\\mathcal{T}(s_{t+1}|s_{t}, a_{t})}{\\mathcal{T_{model}}(s_{t+1}|s_{t}, a_{t})}\\right) \\cdot \\frac{\\exp(\\bar{A_i}(a_t, s_t)/\\eta_{i}^*) \\cdot \\pi_i(a_t|s_t, \\theta_i)}{\\pi_i(a_t|s_t, \\theta_i)}$$\n\n2. You are correct, the target distribution over sequences utilises the sum of advantages and not explicitly per step rewards. This does account for the sum of per step rewards, as we calculate $A^{\\pi}(a_t,s_t) = E_{\\pi}[r(a_t,s_t) + \\gamma V(s_{t+1})] - V(s_t)$ therefore for $\\gamma = 1$, over sequences, we have the following estimate $\\sum_{t=0}^h A^{\\pi}(a_t,s_t) = \\sum_{t=0}^h r(a_t,s_t) + V^{\\pi}(s_{h+1}) - V^\\pi(s_{0})$, \nthe sum of per step rewards plus the value of the last state, minus the value of the first state. If the weight only uses the per step rewards, future rewards beyond the planning horizon are not accounted for. This destroys performance, see new results fig 2. Without a baseline, the weight update would be $\\prod_{j=0}^t \\exp(Q^{\\pi}(a_t,s_t)/\\eta^*)$, see line 660 for ablation.\n\n3. In optimising 4, the first term $\\mathbb{E}_{q(a|s)} \\left[Q^q(s,a)\\right]$ is dependent on the scale of reward in the environment. This can make it hard to choose $\\alpha$, as the first and second terms are on arbitrary scales. As explored in previous work [1,69] we choose to change this from a soft constraint to a hard constraint (line 158). Instead of choosing $\\alpha$ we choose $\\epsilon$ which is the maximum KL divergence between $q$ and $\\pi$ which is practically less sensitive to reward scale.\n\n4. Eq 9 is a supervised learning step, we have choice over the parameterisation of our policy and prior for regularisation. RL algorithms often constrain policies from moving too far from the current policy, in this work we utilise a Gaussian prior around $\\theta_i$ optimised from the previous iteration. Therefore, $\\theta \\sim \\mathcal{N}(\\mu, \\Sigma)$ where  $\\mu = \\theta_i$ and $\\Sigma^{-1} = \\lambda F(\\theta_i)$, where $F$ is the Fisher information matrix. The prior term in (9) can written as $\\log p(\\theta) = -\\lambda(\\theta - \\theta_i)^T F(\\theta_i)^{-1}(\\theta - \\theta_i) + c$.  Note $c$ is a term that do not depend on $\\theta$. The first term is the quadratic approximation of the KL divergence [2*], so we can rewrite equation (9) as:\n\n$$\n\\max_{\\pi} \\mathbb{E}\\_{\\mu_q(s)} \\left[ \\mathbb{E}_{q(a|s)} \\left[ \\log \\pi(a|s, \\boldsymbol{\\theta}) \\right] - \\lambda \\text{KL} \\left( \\pi(a|s, \\boldsymbol{\\theta_i}), \\pi(a|s, \\boldsymbol{\\theta}) \\right) \\right]\n$$\n\nLike the E-step, choosing $\\lambda$ can be non-trivial so we convert it into a hard constraint optimisation (eq 10). $\\epsilon$ in Eq (10) is different from (5). The constrained M-step (with a Gaussian prior) is not required for SPO, however it adds stability to training.\n\n5. SPO does not require a Q-function and only learns a value function since we leverage observed rewards and state-values from planning to calculate advantages. We learn the value function like PPO, i.e using GAE values. GAE combines multiple N-step td-errors estimates,\n$A_\\text{GAE}(s_t) = \\sum_{l=0}^{N} (\\gamma \\lambda)^l \\delta_{t+l}$\nwhere $\\delta_t = r_t + \\gamma V(s_{t+1}) - V(s_t)$. Such that $V_{target}(s_t) = V(s_t, \\theta) + A_\\text{GAE}(s_t)$. These value targets are not constructed during planning but on real rollouts. It is a practical choice as to what method is used to generate targets, we opt for GAE to balance bias and variance.\n\n6. In Eq 4 when optimising with respect to q there is a moving target, each update to q results in a new critic $Q^q$. Our approach aligns with most actor-critic approaches [65] that fix a critic within an iteration of policy optimisation. For the suggestion of optimising the $Q^q$ term and fixing the acting policy  $q$, this would be a highly non-trivial optimisation. Since $Q^q$ depends on the transition dynamics and reward function of the environment unless we knew these we would need to perform off-policy evaluation over the space of possible policies $q$, with such evaluation known to be extremely challenging [1*], as behaviour of policies is difficult to estimate using different distributions. We are not aware of RL algorithms that approach optimisation from this perspective.\n\n**Minor**\n1. $\\gamma$ is introduced in the background as the discount factor however we will add to 3) for clarity\n\n2. $p(x_t| x_{1:t-1})$ is unnormalised, we define p(x) before eq 2) as an arbitrary target distribution\n\n3. See major q4\n\n4. $\\mu$ is the distribution over states. Any policy acting in the MDP will induce a different state distribution. We therefore denote $\\mu_q$ in Eq 11 as this relies on the policy $q$. We will add this clarification to the manuscipt.\n\n**References**\n\n[1*]  Dudık et al \"Doubly Robust Policy Evaluation and Learning\" ICML 2011\n\n[2*] Schulman et al \"Trust region policy optimization\" International conference on machine learning PMLR 2015", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In this section we provide additional results/experiments in response to reviewer xUvG and reviewer X2nr. 

**Additional Results**

**Figure 1**

While we give in depth details regarding how the KL constraint arises within SPO and the method through with it is enforced (see Appendix G.2). In the original manuscript we did not provide concrete evidence of this KL control aside from showing how important the temperature can be for training (see ablation B.1). Therefore we add an ablation that plots both the performance of various SPO training runs with different values of $\epsilon$ ,on Brax, but also an iteration level mean of the KL divergence between the $\pi_i$ and $\hat{q}$, the current policy and estimate of the target $q$. We expect this value to stay very close to the target, as for each iteration a new temperature value is calculated to directly target this measure. The results show two things, firstly that our method of KL control is very successful and stable. Secondly it shows the importance of KL divergence in convergence in performance.

**Figure 2**

In response to queries regarding the weight update we use in SPO, we choose to add an additional experiment to the ablation in section B.2 . We remove the use of the action independent baseline, but also the use of the bootstrapped value function. This results in the weight update simply according to observed rewards. The provided results in figure 2 show that not taking account of the future rewards in the final state severely impacts performance ontop of the reduction in performance from not utilising the value baseline.

**Figure 3**

In figure 3 we aim to address feedback that we do not clearly show or measure the estimation of the target distribution. We do believe that our performance results are strong evidence for better estimation of the target but agree that some insight into target estimation more directly will help strengthen the conclusions that can be taken away from the paper. Of course in practice the target distribution is unknown. As we do not have perfect advantage estimates. However given a fixed iteration of training, while completely unachievable to perform during training, for the purposes of ablating, we can perform a large scale Monte Carlo Simulation to form unbiased estimates for the advantage function. 

In this ablation we perform Monte Carlo estimation of the Q-value using 1280 rollouts to the end of the episode for every state and every action. Advantage estimates can then be formed using the following

$\hat{A}(a_t,s_t) = \hat{Q}(a_t,s_t) - \sum_{a \in A} \pi(a|s_t) \cdot \hat{Q}(a_t,s_t)$.

This allows us to form an estimate of the target distribution for a single state as follows

$q_{i}(a_t|s_t) \propto \pi(a_t|s_t, \theta_i) \exp\left(\frac{\hat{A}(s_t,a_t)}{\eta^*}\right)$.

Using this accurate unbiased estimate we then measure the KL divergence of the Monte Carlo Oracle to SPO which comparatively uses far lower compute. Within this ablation we scale Depth and Number of particles to demonstrate the impact these variables have on estimation. We provide results on Sokoban across 1280 episodes.

The results in Figure 3 demonstrate how important scaling particles is in improving the estimate of the target distribution, with significant drops of KL achieved. We also see that depth contributes to the reduction of the KL divergence. This highlights that methods that estimate the same target but that do not perform planning i.e. using a depth of 0, or limit the number of actions that the distribution is estimated over [1, 69] will have much worse estimation of the target distribution directly impacting the speed of EM training.

Meta-review: The importance of Monte Carlo planning in RL is well known (see AlphaGO), and this paper proposes extensions to Sequential MC. The proposed methods are interesting and justified mathematically. The scores given by the reviewers were mixed, with an average on the positive side. To emphasize the value of the paper, one could mention that the mathematical justification of this paper establishes a close relationship with the use of Monte Carlo methods in Active Inference, which tries to solve reinforcement learning problems without the use of rewards. In particular, the following past NeurIPS paper is very related: “Fountas, Zafeirios, Noor Sajid, Pedro Mediano, and Karl Friston. Deep active inference agents using Monte-Carlo methods. Advances in neural information processing systems 33 (2020): 11662-11675.”. We believe that the current paper could be seen as a step to narrow the gap between traditional reinforcement learning and active inference (which as seen through the work of Fountas et. al is still problematic). Given this parallel interest in MC methods in neuroscience, we believe that the current paper on pushing forward RL and SMC methods is useful, and it may eventually help to bring the two communities together.
Decision: Accept (poster)