Trust-Region Twisted Policy Improvement
Authors: Joery A. de Vries
Jinke He
Yaniv Oren
Matthijs T. J. Spaan
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning; Sequential Monte-Carlo; Monte-Carlo Tree Search; planning; model-based; policy improvement
Primary area: reinforcement_learning->planning
TL;DR: We tailor the design choices within the particle filter planner for its specific use in reinforcement learning inspired by approaches from related Monte-Carlo tree search literature.

Abstract: Monte-Carlo tree search (MCTS) has driven many recent breakthroughs in deep reinforcement learning (RL).
However, scaling MCTS to parallel compute has proven challenging in practice which has motivated alternative planners like sequential Monte-Carlo (SMC). 
Many of these SMC methods adopt particle filters for smoothing through a reformulation of RL as a policy inference problem.
Yet, persisting design choices of these particle filters often conflict with the aim of online planning in RL, which is to obtain a policy improvement at the start of planning.
Drawing inspiration from MCTS, we tailor SMC planners specifically to RL by improving data generation within the planner through constrained action sampling and explicit terminal state handling, as well as improving policy and value target estimation.
This leads to our *Trust-Region Twisted* SMC (TRT-SMC), which shows improved runtime and sample-efficiency over baseline MCTS and SMC methods in both discrete and continuous domains.

Trust-Region Twisted Policy Improvement
Joery A. de Vries
Jinke He
Yaniv Oren' Matthijs J. Spaan'
Abstract
et al., 2021; Danihelka al., 2022; Antonoglou et al., 2022; Oren et al., 2025).
Monte-Carlo tree search (MCTS) has driven many recent breakthroughs in deep reinforcement Iearn- ing (RL). However, scaling MCTS to parallel com- pute has proven challenging in practice which has motivated alternative planners like sequential Monte-Carlo (SMC). Many of these SMC meth- ods adopt particle filters for smoothing through a reformulation of RL as a policy inference problem. Yet, persisting design choices of these particle fil- ters often conflict with the aim of online planning in RL, which is obtain policy improvement at the start of planning. Drawing inspiration from MCTS, we tailor SMC planners specifically to RL by improving data generation within the planner through constrained action sampling and explicit terminal state handling, as well as improving pol- icy and value target estimation. This leads to our Frust-Region Twisted SMC (TRT-SMC), which shows improved runtime and sample-efficiency over baseline MCTS and SMC methods in both discrete and continuous domains.
Although MCTS has been a leading technology in recent breakthroughs, the tree search is inherently sequential, can deteriorate agent performance at small planning budgets (Grill et al., 2020), and requires significant modifications for general use in RL (Hubert et al., 2021). The sequen- tial nature of MCTS is particularly crippling it limits the full utilization of modern hardware such as GPUs. De- spite follow-up work attempting to address specific issues, alternative planners have since been explored that avoid these flaws. Successful alternatives in this area are often inspired by stochastic control (Del Moral, 2004; Åström, 2006), examples include path integral control (Theodorou et al., 2010; Williams al., 2015; Hansen et al., 2022) and related sequential Monte-Care (SMC) methods (Naesseth et al., 2019; Chopin & Papaspiliopoulos, 2020).
Specifically, recent variational SMC planners (Naesseth et al., 2018; Macfarlane et al., 2024) have shown great po- tential in terms of generality, performance, and scalability to parallel compute. These methods adopt a particle filter for trajectory smoothing enable planning in RL (Piché et al., 2019). However, the distribution of interest for these particle filters do not perfectly align with learning and ex- ploration for RL agents, Namely, recent SMC methods focus on estimation of the trajectory distribution under an unknown policy, and not the actual unknown policy at the state where we initiate the planner. We find that this mis- match can cause SMC planners suffer from unnecessarily high-variance variance estimation and waste much of their compute and data during planning. In other words, online planning in RL should serve as local approximate policy improvement (Chan et al., 2022; Sutton & Barto, 2018). Fortunately, exist- ing MCTS and SMC literature provides various directions to achieve this (Moral et al., 2010; Svensson et al., 2015; Law- son al., 2018; Danihelka et al., 2022: Grill al., 2020), but their use in SMC-planning remains largely unrealized.
1. Introduction
Monte-Carlo tree search (MCTS) with neural networks (Browne et al., 2012) has enabled many recent successes in sequential decision making problems such as board games (Silver et al., 2018), Atari (Ye et al., 2021), and algorithm discovery (Fawzi et al., 2022; Mankowitz et al., 2023). These successes demonstrate that combining decision-time planning and reinforcement learning (RI.) often outperforms methods that utilize search or deep neural networks in iso- lation. Naturally, this has stimulated many studies to un- derstand the role of planning in combination with learning (Guez et al., 2019; de Vries et al., 2021; Hamrick et al., 2021; Bertsekas, 2022: He et al., 2024) along with studies improve specific aspects of the base algorithm (Hubert
This paper aims to address the current limitations of boot- strapped particle filter planners for RL by drawing inspira- tion from MCTS. Our contributions are 1) to make more accurate estimates of statistics extracted by the planner, at the start of planning, and 2) enhancing data-efficiency inside the planner. We address the pervasive path nath-degeneracy prob-
Delft University of Technology, Delft, the Netherlands. Corre- spondence to: Joery A. de Vries <.J.A.deVries @uueeettu...........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Trust- Trust-Region Twisted Policy Improvement
lem in SMC by backing up accumulated reward and value data to perform policy inference and construct value targets. Next, to reduce the variance of estimated statistics due to re- sampling, we use exponential twisting functions to improve the sampling trajectories inside the planner. We also impose adaptive trust-region constraints to a prior policy, to control the vars-nariance trade-off in sampling proposal trajecto- ries. Finally, we modify the resampling method (Naesseth et al., 2019) correct partieles that become permanently stuck in absorbing states due to termination in the baseline SMC. We dub our new method Trusi-Region Twisted SMC and demonstrate improved sample-efficieny and runtime scaling over SMC and MCTS baselines in discrete and con- tinuous domains.
Definition 2.1. The posterior factorizes
T Pr (=(H::(O1rr) = PT (A.lS, (ar)) 1 Ar-)),
assuming that S141 OurT/S, A. and A, 11 OctlS
The key part of Definition 2.1 is the posterior policy P- (A,)S,, OCT) which, using Bayes rule, reads as
Pr P+ (A. OcT) x (Ar (S)) exp(In((O..1(s,. A:)] (2)
This connects us to an (expected-reward) maximum-entropy reinforcement learning setting (Toussaint, 2009; Ziebart, 2010), since the exponent admits a offt-Bellman recursion,
2. Background
In pmp(Or))S A.) = R(S,, A.) + In E Ee (S+(1)
(3)
We want to find an optimal policy for sequential decision- making problem, which we formalize as an infinite-horizo Markov decision process (MDP) (Sutton & Barto, 2018). We define states S E S, actions A E A., and rewards R E R as random variables, where we write H1.r = {St, A.
where the expectation is over the dynamics p(S |St, At). An important property of the posterior policy is that it not an optimal policy in the traditional objective E,,, |E, Re). but only provides improvement over a prior policy n.
as the joint random variable of finite sequence.
Theorem 2.2.
The posterior policy p(Ar|S,, OcT),, Vt, is the optimal policy q' for the regularized MDP,
Pr (*(H)T)
II T (4115))(5
(1)
q' arg max E GETT
Σ - KK(g(a(s)))(((sss))
where Ao. Su) e p(S) is the initial state distribution, (S Ar) is the transition model, and o (Ac(SS) is the
where q" guarantees policy improvement in the unregudar- ized MDP, pp, (H.rr) E
policy. We denote the set of admissible policies as II A {π|m: S {[))], our aim is to find parametric mo IT (e.g., a neural network) such that Ep.p (H.rr) EEi=1 R&] is maximized, where we abbreviate R₂ = R(S,, Ac) for the rewards. For convenience, we subsume the discount fac- t E [0. 1] into the transition probabilities as a termina- tion probability of Prem = 1 y assuming that the MDP always ends up in absorbing state Sz with zero rewards.
R.) 2 Ep-(Her)) E2 Ra).
Proof. See Appendix A.3 or Sec. 2.4 of (Levine, 2018).
The optimal regularized policy q" can be interpreted variational policy' that minimizes the evidence gap (Bishop, 2007), or conversely, maximizes the evidence lower-bound to In (O)r)). In practice, this can be used in expectation- maximization (EM) methods (Neal & Hinton, 1998) to itera- tively update the prior policy ([++1) <- Knn) This gives rise to an effective framework for approximate policy iteration that is both amenable gradient based updating of Ta and provably recovers the traditional (locally) optimal policy ra Ep, |E, R.] as + oo (see Appendix A).
2.1. Control as Inference
The reinforcement learning problem can be recast prob- abilistic inference problem through the control as inference framework (Levine, 2018). This reformulation has lead to successful algorithms like MPO (Abdolmaleki et al., 2018) that naturally allow regularized policy iteration (Geist et al., 2019), which is highly effective in practice with neural net- work approximation. Additionally, it enables us to directly use tools from Bayesian inference on our graphical model.
2.2. Particle Filter Planning for RL
Although the estimation of the regularized optimal policy 9" mitigates number of practical challenges in deep RI., it also requires re-estimating Gin) after each update to the prior ()+ Additionally, the posterior policy for any x always requires the solutions to the soft value-estimation problem for 2%0ft and ya
To formalize this, the distribution for HAr can be condi- tioned on binary outcome variable O, E (0. 1}. then given a likelihood for p = 1|Hirr) = p(O this Typ- ically, we use the exponentiated sum of rewards for the (unnormalized) likelihood p(O H1r)) x IT= exp Re.
soft- Algorithms like MPO deal
gives rise to a posterior distribution p(Hrr|
From this point we will interchange the regularized optimal policy Loog (Ae Ss) with the posterior policy PT(.) (Ar (Se Ocr).
2

---

Trust- Trust-Region Twisted Policy Improvement
with this by approximating the value using neural net- works Q Loof to then estimate the posterior policy through Monte-Carlo. Intuitively, this can be interpreted as a 1-timestep approximation to q² To see this, evaluat- ing Q (St, Ar) over samples A, ~ T ( (A.)St) amortizes the message-passingg process for evaluating Q=oftt into a direct (cheap) mapping from S x A + RR. The main limitation of this approach is the bias induced by this new function Q7. Towards this end, sequential Monte-Carlo (SMC) methods (Piché et al., 2019; Naesseth et al., 2019) offer power- ful model-based strategy for improving the estimate of q² through a multi-timestep.
Algorithm 1 Bootstrapped Particle Filter for RL
Require: K (number of particles), m (depth)
1: Initialize:
Ancestor identifier {1(²) i
States {s(() ~ p(Si));=
Weights (w7)) 1)
for t to m do
// Update particles
3: {4,0) (8) ~ (AA
4: [ ~ P(S ((()
)
5: ( = : ald, o 15(0
*<A R Eexp V2 (S
E
exp V (s liml
The SMC algorithm for RL (Piché et al., 2019) is a sequen- tial importance sampling (IS) method that aims to draw samples H1 from the posterior through a proposal distribu- tion Pa (H1:) (as in Eq. I for some q E II). By definition, our graphical model (Def. 2. factorizes recursively,
// Bootstrap (periodically) through resampling
{(J. s(f)
( Multinomial(K, w.)
(411
7: { 1)
8: end for
{
(1) 1K Hlom
Pr Pr.Hir Pr (H1L)O0 ( (H++1 HH, Oct1:), (4)
9: return
11mm J i=1
meaning that we can sample data from Hi Pg (H1)) and accumulate the wS-weigs sequentially.
Bootstrapped Filter The bootstrapped particle filter for RL, as proposed by Piché et al. (2019), improves the esti- mation in Eq. 5 through (periodic) resampling. This miti- gates the issue of weight-imppverishment in sequential-IS, where some normalized weights dominate others wF(") > of .j + i. A common strategy for this is multino- mial resampling (Chopin & Papaspiliopoulos, 2020), as shown in Algorithm 1. This method samples a number of traces H ~ Pa. i E [], referred to as particles, and periodically drops or duplicates these samples based on their weights iim m E [1, before resetting their weights back to a uniform distribution (bootstrap). Prior D
Corollary 2.3 (Sequential importance sampling). Assuming access to the transition model p(S++1 S,, As). we obtain the importance sampling weights for ps 1: (.TT) h/T) 1H))),
T(Ac(S:) (()))
E Texp Vooee (S1+1) exp
W/= = W1-1
Re)
V saft (S))
(())
Proof. The dynamics terms in the weights cancel out, the rest follows by definition from Equations 2, 3, and 4.
work (Piché et al., 2019; Macfarlane et al., 2024) then esti- mates the policy q" as a weighted mixture of point-masses,
In practice we cannot realistically compute We since it re- quires the valu-s soft- However, we can again approx- imate this Wi R we with the amortized estimate V/" ~ V at every intermediate timestep (Pitt & Shephard, 1999; Law- son et al., 2018). With this, we can estimate posterior statistics using a single forward pass through: sampling traces (1()), accumulating their approximate weights (0)
(A, = a = Σ
W
(i) 4+m 5
(AA
=a),
(6)
where _( E [1, K] is the index tracking each samples' ancestor at the start of planning t and 8(-) is Dirac delta function over A for the ancestor action-particles.
to obtain and then normalizing these w!() E, mp
A key property of Algorithm is that it can estimate i' through a single forward pass of K particles from t to t + m (Moral et al., 2010). This allows for parallel sampling and updating of particles, with communication only required dur- ing the resampling step. As a result, it scales efficiently on modern GPU hardware, with memory complexity of O(K) and a parallelized time complexity of O(m).
E (0) = Epg [w: [(H::))
(5)
(3) w ((H()))
H12 ~ Pr:
where f 7 + y is some arbitrary function over the data. For instance, the statistic fHzx) = EE== Ry would yield an estimate of the expected hnite-horizon sum of rewards.
Variational SMC Intuitively, resampling improves our es- timate of Eq. 5 by 'correcting' particles with oilliikel
3

---

Trust-Region Twisted Policy Improvement
to higher likelihood regions of the target distribution. How- ever, resampling also introduces noise and would ideally not be needed (or to lesser extent) if our proposal distribu- tion Py produced samples that better match our target. To this end, variational SMC methods (Naesseth et al., 2018; 8; Gu et al., 2015) learn a proposal distribution do from sam- ple estimates of the posterior target E[[""] = q². Macfar- lane et al. (2024) use this strategy with neural networks to learn Go q²" Given the neural network iterates (0(m) } their method also updates the prior policy T(++1) f (a at every step In other words, they combine variational SMC in an EM-loop (Neal & Hinton, 1998) by using the learned proposals .0 as the prior to regularize the poste- rior P (A.|S,, OLT))
quent resampling, as it can take multiple transitions before moving particles to rewarding trajectories (Lioutas et al., 2023).
For this reason, we extend the proposal distribution to more accurately reflect the next posterior policy Pro., (A, (Ar]St, Or.T) by using both a learned policy go = ((n) and value exp Q5. This is also known as exponential twist- ing (Asmussen & Glynn, 2007) of the proposal, which is similarly used in the target distribution (Corollary 2.3). Twisting reduces the variance for q" at the cost of some bias due to Qī. This enables lower partiele budgets through im- proved particle efficiency, but also introduces the difficulty in controlling this trade-off. In this regard, MCTS- based methods can offer some insight for dealing with this.
3. Tailoring Particle Filter Planning to RL
Since AlphaZero (Silver et al., 2018), MCTS-based meth- ods also often use trained policy Ta (conflated with the name prior distribution) to guide the search in combination with a P-UCT algorithm (Rosin, 2011). Crucially, the ini- tial iterations of the algorithm rely more on Ta, which is interpolated to greedy policy over the estimated values Q in later iterations. Grill et al. (2020) show how this causes inferred policies from MCTS track a regularized objec- tive similarly to Theorem 2.2 (with an added greediness parameter) over consecutive iterations. This is relevant to our work, because it links the estimated quantity of our SMC planner to the approach taken by MCTS. The main difference persists in that the iterations of MCTS induce an adaptive regularizer through a proxy for accuraccc
A key benefit of online planning in reinforcement learn- ing (RL) is generating locally improved policies at each timestep & compared to a prior policy, which enhances learn- ing targets and diversity of data (Hamrick et al., 2021; Silver et al., 2018). Despite recent progress in improving sequen- tial Monte-Carlo (SMC) methods for local approximate policy improvement (Chan et al., 2022), persisting design choices from conventional particle filtering do not align di- rectly with this goal. For instance, we find that the problem of path degeneracy, inherent to particle filtering (Svens- son et al., 2015), can cause policy inference in Eq. to degenerate with deep search and waste much of the plan- ner's generated data. We also argue that variational SMC planners still make inefficient use of their particle budget due to periodic resampling, which can cause particles to get stuck in terminal states and delays using value information. By contrasting this with Monte-Carke tree search (MCTS), which specifically focuses on improving the policy at the root (Grill et al., 2020), we tailor the SMC-planner for local policy improvement. This preserves the forward-only imple- mentation of partiele filtering while incorporating benefits inspired by MCTS.
Inspired by this, we formulate our proposal distribution through constrained optimization problem with adap- tive trust-region parameter en E IRzo. At each state S,, the proposal solves for a locally constrained program,
Q7 (S.. At),
max gEP(AASE) s.t., KL(g(a/Si)] S Ea:
3.1. Adapting the Proposals for Sample-Efticieney
The resampling step in SMC (line 6, Alg. 1) is essential for redistributing particles that are unlikely under the posterior policy. Simultaneously, variational SMC methods ( (Mac- farlane et al., 2024) can shift some of this responsibility to a learned proposal distribution go x g" by generating better samples. This is useful as reduces variance induced by resampling and amortizes posterior inference (Naesseth et al., 2018). However, in EM-loop style algorithms, the learned proposal distribution go is an estimate of the pos- terior policy that is regularized to prior from a previous iteration & 1T (A, St, Over), ), and not to the current iteration ((n) This is computationally wasteful with infre-
where E [0, 1] is greediness tolerance level. The actual trust-region e,, is then sandwiched between the prior Te and the greedy policy π² over Q7. such that, (a = a KL(T" In similar spirit to MCTS, this guaran- tees that SMC always searches trajectories that interpolate between maximizing Q, or sticking to the prior TT. The La- grangian of this program is similar to Theorem 2.2 with the sum of rewards replaced by Q and the KL term scaled by temperature. Furthermore, Eq. 7 requires KT(g/TT) S at every S, E S instead of in expectation over (H). For solving this program, we use a bisection search using boot- strapped atoms from ne. which is both general and compu- tationally cheap (see Appendix B.5 for details).
4

---

Trust-Region Twisted Policy Improvement
3.2. Handling Terminal States with Revived Resampling
State
Although the control as inference framework from Sec- tion 2.1 enables the use of SMC methods for policy in- ference, it also introduces non-trivial caveats. In particular, the infinite horizon formulation in Eq. 1 can lead to wasting compute when handling terminal states as absorbing states in a foly SMC algorithm. Absorbing states result in transitions that loop back to the same state with zero reward, effectively treating this as part of the environment dynamics. In an SMC planner, this can cause some of the K particles to become trapped in these absorbing states. Al- though resampling should correct for this, reward sparsity and errors in value predictions V = soft can render the weights to become nearly uniform, making these trapped particles indistinguishable from non-trapped ones. This is a problem, because trapped particles do not contribute to gathering information about future values.
15
20
35
Depth
Dirac Mixture (baseline)
Message Passing (ours)
2.0
Num Partides
KLiH" [[swe]]
K=2
1.5-
K-4
1.0
K=8
K=24
0.5
K=2
20
Depth
Figure I. Illustration of path degeneracy in particle filters (top; adapted from Svensson et al., 2015) and the consequence on diver- gence from the optimal policy x¹ (lower is better). With a finite budget of particles and resampling, deep search will concentrate the Dirac mixture of remaining ancestors a single atom (hottom- left). We improve this through approximate message-passing to the ancestors, which does not degenerate the policy (bottom-right)
Furthermore, the resampling step can also move particles towards absorbing states due to rewarding trajectories in previous steps. Consecutive resampling in the SMC planner, however, is then likely to move these particles away from these states again because they no longer accumulate reward. This phenomena mostly problematic when performing policy inference according to Eq. 6 and is consequence of the path degeneracy problem (Svensson et al., 2015).
masses from Eq. 6 show deteriorating approximation quality as depth increases, as shown in Figure (bottom-left).
In contrast, MCTS does not suffer from path degeneracy because it does not discard any data. Policy inference is also typically done in MCTS by tracking normalized state- action visitation counter that is updated in each iteration of the algorithm (Browne et al., 2012). Importantly, the normalized visit-counts can be used as both a behavior pol- icy and learning target (Silver et al., 2018). Recent work however, has shown that using such a visitation-count policy can degrade performance when using low planning budgets. Instead, Danihelka et al. (2022) show that inferring regu- larized policy (Grill et al., 2020) using the value statistics from search for the root state-actions avoids this problem.
To address the trapped particles, we leverage the strategy of MCTS to reset trajectories to earlier states within the search tree. In MCTS, each iteration of the algorithm completely resets the agent to the root state, enabling the accumulation of new trajectory and value information. However, this full reset limits the ability to explore deep inside the search tree, which is key benefit of SMC with its depth parameter m. Therefore, we propose a 'revived resampling' strategy to move particles back to their last non-terminal state. This only requires caching an additional reference state for each particle, assuming that the environment correctly flags these as non-terminal. Resampling is then performed to these ref- erence states instead of the current states (see Appendix C).
However, SMC planners do not track visit-counts or per- form backpropagation to accumulate value statistics. Since the importance sampling weights factorize recursively, we can adopt the approach by Moral al. (2010) to perform online estimation to QSMC Q inside SMC using Eq. 3, to then construct policy estimate similarly to Dani- helka et al. (2022). At each SMC step t, we accumulate a current estimate of SMMC for any root-ancestor j as,
3.3. Mitigating Path Degeneracy for Policy Inference
common problem in particle filtering is path degeneracy, where most particles collapse to a single ancestor due to resampling. This is illustrated in the top of Figure 1, the grayed-out trajectories highlight the discarded data due to resampling in typical forward-only SMC methods. This loss of ancestor diversity leads to a worse approximation of the distribution over states under our posterior policy (Svensson et al., 2015). We remark that in RL the consequence of this problem is exacerbated since we predominantly care about the root-ancestors. For this reason, SMC planners (Piché et al., 2019; Macfarlane et al., 2024; Lioutas et al., 2023) that perform policy inference for q" using mixtures of point-
0,
J:0) 0.
210h = Q
+ In
Σ
Eiego
g(00) # 0.
where J. {i e [L,K]
= j} and Q 0.
In essence, this expression accumulates an average log- probability for the remaining particles belonging to ances- tor j. We then use the values Q ~ Q (St, A!") to infer q" by bootstrapping the atoms proportionally to (A()) exp Q - Figure (bottom-right) shows that
5

---

Trust- Trust-Region Twisted Policy Improvement
this eliminates policy deterioration as function of depth, reducing the consequence of path degeneracy.
clock time). We compared our TRT-SMC against the vari- ational SMC method by Macfarlane al. (2024), and the current strongest Monte-Carlo tree search (MCTS) method, Gumbel AlphaZero by Danihelka et al. (2022).
3.4. Search-Based Values for Learning Targets
The approximate message passing from the previous sub- section mitigates the path degeneracy problem by utilizing all SMC generated data for policy inference. In essence, we are constructing a better estimate of the soft-value functions VØ" ~ soft (Lawson et al., 2018; Piché et al., 2019) to then estimate q" as given by Eq. 2. The value functions themselves are then trained using some temporal difference (TD) method in an outer learning loop, e.g., using n-step re- turns (Sutton& & Barto, 2018) given environment interactions outside of the planner (see also Appendix B.3). So far, most prior work constructs these TD-learning targets by value- bootstrapping from one-step predictions given states in the generated datasets (Piché et al., 2019; Lioutas et al., 2023; Macfarlane et al., 2024). For instance, given a transition (S,, Ac. R2, St+1). a 1-step TD target would be computed as Y, R₂ + VØ (S++1). However, this approach neglects most data generated by the planner by only considering the 1 1-step value at the next states S
We performed experiments in the Brax continuous control tasks (Freeman et al., 2021) and Jumanji discrete environ- ments (Bonnet et al.. 2024), using the authors' A2C and SAC results as baselines alongside our PPO implementation (Schulman et al., 2017). Although we compare sample- efficiency to the model-free baselines, this is only for ref- erence since we do not account for the additional observed transitions by the planner in the main results. In other words, we are testing the setting where we assume access to a highly accurate simulator for planning purposes (see Appendix I) for additional results that also count the simulator samples), Performance is reported as the average offline return over 128 episodes. Unless otherwise stated, all experiments were repeated (retrained) across 30 seeds, with 99% two-sided BCa-bootstrap confidence intervals (Efron, 1987). For more details, see Appendix B.
4.1. Main Results
Similarly to recent versions of MuZero (Schrittwieser et al., 2020; Ye et al., 2021; Danihelka et al., 2022), we instead use the values estimated by the search algorithm V++1 over (5 to compute these outer TD-targets. Not only does this exploit the planner data, V is also objectively a better value estimator to use for policy improvement (i.e., within our EM-loop). Namely, the predictions by Vi give the value of previous posterior policy (off-policy) whereas V2 is an estimate of the current posterior policy (on-policy). During our testing, we found at low particle budgets for the SMC planner that it is important to control the vari- ance of the inner value estimation. For this reason, we compute V2 for value-learning using the Retrace(A) returns (Munos et al., 2016) instead of the importance-weighted (soft) Monte-Care estimate used for the policy. This only requires tracking secondary statistic along with Q0).
We show the evaluation curves for comparing sample- efficiency in Figure 2. where the planner- based methods used a budget of N 16 transitions. For simplicity, we kept the depth m of the SMC planner uniform the number of particles K, such that K m vN. Our ablations in the subsection 4.2 also show that keeping and K some- what in tandem is ideal for SMC. On each individual envi- ronment we observe that our TRT-SMC shows steeper and higher evaluation curves than the baseline variational SMC method, which is in line with our expectations.
Additionally, we compare the runtime scaling for additional planning budget in terms of normalized returns on the dis- crete Jumanji environments in Figure 3. We measured this by aggregating the average returns scaled by their environ- ments' known min-max bounds. We see that with more planning budget that the bascline SMC often starts to ap- proach our TRT-SMC, and that the Gumbel MCTS method scales poorly in wallclock time. Most importantly, these re- sults show that our TRT-SMC reliably scales in performance with additional budget, runtime, and training samples.
4. Experiments
We introduce our new method, Peut-RReiion Twisted SMC (TRT-SMC), variational SMC method that is tailored for planning in RL. We claim that TRT-SMC implements a stronger approximate policy improvement over baseline ap- proaches when used in an expectation-maxinizatii frame- work (Abdolmaleki et al., 2018; Chan et al., 2022). The contribution of a stronger policy improvement operator can be isolated to 1) enhanced action-selection during training, and 2) improved learning targets (Hamrick et al., 2021). Therefore, we expect our method to yield higher final test returns and steeper learning curves in terms of sample- efficiency (training samples) and runtime efficiency (wall-
4.2. Ablations
The main results show that our TRT-SMC method can per- form better compared to the baseline model-free, variational SMC, and MCTS methods in terms of sample-efficiens and training runtime. However, we also want to asses: to what extent do each of our separate contributions improve the base method? Therefore, this section quantifies this across the different environments and parameter settings.
6

---

Trust-Region Twisted Policy Improvement
Snake
Rublkscube
Halfcheetah
Ant
1.0
Episode Returns
100
SMC
Gumbel MCTS
luust
0.8
Gumbel MCTS
SMC
8000
80
PPO
A2C
0.6
3000
PPD
6000
4000
0.4
1000
20
0.2
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Training Samples
Figure 2. Per environment evaluation curves for a planning budget of N = K x m = 16. Shaded regions give 99% two-sided BCa- bootstrap intervals over 30 seeds. This plot shows improved sample-efficient of our method when a highly accurate simulator available.
Budget: 2 x 2 =
Budget: x 4= 16
Budget: 8 x 8 64
Budget: 16 x 16 256
Normalized Returns
TKT SNC (sura) SWC
Gumbel NCTS PPC
0.0
100
200
300
400
500
600
200
400
600
800
1000
1200 0
500
1000
1500
500
1000
1500
2000
Wall-Clock Training Time (seconds)
Figure 3. Normalized average curves for the discrete environments over increasing planning budgets to compare performance runtime. Shaded regions give 99% two-sided BCa-bootstrap intervals over 2 times 30 seeds. Runtime was estimated by multiplying the training step with an interquartile mean of the runtime- per-step (see Appendix B.2 for details and Appendix D for sample-officiency).
Proposal Distributions. Firstly, we assess the interplay of the proposal distribution with the planning budget through the aggregated final test performance the Jumanji envi- ronments in Figure 4. This compares our TRT-SMC method when using either a twisted proposal distribution with greediness tolerance of a = 0.1, or a = 0 (which only uses the prior πT). Similarly to the main results, at low particle budgets we observe significantly improved perfor- mance with the trust-region twisted proposal, but this gap decreases for 0 at higher particle budgets. It also shows that performance scales favorably when the particle budget and the planner depth are in tandem with cachother. This performance improvement at low particle budgets matches our aim of increasing
Policy Inference. We compare our method for policy in- ference to that of Eq. 6 by adopting a similar experiment setup we used for the proposal ablations and varying these two choices. We report the results for this experiment in terms of their evaluation curves in the right of Figure 4 to also observe learning stability. We find that the final performance of the Dirac policy at K = 4 and a = 0 shows decreased final performance when the planner depth is increased from effect does not occur for our message-passingg method, al- though the proposal twisting seems to diminish this effect also. Most importantly, our approach for computing q* demonstrates a monotone improvement.
4 to m = 16. As expected, this
Value Targets. To compare the improvement by the search-based value targets, we tested on the Ant environ- ment for experiment variety. We compared three variations of our TRT-SMC for constructing the outer learning targets by linearly interpolating the I-step V, and the SMC-based value estimate VsMG, such that V = a- Vo V V1-(1-0), (1-o))Vsscc where we used a E {0, 1. 1}. We aggregated the final mean performances across different planner depths E {4, 8), particle budgets K E {4, 8), resampling periods E 3), and whether to use our revived resampling or not. To reduce moving parts, we did not use a twisted proposal (c = 0). In total, across 30 repetitions, this gave us 480 experiments for which we report final marginal performance in the top of
Secondly, we evaluated our TRT-SMC method varying the [0, 1] parameter and uniformly scaling up the budget and depth (as done in Figure 3). We aggregated the normal- ized final test results over all tested environments as given by the sensitivity plot in Figure 5. Although we show the aggregated results here, we found that this pattern highly de- pends on the specific environment, we show the individual results in Appendix D. essence, these results show that mixing between the prior na and maximizing the predicted state-action value Q3 improves performance compared to completely relying on either of them.
7

---

Trust-Region Twisted Policy Improvement
Policy Inference Comparison
(Twisted) Proposal Comparison
m=4 K=4
0.8
m 4
16
Normalized Episode
0.5
0.0
m-4 K-16
m- 16 K-16
0.8
0.3
0.4
0.0
0
0.1
a-0
0.1
a- a 0.1
0.0
0.0
Num Training Samples
0.5
1.0
1.5
167
2.0
0.0
Num Training Samples
0.5
1.0
1.5
1c7
2.0
K-4
K=8
K=
Backups
Backups
Figure 4. Final expected performance (left) and evaluation eurves training (right), for different proposal trust-region levels and policy inference methods. The left barplot only uses our message-passing method (backups) for estimating (", whereas the right plot compares both the Dirac mixture and our method. Both plots show that constrained proposals proposal a = 0 at low particle budgets. The right plot also shows that our backup method for q does not degenerate with deep planning.
improve performance over the prior
Parameter Sensitivity: Discrete + Continuous Envs
Table 1. Confidence intervals for the final expected episodc returns on Brax Ant for different value-estimation methods (top) and Jumanji Snake for the resampling strategies (bottom).
Normalized Average Returns
1.0
Ablation Value
la/2 Q1-a/2
0.8
Value Targets on Ant
0.6
Va
6911.8
6669.7
7146.7
$ Vo + VSMc
7214.7
6973.9
7455.6
0.4
VSMC
7457.1
7200.3
7715.3
16
Resampling on Snake
0.2
N 64
Baseline
44.7
43.7
45.7
N 256
Revived
45.7
44.5 47.7
0.0
0.001
0.01
0.1
0.4
Trust-Region Factor a
ft'
resampling does not significantly from the baseline.
5. Related Work
Figure 5. Sensitivity plot for the adaptive trust-region parameter d over increasing planning budgets N K m (where K m). The y-axis indicates the normalized final test performance, aggregated across all tested environments. The pattern shows that small planning budgets, proposal distribution which accounts for the predicted Q performs marginally better.
The connection of reinforcement learning (RL) to the statis- tical estimation of probabilistic graphical model (Levine, 2018) has in recent years proven useful in borrowing tools from Bayesian estimation for optimal policy inference. Al- though we focus sequential Monte-Carlo (SMC) methods (Hoffman et al., 2007; Piché et al.. 2019), this connection has also been used to exploit stochastic control methods like TD-MPC (Hansen et al., 2024; Theodorou et al., 2010). Similarly, prior work has partially explored some of the mod- ifications that we make SMC-planners, either isolation in different contexts. This paper therefore reinforces the connection between probabilistic inference and RL, but also links it back to recent approaches in Monte-Carke tree search (MCTS) (Browne et al., 2012: Silver et al., 2018; Schrittwieser et al., 2020; Wang et al., 2024).
Table 1. Although the intervals overlap slightly, there is a trend that favors using SMC data for the learning targets.
Revived Resampling. We evaluated the effect of using the revived particle resampling in similar, experiment setup to the value-targets ablations. We tested on Jumanji Snake due to its sparse rewards and high likelihood of encountering ter- minal states (see Appendix B.I) Interestingly, the marginal test results the bottom of Table show that the revived
As discussed in Section 2.1, our method builds on the varia-
8

---

Trust-Region Twisted Policy Improvement
tional SMC approach by Macfarlane al. (2024). Similarly, they also utilize trust-region methods, but on the neural network parameters during optimization and on the target distribution for SMC. Our setup is more comparable to re- cent MCTS methods (Danihelka et al., 2022; Wang et al., 2024) since we only impose trust-regions on the propos- als and nothing else. other words, we focus specifically on the policy improvement part of the algorithm. Then, Lioutas al. (2023) also explore type of 'twisted' pro- posals, they sample auxiliary action-particles and weight these with heuristic factors exp Q3 before computing tran- sitions (Stuhlmülller et al., 2015). However, their approach has two issues: they mix the normalization of the auxiliary actions across particles and they do not impose sufficient regularization to trade-off the prior policy and the values.
Impact Statement
This paper advances planning algorithms for use in rein- forcement learning. Our improvements specifically enable improved compute scaling, this has the potential to make this field of research more accessible to those with fewer compute resources or improve existing methods at reduced computational cost. This can have diverse societal con- sequences, none which we feel must be specifically high- lighted here.
References
Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. Maximum a Posteriori Pol- icy Optimisation. In The Sixth International Conference on Learning Representations, 2018
Finally, our contributions are strongly tied to mitigating path degeneracy in SMC planners, which is common theme in particle filtering (Chopin & Papaspiliopoulos, 2020). For instance, our estimation of the policy (and values) is similar to the online estimation of any sepereba function de- scribed by Moral et al. (2010), which was also motivated by path degeneracy. Although we did not consider it here, there are many promising directions for future work in this area, like using anchor particles (Svensson et al., 2015), adaptive resampling (Naesseth et al., 2019), or Rao- Mao-Blackwellle Blackwell (Casella & Robert, 1996; Danihelka et al., 2022).
Antonoglou, I., Schrittwieser, J., Ozair, S., Hubert, T. K., and Silver, D. Planning in Stochastic Environments with a Learned Model. In The Tenth International Conference on Learning Representations, 2022.
Asmussen, S. and Glynn, P. W. Stochastic Simulation: Algo- rithms and Analysis, volume 57 of Stochastic Modelling and Applied Probability. Springer, New York, NY, 2007. doi: 10.1007/978-0-388--603333
Åström, K. J. Introduction to Stochastic Control Theory, volume 70 of Dover Books on Electrical Engineering. Dover Publications, New York, NY, 2006.
6. Conclusion
This paper tailors particle filter planner for its use within deep reinforcement learning. Specifically, we address de- fault design choices within variational sequential Monte- Carlo that become problematic when applying these meth- ods to perform policy inference. Our contributions take inspiration from recent Monte-Carlo tree search methods to mitigate the path degeneracy problem, make better use of the data generated by the planner, and to improve planning budget utilization. Experiments show that our Trust- Prust-Region Twisted sequential Monte- Carllo (TRT-SMC) scales favor- ably to varying planning budgets in terms of runtime and sample-efficiency over the baseline policy improvement methods. We hope that our approach inspires others to lever- age more of the tools from both the planning and Bayesian inference literature, to further enhance the sample-efficiency and runtime properties of reinforcement learning algorithms.
Bertsekas, D. Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control. Athena Scientific optimization and computation series. Athena Scientific, Nashua, NH, 2022.
Bishop, C. M. Pattern Recognition and Machine Learning
(Information Science and Statisties). Springer, New York, NY, 1st edition, 2007.
Bonnet, C., Luo, D., Byrne, J. Surana, S., Abramowitz, S., Duckworth, P., Coyette, V., Midgley, L. I., Tegegn, E., Kalloniatis, T., Mahjoub, O., Macfarlane, M., Smit, A. P., Grinsztajn, N., Boige, R., Waters, C. N., Mimouni, M. A. A., Sob, U. A. M., de Kock, J., Singh, S., Furelos- Blanco, D., Le, V., Pretorius, A., and Laterre, A. Jumanji: a Diverse Suite of Scalable Reinforcement Learning En- vironments in JAX. In The Twelfth International Confer- ence on Learning Representations, 2024.
Acknowledgements
JdV and MS are supported by the Al4b.io program, a col- laboration between TU Delft and dsm- can-firmenich, which is fully funded by fimmfirmente and the RVO (Rijksdienst voor Ondernemend Nederland).
Browne, C. B., Powley, E.,, Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohifshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. A Survey of Monte Carlo Tree Search methods. IEEE Transactions on Computa- tional Intelligence and Al in Games, 4(1):1-43, 2012. doi: 10.1109/TCIAIG.001222

---

Trust-Region Twisted Policy Improvement
Casella, G. and Robert, C. P. Rao-Blackwellisationn of Sam- pling Schemes. Biometrika, 83(1):81-944 1996.
the 37th International Conference on Machine Learning, volume 119, pp. 3769-3778. PMLR, 2020.
Chan, A., Silva, H., Lim, S., Kozuno, T., Mahmood, A. R., and White, M. Greedification Operators for Policy Opti- mization: Investigating Forward and Reverse KL Diver- gences. Journal of Machine Learning Research, 23(253);
Gu, S. S., Ghahramani, Z., and Turner, R. E. Neural Adap- tive Sequential Monte Carlo. In Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015.
1-79, 2022.
Guez, A., Mirza, M., Gregor, K., Kabra, R., Racaniere, S., Weber, T., Raposo, D., Santoro, A., Orseau, L.,, Eccles, T., Wayne, G., Silver, D., and Lillicrap. An Investigation of Model-Free Planning. In Proceedings of the 36th In- ternational Conference Machine Learning, volume 97, pp. 2464-2473, PMLR, 2019.
Chopin, N. and Papaspiliopoulos, O. An Introduction to Sequential Monte Carlo. Springer Series Statistics. Springer International Publishing, Cham, Switzerland, 2020. doi: 00.1007/978-33000-4778888
Danihelka, I., Guez, A., Schrittieser, J., and Silver, D. Pol- icy improvement by planning with Gumbel. The Tenth International Conference on Learning Representations, 2022.
Haarnoja. T., Zhou, A., Abbeel, P., and Levine, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Rein- forcement Learning with Stochastic Actor. In Proceed- ings of the 35th International Conference on Machine Learning, volume 80, pp. 1861-1870. PMLR, 2018.
de Vries, J. A., Voskuil, K., Moerland, T. M., and Plaat, A. Visualizing MuZero Models. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021.
Hamrick, J. B., Friesen, A. L., Behbahani, F., Guez, A., Viola, F., Witherspoon, S., Anthony, T., Buesing, I.. H., Veličković, P., and Weber, T. On the role of planning in model-based deep reinforcement learning. In The Ninth International Conference on Learning Representations, 2021.
Del Moral, Feynman-Kac Formulae. Probability and its Applications. Springer, New York, NY, 2004. doi:
Delft AI Cluster (DAIC). The Delft AI Cluster (DAIC), RRID:SCR 2024.
Delft High Performance Computing Centre (DHPC). Delft- Blue Supercomputer (Phase 2), 2024.
Hansen, N., Su, H., and Wang, X. TD-MPC2: Scalable, Robust World Models for Continuous Control. In The Twelfth International Conference on Learning Represen- tations, 2024.
Efron, Better Bootstrap Confidence Intervals. Journal of the American Statistical Association, 12(397):171-185, 1987. doi: 10.2307/2289144
Hansen, N. A., Su, H., and Wang. X. Temporal Difference Learning for Model Predictive Control. In Proceedings of the 39th International Conference on Machine Learning, pp. 8387-8406. PMLR, 2022.
Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera- Paredes, B., Barekatain, M., Novikov, A., R. Ruiz, J., Schrittwieser, J., Swirszez, G., Silver, D.,, Hassabis, D., and Kohli, P. Discovering faster matrix multiplication al- gorithms with reinforcement learning. Nature, 610(7930); 47-53, 2022. doi: 10. 10.1038/441556-02--055
He, J., Moerland, T. M., de Vries, J. A., and Oliehoek, F. A. What model does MuZero learn? In ECAI 2024 27th European Conference on Artificial Intelligence, volume 392 of Frontiers in Artificial Intelligence and Applications, pp. 1599-1606. IOS Press, 2024, doi: 10. 3233/FAIA240666.
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mor- datch, I., and Bachem, O. Brax A Differentiable Physics Engine for Large Scale Rigid Body Simulation. In Thirty- fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Truck (Round 1), 2021.
Hoffman, M., Doucet, A., Freitas, N., and Jasra, A. Bayesian Policy Learning with Trans-Dimensional MCMC. In Advances in Neural Information Process- ing Systems, volume 20, Curran Associates, Inc., 2007.
Geist, M., Scherrer, B., and Pietquin, O. A Theory of Reg- ularized Markov Decision Processes. Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2160-2169, PMLR, 2019.
Hubert, T., Schrittwieser, J., Antonoglou, I., Barekatain, M., Schmitt, S., and Silver, D. Learning and Planning in Complex Action Spaces. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 4476 4486. PMLR,, 2021.
Grill, J.-B., Altché, F., Tang, Y., Hubert, T., Valko, M., Antonoglou, I., and Munos, R. Monte-Carlo Tree Search as Regularized Policy Optimization. In Proceedings of
10

---

Trust-Region Twisted Policy Improvement
Lawson, D., Tucker, G., Naesseth, C. A., Maddison, C., Adams, R. P., and Teh, Y. W. Twisted Variational Sequen- tial Monte Carlo. In Third workshop on Bayesian Deep Learning (NeurIPS), 2018
Oren, Y., Vadocz, V., Spaan, M. T. J., and Boehmer, W. Epistemic Monte Carlo Tree Search. In The Thirteenth International Conference Learning Representations, 2025.
Levine, S.
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review. arXiv: 1eii::80550000 805.00909, 2018.
Patterson, A., Neumann, S., White, M., and White, A. Em- pirical Design in Reinforcement Learning. Journal of Machine Learning Research, 25(318):1-63 2024.
Lioutas, V., Lavington, J. W., Sefas, J., Niedoba, M., Liu, Y., Zwartsenberg, B., Dabiri, S., Wood, F,, and Scibior, A. Critic Sequential Monte Carlo. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Piché, A., Thomas, V., Ibrahim, C., Bengio, Y., and Pal, C. Probabilistic Planning with Sequential Monte Carlo methods. In The Seventh International Conference on Learning Representations, 2019.
Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. In The Seventh International Conference on Learning Representations, 2019.
Pitt, K. and Shephard, N. Filtering via Simulation: Auxiliary Particle Filters. Journal of the American Statistical Association, 94(446)5500599, 1999. doi: 10. 10.1080/01621 110000621459...0 195.9999999.11444111
Macfarlane, M., Toledo, E.,, Byrne, D. J., Duckworth, P., and Laterre, A. SPO: Sequential Monte Carlo Policy Optimisation. In The eiirth- Annual Conference on Neural Information Processing Systems, volume 37. Curran Associates, Inc., 2024.
Rosin, C. D. Multi-armas bandits with episode context. Annals of Mathematics and Artificial Intelligence (3):
203-230, 2011. doi:
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., and Silver, D. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839) (333))60-660000 2020. doi: 10.1038/ $41586-020-03051
Mankowitz, D.J., Michi, A., Zhernov, A., Gelmi, M., Selvi, M., Paduraru, C., Leurent, E., Iqbal, S., Lespiau, J.- B., Ahern, A., Köppe, T., Millikin, K., Gaffney, S., El- ster, S., Broshcar, J., Gamble, C., Milan, K., Tung, R., Hwang, M., Cemgil, T., Barekatain, M., Li, Y., Mand- hane, A., Hubert, T., Schrittwieser, J., Hassabis, D., Kohli, P., Riedmiller, M., Vinyals, O., and Silver, D. Faster sorting algorithms discovered using deep reinforce- ment learning. Nature, 618(7964):257-200) 2023, doi: 10.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms. arXiv:17011 707.06347, 2017.
Seijen, H. and Sutton, R. True Online TD(lambda). In Proceedings of the 31st International Conference on Ma- chine Learning, volume 32, pp. 692-700, Bejing, China, 2014. PMLR.
Moral, P. D., Doucet, A., and Singh, S. Forward Smoothing using Sequential Monte Carlo. arXiv: 1012::001,,5390, 2010.
Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and Efficient Off-Policy Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel, T., Lillicrap, T., Simonyan, K., and Hassabis, general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362 (6419):1140-1144 2018. doi: 10. 10.1126/science.aarr//
Naesseth, C., Linderman, S., Ranganath, R., and Blei, D. Variational Sequential Monte Carlo. In Proceedings of the Twenty-Firs International Conference on Artificial In- telligence and Statistics, volume 84, pp. 968-977. PMLR, 2018.
Stuhlmülller, A., Hawkins, X. D., Siddharth, N., and Goodman, D. Coarse-to Sequential Monte Carlo for Probabilistic Programs. arXiv: 1509 100.0062,,,, 2015.
Naesseth, C. Lindsten, F., and Schön, T. B. Elements of Sequential Monte Carlo. Foundations and Trends@ in Machine Learning, 12(3):307-392, 2019. doi: 10.1561/ 2200000074.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, 2nd edition, 2018.
Neal, R. M. and Hinton, G. E. A View of the EM Algorithm that Justifies Incremental, Sparse, and other Variants, pp- 365..66. Springer Netherlands, Dordrecht, 1998. doi: 10. 100799777 94-0114-1111 5014-99 12.
Svensson, A., Schön, T. B., and Kok, M. Nonlinear State Space Smoothing Using the Conditional Particle Filter. PAAC-PaperSOnLine, 48(28):975 2015. 101 1016/
doi: 10.

---

Trust-Region Twisted Policy Improvement
Theodorou, E., Buchli, J., and Schaal, S. A Generalized Path Integral Control Approach to Reinforcement Learning- Journal of Machine Learning Research, 113104):3137- 2010.
Toussaint, M. Robot trajectory optimization using approx- imate inference. In Proceedings of the 26th Annual In- ternational Conference Machine Learning, pp. 1049- 1056. Association for Computing Machinery, 2009. doi: 10.1 05555/1533374115553335
Wang, S., Liu, S., Ye, W., You, J., and Gao, Y. EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data. arXiv:2403.00564 2024.
Williams, G., Aldrich, A., and Theodorou, E. Model Pre- dictive Path Integral Control using Covariance Variable Importance Sampling. arXiv:1509.01 149, 2015.
Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Mas- tering Atari Games with Limited Data. In Advances in Neural Information Processing Systems, volume 34, Cur- Associates, Inc., 2021.
Ziebart, B. D. Modeling Purposeful Adaptive Behavior with the Principle of Maximun Causal Entropy. PhD thesis, Carnegie Mellon University, 7 2010.
12

---

Trust-Region Twisted Policy Improvement
A. Derivations
first restate the factorization the marginal in Eq.
from the main text,
Pr Pr (H.T) = II π (A A--1),
1=1
with (S Su) _ p(S)) being the initial state distribution, p(St+a|St,, Ag) the transition model, and π (Ac(S)) the policy. We denote the set of admissible policies as _ {ग| S P(A)). We drop subscripts for Hir if the indexing clear from the text.
A.1. Lower-bound
For completeness, we show below that the factorization of p- P~ (= = 1), by Definition 2.1, recovers the lower-bound term for 9º shown in Theorem 2.2. This done through the well-known decomposition of the log-likelihood the marginal distribution for the outcome variable O.rr {0, 1)". See Section 2.1 for a description the meaning of this variable, again we will abbreviate O = simply as O.
Lemma A.1 (Decomposition likeiiiiihodd, c.f., Ch 9.4 of Bishop (2007)).
In (O) (H)
Σ R. ([(g((())
KL (I)))))
(8)
Evidence
Evidence Gap
Proof. Assume an importance sampling distribution E II for ı E II such that it has sufficient support over H and Pa (H) > 0 + (H) > 0 almost everywhere.
In (O) = In Pr (O, H) Pr (H
(O H) (H)
Px (H) (H(H) ]
En,(m)) In
(O,
E (,)( In Pq
Ep,(H) In
(((
Pr (H1O)
Inp(O]H) + in (H)
Em) In PH(H)
Ep,(HT)
Py(H)
P=(H(O)
Ep,(H)) [mp(( KLln, (H) + (Upp(4))llP
= (H)
Σ R. KT(g(a)S))
where in the last step the transition terms for pa. (H) and py (H) cancel out. See the work by Levine (2018) for comparison.
The result from Lemma A.1 shows that the log- likelihood is decomposed into evidence lownr-bound and evidence gap. This inequality becomes tight when is simply set to the posterior policy q" (H) = P- (H)HO) Thus, motivating the maximization objective for the lower-bound as given in Theorem 2.2, or equivalently, by minimizing the evidence gap as considered by Levine (2018).
A.2. Regularized Policy Improvement
The result below gives brief sketch that the expectation-maxiiizztioo loop (Neal & Hinton, 1998) generates a sequence of regularized Markov decision processes (MDPs) that eventually converges to a locally optimal policy. This result is nice consequence of the control-as-ifferent framework. more general discussion outside the expectation moiooonmamiiiiiiiiooooo framework can be found in the work by Geist et (2019).
13

---

Trust-Region Twisted Policy Improvement
Lemma A.2 (Regularized Policy Improvement). The solution qf the problem,
T
max
Σ R gell
E
guarantees a policy improvement in the unregularized MDP, E., (IT) |E, Rel 2 (p) 12, Rel.
Proof. Lemma A.I shows that the solution q² is equivalent to the posterior policy distribution pa (1.r)) This implies
that for each state s E S, we have, q" (als) x [[ass) (s) exp Q (s, a). The exponential over Qo (5, a) in the posterior policy q² interpolates æ to the greedy policy by shifting probability density to actions with larger expected cumulative reward. Thus, q² provides policy improvement over
A.3. Proof of Theorem 2.2
Proof of Theorem 2.2. Lemma A. shows how the posterior policy coincides with the optimal policy q" in a regularized Markov decision process (MDP). Then Lemma A.2 describes that this gives policy improvement in the unregularized MDP. Iterating this process (e.g., expectation-maximzztiio loop) yields consecutive improvements the prior T(n) (n- and guarantees a locally optimal n² in the unregularized MDP as n + oo, which also implies KL(g()) (m-))) -
A.4. Importance sampling weights
For completeness, we give a detailed derivation for the result presented in Corollary 2.3. This derivation differs from the one given by Piché et al. (2019) in Appendix A.4. Our derivation corrects for the fact that we don't 't need to compute an expectation over the transition function for exp (S.)) in the denominator. However, this only a practical difference (i.e., how the algorithm is implemented) to justify our calculation.
(S.)
Corollary A.3 (Restated Corollary 2.3). Assuming access to the transition model p(S Ad), we obtain the importance sampling weights for Pa 1/Tp///
TT (Ar(S,) exp(Rt) W/ = Wt-1 g(A.)S))
Elexp 2o (S++1)]
exp V soft (S:)
Proof. For any statistic f(-). we have.
(H)() [wo f(H11)]
Pr Pm f( ((H.)) (H
Pa (Sr, Arl cr, ( [(H1a))
= (Hie(H
PS(Se, A, Hzt) (Hzz))
Pa (S,, 1, A, Ou:T)
-((He))
(Hp,(H)))))
W1
Pg (Se, A
where the last step follows from the Markov property. Then, we get,
(S,S, A, Ar-1, O..T)
(Ar Our (1 P= (Ar(SS, O1:)) q(A S.)
A,5,,,,,
7(A((8) (O Ar) T(A((S)) exp Qo (St, As)
((Aa(St) Pa (O St) (()
exp V soft (St)
((A((S.)
Elexp ViLjA (S++1)
((A.)S.) exp(
cxp V (S,)
14

---

Trust-Region Twisted Policy Improvement
B. Experiment Details
Our code can be found at htttpp///gghuub...o///orr///////
B.1. Environments
We used the Jumanji 1.0.1 implementations the Snake-vl and Ruiikssulk--pprtty--cammhee--- environments (Bonnet et al., 2024), code is available at https:: https::/ com/ instaceepaaijummanee For the Brax 0.10.5 implementation we used the Ant and Halfcheetah environments using the 'spring' backend, code is available https: /g/thub..tt///// com/ google/brz /brax.
Snake environment details:
The observation a 12x12 image with channels that indicate the position of the fruit and positional features of the snake, also gives the integer number steps taken which we encode as a bit-vector the neural network.
The action space is a choice over integers that indicate moving the snake: up, down, left, or right.
The reward is zero everywhere, except for +1 when the fruit is picked up.
The agent terminates after 4000 steps when colliding with itself.
Rubikscube environment details:
The observation is 3D integer tensor of shape x 3 x 3 with values in 2,, indicating the color. It also gives the integer number of steps taken which encode bit-vector for the neural network.
The action space is 3 dimensional integer array to choose the face to turn, depth of the turn, and direction of the turn. For a 3x3 cube this action-space induced 18 combinations.
The reward is zero everywhere, except for +1 when the puzzle is solved.
The agent terminates after 20 steps or when solving the puzzle.
We used the partly-scrambled version of this environment, which means that the solution is at most 7 actions removed from any of the starting states.
Brax environment details:
The observations continuous values with 27 dimensions for Ant and 18 dimensions for Halfcheetah.
The action space is a bounded continuous vector between [|1, with D 8 for Ant and D 6 for Halfcheetah.
The reward is dense, but involved formula that penalizes energy expenditure (norm of the action) and rewards the agent for moving in space (in terms of spatial coordinates).
The agent terminates after 4000 steps or when ending up in a unhealthy joint-configuration
B.2. Hardware Requirements
All experiments were run on a GPU cluster with mix of NVIDIA GeForce RTX 2080 TI IIGB, Tesla V100-SXM2 32GB, NVIDIA A40 48GB, and A100 80GB GPU cards (Delft AI Cluster (DAIC), 2024; Delft High Performance Computing Centre DHPC). Each run (random seed/ repetition) required only few CPU cores (2 logical cores) with a low memory budget (c.g., 4GB). For our most expensive singular experiments we found that we needed about 6GB of VRAM at most, and that the replay buffer size is the most important parameter in this regard. Roughly speaking, we found that the SMC based agents all completed both training and evaluation under 30 minutes on Snake with a budget of K particles and a depth of 8, the Gumbel MCTS required 3 hours for a budget of N 64.
15

---

Trust- Prust-Region Twisted Policy Improvement
Training Time Estimation. To estimate the training runtime in seconds (in Figure 3), used estimator of the the runtime-per-step and multiplied this by the current training iteration obtain cumulative estimate. For each training configuration, we measured the runtime-per-step and computed an interquartile mean over 1) the random seeds for relative wallcock time and 2) the training iterations themselves. This estimator should more robustly deal with the variations in hardware, the compute clusters' background load, and XIA dependent compilation. Of course, estimating runtime strongly limited to the hardware and software implementation, and our results should only hint towards a trend of improved scaling to parallel compute for the planner algorithms.
B.3. Hyperparameters, Model Training, and Software Versioning
The hyperparameters for our experiments are summarized in the following tables:
Table 3:
Shared parameters across experiments.
Table 4: PPO-specific parameters.
Table 5: -pTS-specific parameters.
Table 6: Shared SMC parameters.
Table 7: Parameters for our extended agent.
We underline all default values in bold, all other parameter values indicated in the sets were run in an exhaustive grid for the ablations. The ablation results then report marginal performance over configurations and over seeds (repetitions). These experimental design decisions closely follow the suggestions laid out the work by Patterson et al. (2024).
Despite conflating all model parameters into one joint set we used separate neural network parameters for the policy, state, and state-action value models. Given the current dataset (replay buffer) within the training loop P(n), the loss simple empirical cross-entropy of collective terms,
((0) = E. A..
Ce (V. Lee (V, - VS,())) + E (V, A.)) - cx Balg In Te( - Cent Cent 2
(a(a(())))
(9)
where d. and Vú are estimated targets for the policy and value respectively (see main text), and H[TTo] E,, In no an entropy penalty for the policy. We approximated L. with stochastic gradient descent (see the hyperparameter tables), where always used the AdamW optimizer (Loshchilov & Hutter, 2019) with an l2 penalty of 10-6 and learning rate of 3 10-3 Gradients were clipped using two methods, in order: max absolute value of 10 and global norm limit of 10.
The replay buffer was implemented as a uniform circular buffer with its size calculated as:
max-age of data number of parallel environments number of unroll steps.
The max-age of data tuned to fit into reasonable GPU memory.
For the A2C baseline, hyperparameters are detailed in Bonnet et al. (2024), and for the SAC baseline, refer to Freeman et al. (2021). Additionally, Table 2 provides version information for key software packages, this also directs towards default hyperparameters of baselines not listed here. We implemented everything based on Jax 0.4.30 in Python 3.12.4.
B.4. Neural Network Architectures
Neural network designs were largely adapted from the A2C reference (Bonnet et al., 2024) and Macfarlane et al. (2024). Minor modifications were made to handle the heterogeneity environment action spaces. Notably, we standardized network architectures across environments wherever feasible, adjusting input embedding and output construction as needed. The specific configurations are listed below.
Brax Environments
-2-layer ML. with 256 nodes per layer.
16

---

Trust-Region Twisted Policy Improvement
Table 2. Software module versioning that we used for experiments (also includes default parameter settings).
Package
Version
brax
0.10.5
optax
0.2.3
flashbax
0.1.2
rlax
0.1.6
metx
0.0.5
flax
0.8.4
jumanji
1.0.1
Leaky-ReLU activations followed by LayerNorm.
Outputs parameterized a diagonal multivariate Gaussian squashed via Tanh, as in Haarnoja et al. (2018).
Jumanji RubiksCube Environment
2-layer MLP with 256 nodes per layer (same Brax).
We used flat representation for the 3-dimensional categorical action space (logits over all item-combinations) In contrast: the A2C baseline used structured representation with three separate categorical outputs (logits per item).
Jumanji Snake Environment
2-layer MLP with 128 nodes per layer and Leaky-Rell activations, followed by LayerNorm for the main module.
Based on Bonnet et al. (2024), the input image is embedded using a single 3x3 convolutional layer with:
3 channels.
Leaky-ReLL activation.
No LayerNorm.
The resulting embedding was flattened before being passed to the main MLP module.
17

---

Trust-Region Twisted Policy Improvement
Table Shared experiment hyperparamicters.
Name
Symbol
Value Jumanji
Value Brax
SGD Minibatch size
256
256
SGD update steps
100
64
Unroll length (nr. steps in environment)
64
64
Batch-Size (nr. parallel environments)
128
64
(outer-loop) TD-Lambda
A
0.95
0.9
(outer-loop) Discount
?
0.997
0.99
Value Loss Scale
Cp
0.5
Policy Loss Scale
Cx
1.0
1.0
Entropy Loss Scale
Cent
0.1
0.0003
Table 4. Proximal Policy Optimization hyperparameters. We did not advantage- noriiaizannnmmmliiiaii computed the policy entropy exactly.
Name
Symbol
Value Jumanji
Value Brax
Policy-Ree clipping
0.3
0.3
Value Loss Scale
Cv
1.0
0.5
Policy Loss Scale
Cx
1.0
1.0
Entropy Loss Scale
Centt
0.1
0.0003
Table 5. Gumbel Monte-Carlo tree search experiment hyperparameters (Danihelka al., 2022).
Name
Symbol
Value Jumanji
Value Brax
Replay Buffer max-age
64
64
Nr. bootstrap atoms и
B
30
30
Search budget
N
{16, 64)
{16, 64)
Max depth
16
16
Max breadth
16
16
Table 6. Shared Sequential Monte-Carlo hyperparameters (ours and Macfarlane al., 2024). Bold values indicate those used in the main results, with the remaining values in the set being explored the ablations.
Name
Symbol
Value Jumanji
Value Brax
Replay Buffer max-age
64
64
Planner Depth
{4, 8, 16)
{4, 8)
Number of particles
K
{4,8, 16}
{4, 8)
Resampling period
(1,3)
(1,3)
Target temperature (env. reward scale)
T
{1.0, 0.13
(1.0,0.1)
Nr. bootstrap atoms æ
B
30
30
Table Trust-Region Twisted Sequential Monte-Carke hyperparameters (i.e., ours only). The underlined values recover SMC.
Name
Symbol
Value Jumanji
Value Brax
(inner-loop) Retrace(X)
ÀSMC
0.95
0.9
(inner-loop) Discount
TSMC
0.997
0.99
(outer-loop) Value mixing V
0.5 (0.0)
{0.0, 0.5, 1.0) (Dirac, Message- Passing [False, True) {0.0. 0.1, 0.3]
Estimation q"
Dirac, Plessage-Passingg [False, True}
Revived resampling
Proposal (adaptive) Trust-Region
Ea
{0.0. 0.1, 0.3)
18

---

Trust-Region Twisted Policy Improvement
B.5. Details Constrained Proposals
We solve the constrained program in Eq. 7 in the SMC planner in Algorithm I for each individual state-particle S To deal with general action-spaces (e.g., continuous), we sample B atoms from the prior policy Ty for uniform bootstrapping. As stated in Theorem 2.2 and accompanying text, the Lagrangian of this program can used to define a Boltzmann policy,
L(q, 3.., = Q7 + (ca - 3- (L(gl + (1 n)),
(10)
where taking the partial derivatives and setting them to zero gives q" x Te exp 3Q8. which is analytically normalizable (see Grill et al. (2020) for comparison). Given this distribution q², we found that the following minimization problem was the most numerically stable in finding the optimal temperature parameter 3-11
min || ta - Eay q In no (a(a
(11)
which we solve with bisection search. combination with bootstrapping from wo, the above In (a) is essentially an entropy constraint on q², As stated in Subsection 3.1, we ea adaptively based on some greediness tolerance [0, 1].
B.6. Details on Figure!
We adapted the top-figure with the colored and grayed out trajectories from Figure of Svensson al. (2015) to show the discarded data in naive forward-only sequential Monte-Carlo (SMC) planner. We generated the two bottom figures using evaluated by: running SMC timestep 0 dummy
SMC planner implementation. The KL-divergence environment, extracting the sampled policy as by Eq. 6 or from Section 3.3, projecting the sampled logits back to their original action-space, and calculating the Kl--divergence of the optimal stochastic policy to the canonical (non-bootstrapped) logits from SMC. Most importantly this environment had discrete actions and zero reward everywhere, making the dynamics function like an absorbing state. For this reason, the optimal (stochastic) policy is uniformly random with a value V of zero everywhere. The bottom-left of Figure 1, thus, visualizes the consequence of recursive bootstrapping. which degenerates the policy in terms KL-divergence from the optimal policy. Our method does incur this in the bottom-right aside from the effect caused by the particle budget.
C. Pseudocode
We give a simplified overview of our method in Algorithm 2, which an extended version of the one from the main paper (Algorithm 1). The pseudocode documents our specific contributions in comparison to the base SMC planner.
Note that the implementation for the online tracking of ancestor values, following Moral et al. (2010) is very similar the eligibility trace known in reinforcement learning (Sutton & Barto, 2018; Seijen & Sutton, 2014). Fundamentally, this be considered initializing the eligibilities of the ancestor particles to one, and continuously decaying these eligibilities while accumulating value updates (i.e., without updating the ancestor-eliiibility)
19

---

Trust-Region Twisted Policy Improvement
Algorithm 2 Bootstrapped Particle Filter for RL (Our TRT-SMC Pseudocode based on Algorithm 1).
Require: K (number of particles), m (depth), r (resampling period), (proposal greediness)
1: Initialize:
Ancestor identifier {J1" = 1
States {8}
p(Si)]==
Reference States (5(() t- 5 LK
// Revived resampling: track last non-terminal states
Ji=1
Weights (wo
1 |==1
Ancestor Log- Prg-brrbbasl (0) = 0)
Policy Inference
2: for t= = 1 to m do
// TRT-SMC: Create a set of Trust- Brust-Region twisted proposal distributions
3: (a()) where ( solves Eq. for given a, S( Ji=1
// Default SMC: Update particles
4: ( qi (1)
(A:IS{))
5: (1 ~ p(S.+118 15 a
6: (w) = - (s
1540 m!
Eexp (S 1K
exp (s())) J==1
// Revived resampling: Track the last non-terminal states
59 5
1+1 is terminal,
7: (5
SQ 4+1
Otherwise,
// Policy inference & Value Estimation: online accumulation of ancestor statistics
8: (Jh)- Eie [1,
0,
J: 0,
6 6 +
9:
) + // Default SMC: Bootstrap (periodically) through resampling
In
Σ (()
J
|5
10:
if't mod r == 0 then
11:
Normalized probability vector: [
(J(()) (T
12:
// Revived resampling: resample particles by their reference states, then reset the references.
[(S[)), A K - ((5(4) (1)))) K
13:
(5(1) <- ((0) K
(w(") 1
15:
16: end if
17:
end for
// Return the estimated values for performing policy inference
return ( K
20

---

Trust- Trust-Region Twisted Policy Improvement
Algorithm 3 Outer EM-loop for Approximate Policy Iteration
Require: Initial iterate (1) for neural networks, replay buffer (1)-
1: for 1 to N do
S1 ~p((S
3: to T do
// IIIner--loo:: Model-Predictive Control (Algorithm 1)
~01) K SMC(S;;; (T) Vi(m)) {Jettm 1 Whi+mJ
4:
5:
Estimate o a policy using SMC-output (e.g., Eq 6).
6: Estimate V/ (inner) value using SMC-output (e.g.. Vo (S:)).
Sample action from search policy A; 4f
// Environment step and data collection
8:
Se+1 (- Ac), R2 R(S,, As) Append (St, Ar, Re, ft V/) to buffer D ((n)
9:
10:
end for
// Outer loop; learning through SGD
Compute (outer) value estimators V using rewards R2 and (inner) values V from Pin) (e.g., truncated TD(A))
12: Update (m+1) with SGD on (() P(is))) (Equation 9)
13:
(Optionally: Circularly wrap D (m++)) from (()))
14: end for
15: return (x)
21

---

Trust- Prust-Region Twisted Policy Improvement
D. Supplementary Results
Budget: 4
Budget: 4 x 4
Budget: 8 x 64
Budget: 16 16 256
Normalized Returns
TRI-SNC (oun))
0.8-
SMC
Gumbel MCTS
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0
1.3
2.0
0.0
0.5
1.0
15
2,0
0.0
0.5
1.0
1.5
2,0
167
167
167
167
Training Samples
Figure 6. Normalized expected evaluation curves for the discrete environments over increasing planning budgets. These budgets were calculated N K m, where K m, keep the number of transitions uniform between the MCTS and SMC methods. Shaded regions give BCa 0.01 intervals over 2 times 30 seeds. See also Figure in the main text for similar comparison to runtime scaling.
(Twisted) Proposal Comparison
0.8
m
16
Normalized Episode Returns
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Te
B 1 = 0.1
B = 0.1
7e 0.1
K= 4
K=8
K= 16
Figure 7. Final expected performance for the prior wo and the regularized proposal with static temperature of g- = 0.1. This plot identical to the left barplot in Figure 4 and shows that taking values into account for the proposal doesn' 't directly translate improved performance (i.e.. the approach taken by Lioutas al., 2023). We found it essential for performance that the temperature 3 must be set adequately, which we achieved with the adaptive trust-region method.
22

---

Trust- Prust-Region Twisted Policy Improvement
Snake
Rubikscube
Halfcheetah
Ant
120
1.0
(ours)
(eurs)
Episode Retum
100
0.8
SMC
Gumbel MCTS
Gumbel MCTS
SMC
80
PPO
PPO
6000
0.6
AZC
SAC
4000
0.4
40
2000
0.0
0
1
2
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
1c8
167
(Planning-Budde Adjusted) Number of Training Samples
Figure 8. Main results from Figure with the adjusted w-axis account for additional true environment samples during planning. In terms of sample-effcienncy this gives a stark contrast to the result of the main paper, and shows that the model-free methods are more sample efficient. However, methods that utilize planning always utilize accurate simulator skew the z-axis similarly the result of Figure 2.
Parameter Sensitivity: Discrete Envs
Parameter Sensitivity: Continuous Envs
Normalized Average Retumss
1.0
N 16
64
0.8
256
0.6
0.4
16
0.2
N 64
256
0.0
0.001
0.01
0.1
0.2
0.4
0.8
0.001
0.01
0.1
0.2
0.4
0.8
Ne
Trust-Region Factor a
f7
Trust-Region Factor a
n
Figure Sensitivity plot the adaptive trust-region parameter a split for the discrete (Jumanji) continuous (Brax) environments. This splits up the results shown in Figure from the main paper.
23


Reviews:
[{"review_id": "B9OxctixGI", "reviewer": "Reviewer_bnoX", "summary": "- The paper creates a framework to apply sequential Monte Carlo planners to reinforcement learning settings by improving data generation through constrained action sampling and explicit terminal state handling,\n- Specifically, SMC methods traditionally estimate the trajectory distribution under an unknown policy, rather than the specific unknown policy at the start of the planning, which causes high-variance estimation mismatches leading to a \"trust-region twisted SMC\" (TRT-SMC); so too reduce the estimates' variance, the paper uses exponential twisting functions to improve sampling trajectories inside the planner,\n- The paper proposes a (to the best of my knowledge) novel method to deal with particles stuck in terminal states using a particular resampling strategy, and\n- Finally, the paper demonstrates a better sample efficiency and runtime scaling through experiments in continuous control tasks and ablation studies.\n\n\n## update after rebuttal\n\nI have raised my score following the author's response.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort and accurate summary of our paper. We address your points below.\n\n---\n\n**Theoretical results:** \n\nOur paper does not introduce new theoretical results, **nor do we claim so**. It is correct that theorem 2.2 is known, it also goes quite farther back than Levine (2018), however this paper has a nice presentation on the theory. We restated this property since, as you also write, to motivate how we use the sequential Monte-Carlo planner and our proposed changes. \n\nThe theorem shows that 1) posterior-policy estimation implies a regularized policy improvement, and 2) that learning on samples of this posterior-policy should maximize this lower-bound. We used the former insight to impose trust-regions on the proposal distribution in the SMC-planner and the latter to perform neural network training in the outer loop.\n\n**We believe that motivating our new method with known prior results should not merit rejection but instead strengthen our approach.** But, could you elaborate whether the critique is that we put too much focus on the background? If so, we will shorten this part.\n\n---\n\n**Response to Technical difficulty:**\n\n > It is unclear why the result is novel, however, as there does not seem to be a technical difficulty underpinning the result\n\nIt is unclear whether you refer to the theory, our empirical results, or our method.\n\nAs mentioned above, the theory is there to support our modifications and show how SMC planning works (e.g., the importance-sampling weights). We don't understand why supporting a newly proposed method with known previous results should be a basis for rejection.\n\nRegarding our method, we discovered and described non-trivial issues in current approaches for SMC based planning in RL, and present concrete approaches on how to deal with this. Prior work overlooked these issues, and their identification requires **deep technical understanding** on integrating RL with SMC. \n\nAlso note that simplicity (if that is the concern) of a new method does not detract from its novelty, instead it should improve its clarity and broader applicability. \n\nPerhaps you could clarify your position?\n\n---\n\n**Minor comments:**\n\nOur method is technically not a parallel TRPO (cf. Fickinger et al. 2021). The main distinctions are:\n 1) The KL-divergence constraint: the arguments to this constraint would be swapped for TRPO vs. our approach (MPO-based; theorem 2.2). \n 2) TRPO can already be parallelized, simply by increasing the number of asynchronous environments.\n 3) There is a fundamental difference in how data is collected and how policy learning is performed -- relating back to our previous comment, we can extend the outer-loop discussion in the background to make this distinction more clear. \n\n---\n\n**References**\n\nFickinger, A., Hu, H., Amos, B., Russell, S., \\& Brown, N. (2021). Scalable online planning via reinforcement learning fine-tuning. *Advances in Neural Information Processing Systems*, *34*, 16951-16963.", "claims_and_evidence": "- The paper rehashes a theoretical insight from Levine (2018) to prove theorem 2.2,\n- The paper does not make any other formal theoretical claims, but rigorously justifies the algorithm design for the MCTS process in Algorithm 1", "methods_and_evaluation_criteria": "The methods seem reasonable.", "theoretical_claims": "I checked the proofs of correctness of the theoretical claims, but it seems to mostly re-hash existing results from Levine (2018).", "experimental_designs_or_analyses": "I did not check the experimental design in detail.", "relation_to_broader_scientific_literature": "The paper provides a nice approach to make parallel TRPO possible.", "essential_references_not_discussed": "- I am not well versed with the new line of literature in this field.", "other_strengths_and_weaknesses": "- The paper is exceptionally well written,\n- It is unclear why the result is novel, however, as there does not seem to be a technical difficulty underpinning the result", "other_comments_or_suggestions": "None", "questions_for_authors": "I'd appreciate an answer to the point above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7xOc0vpkxk", "reviewer": "Reviewer_SRnR", "summary": "Trust-Region Twisted Policy Improvement develops a new algorithm for performing Sequential Monte-Carlo (SMC) inspired by Monte-Carlo Tree Search (MCTS) for performing online planning in RL algorithms, which scales in performance with respect to the number of particles sampled during each planning step. The algorithm modifies and adapts several aspects of prior SMC approaches to be more RL specific - namely 1) fixing the path degeneracy problem through approximate message-passing  2) \"twisting\" the sampling by mixing $\\exp Q$ with the current policy $\\pi$ tradeoff sampling variance for bias  3) modifying value targets to use the values estimated by the search 4) revived resampling of absorbed states to avoid wasting compute on trapped particles", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time, effort and, accurate summary of our paper. \n\nRegarding limited novelty, we understand the concern but would also like to point to a slightly different perspective. With the benefit of hindsight it is easy to dismiss contributions by being reductive. For instance, the SMC-planner by Piche (2018) combined soft-actor critic with a policy extracted from a bootstrapped particle-filter, without significant changes to this filter. An overly reductive summary would say that they simply 'copy-paste' a sequential importance resampler for policy inference into the RL-loop. A similar thing can be stated about SPO by Macfarlane (2024) which replaced the MCTS-planner in AlphaZero with SMC, and then merged this into the MPO framework. There are always components being rehashed, but just like these prior works, it requires deep understanding to identify which parts of an algorithm work or don't work well, and how to deal with this.  \n\nWe directly point out **non-trivial issues** concerning SMC-planning for RL that were not yet addressed, or overlooked, in the literature. Their identification requires **deep technical understanding** on integrating RL with SMC. They are non-trivial since applying a particle-filter for policy inference is fundamentally different to how the method should be used for conventional state-estimation, this introduces challenges that only emerge when integrating them. These problems relate to path-degeneracy, inefficient trajectory sampling, and wasteful data generation. We also directly provide solutions for these issues that work well and have sound theoretical grounding. We also believe that further study of this topic holds significant value to the broader RL community.\n\n---\n\n**On figure 2:**\n\nWe understand your suggestion to adjust the $x$-axis of figure 2 to account for the extra environment samples due to the planning budget. In this case, PPO and A2C would indeed perform much better in comparison to the other agents! For completeness, we added this figure [here](https://www.dropbox.com/scl/fi/l1oswifleo63zsm1x86mb/new_results_adj.png?rlkey=yq2vxtrftt56r9cb00cccnd7h&st=dxjl93m9&dl=0). However, this sidesteps the research question as we wrote at the start of section 5. \n\nWe really want to study which method implements a stronger local policy improvement operator, for which we compare against baseline SMC and gumbel MCTS. These all use the exact same planning budget and number of environment samples. This setting assumes we have a highly accurate simulator, which is also standard in prior work (AlphaZero, SMC for RL, SPO). \n\nWe will include the above 2 sentences to the figure 2 caption and emphasize it more in the main text of section 5 as to not mislead the reader. We will include the attached figure to the appendix.\n\nClick here for supplementary results [folder](https://www.dropbox.com/scl/fo/hy8p1nalfduq1tqf2moaw/AJOYML8UEQy5Lb9UmYIHOfg?rlkey=v887m5zaab0hcx260cg8m5yq4&st=oa6h6xtq&dl=0)\n\n---\n\n**On the background:** \n\nIn line with reviewers bnoX and ub84 we will add pseudocode of the outer-loop training (see [link to EM-pseudocode](https://www.dropbox.com/scl/fi/ktxtkcuq00syv0k7i7vew/outer-loop-pseudocode.png?rlkey=dwxroqk5xrl629v7a4oesye9c&st=07wviyru&dl=0)) either in Appendix C or in the main text depending on whether we have space for additional ablation results. We can 1) add a paragraph heading in the last paragraph of section 2.1 with '**Outer-loop**' to emphasize/ structure the approximate policy iteration part, and 2) extend this text on how the estimated posterior policies $\\hat{q}_t$ are used within this Expectation-Maximization loop in combination with neural network training. Section 2.2 and section 3 then continue in making improvements to the E-step approximation of $\\hat{q}_t$. \n\nIf needed, we can shorten the background, however we do need the definitions of the posterior-policy and value functions. Furthermore, the property of theorem 2.2 leads into how we motivate the exponentially twisted proposals. We can definitely shorten the formulas and move part of this derivation to the appendix (e.g., equations 2 and 3 can be shortened to 1 line) so that the proposed pseudocode and guiding text fit.", "claims_and_evidence": "The following major claims are well supported in this paper:\n* Planning with search is an important problem in RL (citations from Silver et al., 2018, Ye et al., 2021, Fawzi et al., 2022; Mankowitz et al., 2023.)\n* Current bootstrapped particle-filtering methods suffer from several issues, including data-inefficiency and path degeneracy (Figure 1)\n* TRT-SMC improves upon previous methods by addressing these issues (Figure 2, Figure 3, Section 5).", "methods_and_evaluation_criteria": "Yes. They performed experiments on the Brax continuous control tasks (Freeman et al., 2021) and Jumanji discrete environments (Bonnet et al., 2024), both of which are good RL test suites.", "theoretical_claims": "I did not carefully check the proofs.", "experimental_designs_or_analyses": "Yes, the experiments appear to be valid. All methods are trained over 30 seeds, which is sufficient for establishing statistical significance, and hyperparameters and code package version are extensively documented in Appendix B. However, the curves can be misleading as the planning budget is not taken into account, exaggerating the sample efficiency of TRT-SMC and other SMC methods. It would be better to include this in the caption of Figure 2 so that readers who skip the authors' disclosure of this in the main text will not be mislead. Additionally, it would be good to include figures on the performance against \"All Samples\" (including online planning samples) rather than just \"Training Samples\". The TRT-SMC results will not look so good under this x-axis, but it is essential to include as total sample-efficiency is one of the primary concerns in RL.", "relation_to_broader_scientific_literature": "This paper adapts the particle filter planner for deep reinforcement learning by addressing issues in variational sequential Monte Carlo that affect policy inference. The authors introduce TRT-SMC, which reduces path degeneracy and optimizes data efficiency, leading to improved runtime and *training* sample efficiency over traditional methods. They aim to inspire further integration of planning and Bayesian inference tools to enhance reinforcement learning algorithms.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper addresses several important problems with existing SMC planners in RL, but does not propose any single novel component. This gives the paper a feeling of being a 'bag-of-tricks', where several somewhat small things were improved upon to stack together a better method. I think this style of research is good and useful, thus I advocate for accept. However, I will flag that the novelty is limited.", "other_comments_or_suggestions": "The control as inference part of the background can be shortened, as Levine 2018 already has an excellent tutorial on this. It would be great to see a longer exposition of SMC and MCTS, as this is the primary focus of the paper and the average RL reader will likely have much less experience with it.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UjTgfjCyCb", "reviewer": "Reviewer_ub84", "summary": "The authors propose Trust-Region Twisted Sequential Monte Carlo (TRT-SMC) to improve the sample efficiency of SMC methods for reinforcement learning. One of the main motivations is that SMC methods scale better than Monte Carlo Tree Search (MCTS) because they enable higher GPU parallelization. The authors propose several changes to SMC: an adaptive trust region defined between the prior and greedy policies, revived terminal states with resampling, and some other improvements to policy and value estimation. TRT-SMC is compared against PPO, A2C, Gumbel MCTS, and SMC in 4 Brax/Jumanji environments, where it is shown to achieve high sample efficiency and (except for SMC) faster runtime.\n\n## Update after rebuttal\n\nI thank the authors for their rebuttal and follow-up discussion. In anticipation of the A2C results and inclusion of the $\\alpha$-ablations that we discussed, I have raised my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort. We divided our rebuttal in two parts, firstly on the experiment design and the second part on answering your comments.\n\n---\n\n**Choice of environments:**\n\nFor the choice of environments, we wanted a mix of discrete and continuous control environments. For this, we tried to closely stick to the choice by Macfarlane et al. (2024) as we believe they achieve this mix well. It also allows us to accurately compare our SMC baseline results. However we replaced Sokoban with Snake to drastically scale down the experiment costs. Based on Bonnet et al. (2024), we believed Snake to be sufficient as it is a sparse reward environment that benefits from planning, and can be learned with a smaller model and fewer environment steps.\n\nConcerning the ablations, the environments in the ablations were chosen to 1) spread out to not overfit results on one single environment, 2) to scale down experiment cost where possible, and 3) to verify that our modification actually solves an issue of interest.\n\nThis third point is important as on some environments the effect of e.g., $\\alpha$ is more profound than others (see extra results below). For instance, sparse reward environments can benefit more from value-guidance in search. Just like how exploration papers often test their method specifically on Montezuma's revenge, we also want to test our modifications in settings where they are most needed. Showing **isolated** effects of our modifications in ablations while still having good performance across the board keeps the discussion self-contained.\n\n**Choice of $\\alpha$**\n\nThe specific choice of $\\alpha=0.1$ we used in the main results (figures 1 and 2) seemed to be robust across environments (not tuned per environment). This value also mirrors common trust-region parameters of $\\epsilon = 0.1$ used in TRPO, MPO, and SPO. \n\n**Extra results:** We did not have complete results for $\\alpha$ ablations at the time of submission (only a few seeds) and have since ran experiments for $\\alpha$ to a greater extent. These results can be found here for: [on discrete envs](https://www.dropbox.com/scl/fi/4syw502kk6sstt6xlxwcd/alpha_sensitivity_disc.png?rlkey=np5ffukyviuat6ft8j43nmeix&st=rwzepxn4&dl=0), [on continuous envs](https://www.dropbox.com/scl/fi/y1ygjj4ez5mzjb0ckoogm/alpha_sensitivity_cont.png?rlkey=6sdycmjr44v6hz73xbeuo4pu6&st=9vip5ugs&dl=0), and [aggregated across envs](https://www.dropbox.com/scl/fi/nf7wrdchu3zl1751cs5mm/alpha_sensitivity_combined.png?rlkey=c98sprlxesszepyekzciz25pz&st=irxuoi50&dl=0). The figures report final min-max normalized average returns on all environments (30 seeds). The different lines show different planning budgets of $N=K \\cdot m$ where $K$ (particles) and $m$ (depth) with $K=m$, and small $\\alpha$ means sticking to the prior $\\pi_\\theta$ and large $\\alpha$ means a greedy policy $\\hat{\\pi}^*$ over the predicted $Q$ values. \n\nSimilar to e.g., the clipping-ratio in PPO, different environments have different optimal $\\alpha$ ($0.1$ for discrete, and $0.01$ for continuous). Overall $\\alpha=0.01$ seems to work even better than what was used in our submission $\\alpha =0.1$ (which we picked from preliminary testing). However, both of these values are robust across environments.\n\nClick here for supplementary results [folder](https://www.dropbox.com/scl/fo/hy8p1nalfduq1tqf2moaw/AJOYML8UEQy5Lb9UmYIHOfg?rlkey=v887m5zaab0hcx260cg8m5yq4&st=oa6h6xtq&dl=0)\n\n**A2C:**\n\nA2C is indeed a weak baseline that could be removed. But, we included it to contrast the difference in implementations so that readers can compare against results by the official jumanji code.\n\n---\n\n**Answers to Questions:**\n\n1) This means that the SMC and MCTS agents use more environment samples (planning budget) per 'real' environment step. We do planning with the real environment model. This mirrors having a highly accurate simulator. See also our response to Reviewer SRnR on Figure 2.\n\n2) PPO is slower in figure 3 due to the bottleneck of neural network training and relative cheapness of the environment model. This means we can use SMC planning to cheaply draw more environment samples which can then 1) extract a more informative learning target and 2) generate more diverse data (Hamrick et al., 2021).\n\n**Outer-loop pseudocode:**\n\nWe prepared pseudocode for the outer loop to extend our discussion on approximate policy iteration in the background (see our response to reviewer SRnR on how we intend to revise this).\n\n**On $q^\\star$**\n\nSadly, the convention of optimal state-action values coincides with variational inference for which $q^*$ is standard to the optimal variational distribution. So we opted to write value functions with capitals $Q$ (with a hat $\\hat{Q}$ for approximations) and use small $q$ for variational distributions. It is a sacrifice.\n\n---\n\nWe hope our response addresses your concerns, and we are happy to discuss on any further concerns or questions, and explain the extra results in more detail.", "claims_and_evidence": "Claims are supported with evidence, but evaluation criteria are sometimes questionable (see below).", "methods_and_evaluation_criteria": "- It seems that only a very small subset of Brax and Jumanji were used for evaluation (Figures 2 and 3). Only 4 environments are reported (2 from each domain), which raises concerns of potential cherry picking. I could not find any justification for the choice of these 4 specific environments in the paper.\n- The A2C results are unreliable. Only 3 seeds are used for the learning curves in Figure 2, which are apparently taken from a previous paper. No confidence intervals are shown, so these results are not statistically significant and should not be reported.", "theoretical_claims": "The core algorithm and the proposed improvements are all reasonable ideas. The rationale behind each design choice is strongly justified. Mathematical results are correct to the best of my knowledge.", "experimental_designs_or_analyses": "- Some hyperparameter choices appear arbitrary and are never justified: e.g., using $\\alpha=0.1$ for the trust region.\n- Ablations are conducted on only 1 environment each (Table 1), and the environment choice varies across ablation. This again gives the impression that the results may have been cherry picked. It would be better to conduct the ablations on all environments and include them in the appendix.", "relation_to_broader_scientific_literature": "Great background discussion and related work. The contributions are clearly placed in the literature.", "essential_references_not_discussed": "None to my knowledge.", "other_strengths_and_weaknesses": "**Strengths**\n\nStrong empirical results in a few Brax and Jumanji environments. TRT-SMC appears to slightly worsen runtime over SMC because of the additional features, but it achieves better sample efficiency in the tested environments. It also appears to be both faster and more sample efficient than Gumbel MCTS and PPO.\n\n**Weaknesses**\n\nAlthough the paper goes into great detail regarding the proposed improvements to planning and particle filtering, the description of the outer loop (which interacts with the environment) is largely missing. What is especially unclear to me is whether the transition model is assumed to known or be must be learned. I think it is the latter based on the discussion of variational SMC in Section 2.2, but it would useful for reproducibility to include this explanation or pseudocode.", "other_comments_or_suggestions": "It is confusing to use $q^*$ for the policy, because $q^*$ is typically reserved for the optimal action-value function. Is there possibly another symbol that can be used?", "questions_for_authors": "1. Could you clarify what is meant here? “Although we compare sample efficiency to the baselines, this is only for reference since we do not account for the additional observed transitions by the planner” (end of p. 6). Are you not counting some of the sample interactions with the environment?\n1. I was really surprised to see that PPO is the slowest of the 4 methods you compared, when it is the only model-free algorithm. Why is that the case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OEJlE53QNV", "reviewer": "Reviewer_Pz2A", "summary": "The authors propose several improvements to SMC:\n\n1. Addressing path degeneracy by performing an online estimate of $Q_{SMC}$.\n2. Using the $V$ values obtained by the search algorithm.\n3. Reviving trapped particles.\n4. Changing the proposal policy to be in the trust region.\n\nThe authors provide experiments to show their improvement over the original SMC and a recent variation of MCTS.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort and accurate summary of our paper, we appreciate that you recognize our work as a solid advancement to SMC-planners for RL and the positive assessment.\n\nWe address your points below.\n\n---\n\n**Minor comments:**\n\n - We included the derivation for Corollary 2.3 in the appendix A.4, but forgot to reference this. We will update this.\n - The reference to Fig.5 vs Fig.3 in the main text and appendix is a mistake, this should be swapped. Good catch!\n\n---\n\n**Figure 1:**\n\nIn Fig.1 we show that running the baseline SMC planner for a longer horizon (depth) causes the distribution over actions $\\pi_{SMC}$ at the start to diverge from their optimal distribution $\\pi^*$. This is indicated in the y-label with $KL(\\pi^* \\Vert \\pi_{SMC})$. We believe it's important to show the pattern that our method doesn't worsen as a function of depth. The complete details on how we generated this figure are in Appendix B.6, we will add a reference for this in the figure 1 caption.\n\nThis pattern is a problem in the baseline because it increases variance of the policy estimation which cascades into exploration and learning (as also noted by Reviewer bnoX). We tested our modification in the ablation-study in figure 4 on the right. We tested this over multiple parameter settings due to hyperparameters interacting with one-another. This follows principles laid out by Patterson et al. (2024). See also our answer below on the ablations.\n\nFor clarity, we will copy the ylabel in Fig.1 from the left-bottom also to the plot in the right-bottom.\n\n---\n\n**Learning diagram:**\n\nWe prepared pseudocode for the outer loop to complement our discussion on approximate policy iteration in the background and the inner-loop pseudocode in the appendix (see also our response to reviewer SRnR on how we intend to revise this). The pseudocode can be found here: [link to EM-pseudocode](https://www.dropbox.com/scl/fi/ktxtkcuq00syv0k7i7vew/outer-loop-pseudocode.png?rlkey=dwxroqk5xrl629v7a4oesye9c&st=07wviyru&dl=0)), we will put this either in Appendix C or in the main text depending on whether we have space for additional ablation results. \n\n**Ablations + Answer to Question:**\n\nWe tested out all the improvements separately to the best of our abilities and resources, across varying environments and marginalizing over many parameter settings. This is mostly shown in figure 4 and table 1. Because there were so many hyperparameters involved, the details of the ablations were put in the appendix B.3. Furthermore, we simply had to make sacrifices in what to test within this immense combinatorial space of hyperparameters.\n\nWe recognize that the details of each ablation study got a bit buried in section 5 when trying to connect this back to section 3, where we described the improvements. \n\n**Q:** As a minor revision, we will annotate each section in the experiments. Would this help? \n\nFor example, the contribution in section 3.2 is tested in the experiments in line 434 (col 1), we propose to 1) reference this part in section 3.2 and 2) cross-reference back to section 3.2 in the paragraph-heading -- e.g., **Revived Resampling (§3.2)**.' \n\nIn summary,\n - §3.1 improves sampling inside SMC, this is studied in figure 4.\n - §3.2 prevents particles from getting trapped in absorbing states, wasting compute resources. This is algorithmically shown in Alg.2 (appendix C). It is studied empirically in table 1, setup in appendix B.3.\n - The problem of §3.3 and §3.4 is illustrated in figure 1. \n - For §3.3, our 'fix' is shown in figure 1 bottom-right and studied in the right of figure 4. \n - §3.4 also improves path-degeneracy, the improvement is studied in table 1, setup in appendix B.3.\n\nDespite the details in the paper, we have included the table below for your convenience that summarizes the performance detriment of turning off each modification for our TRT-SMC shown in Figure 2. Note, that due to limited time and resources within this rebuttal period, we could only show this on RubiksCube for 10m training steps with the same statistical precision (30 seeds 99% intervals). Note that on this environment, and for this combination of hyperparameters, that some choices have less profound effect on the final performance than others. With the twisting having the strongest effect and revived resampling not differing statistically significantly. However, these effects can change as e.g., the depth of the SMC planner increases.\n\n| Rubikscube             | All      | w.o twisting | w.o policy backup | w.o. model-value | w.o revived resampling |\n| ---------------------- | -------- | ------------ | ----------------- | ---------------- | ---------------------- |\n| Solve rate $\\hat{\\mu}$ | $0.5835$ | $0.193$      | $0.5195$          | $0.5629$         | $0.5828$               |\n\nFor the camera-ready, we are currently running experiments for this table on all environments.", "claims_and_evidence": "The authors propose 4 changes to SMC, it would make sense to test the merit of each separately in a clear and simple figure. \nI found the figures to be complicated and I don't think such a simple test was done on all 4 improvements.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No - Thm 2.2 is taken from another paper, Corollary  2.3 has no full proof but makes sense. No other theoretical proof is given.", "experimental_designs_or_analyses": "No. The plots in Figure 2 look reasonable to me.", "relation_to_broader_scientific_literature": "It is an improvement to SMC.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "Even though the paper has mostly technical improvements I think it provides a solid advancement for SMCs and can be beneficial to the community. \n\nIn my view, the paper has two weaknesses:\n\n1. Clarity - The paper is full with details and it's hard to follow. I'd like to be more specific, but I think that it's the entire paper - it's just complicated. What I would think would immensely are diagrams showing the learn process and the proposed improvements. \n\nAlso the plots could be simpler: \n\nIn Figure 1 what is the y-axis of the message passing plot? is it really figure-worthy if its a constant value (maybe it is possible to  calculated it directly?)\n\nFigure 3 is in the main paper but only referenced in the appendix, Figure 5 is in the appendix but referenced in the main.\n\n2. There are several claims like X is problematic in SMC and we solve it with Y. But how much X is problematic is not always clear or shown. Also the improvement with the solution is not always shown and some simple ablation study is needed to just compare the final result each improvement with and without. I understand it's not always easy, but that seems to be the main contribution of the paper.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Can you show that each improvement you proposed on its own improved the training curve, and also show that it address the issue you've raised?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The proposed TRT-SMC algorithm aims to enhance Sequential Monte Carlo (SMC) methods for RL by integrating several techniques targeting issues like path degeneracy and sample efficiency. Reviewers found the constituent ideas reasonable and acknowledged the strong empirical results shown in limited test environments. However, significant concerns were raised regarding the work's novelty, often described as a "bag-of-tricks," and the rigor of the experimental evaluation, particularly the small number of environments and insufficient ablation studies, suggesting potential cherry-picking. Despite these major reservations, the consensus leaned towards weak acceptance, recognizing the potential practical value.
Decision: Accept (poster)