Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
Authors: Hany Abdulsamad
Sahel Iqbal
Simo Särkkä
Venue: NeurIPS 2025 poster ()
Keywords: POMDP
Policy Optimization
Sequential Monte Carlo
Feynman--Kac
Control-as-Inference
Decision-Making under Uncertainty
Primary area: reinforcement_learning
TL;DR: We solve POMDPs by nesting sequential Monte Carlo

Abstract: Optimal decision-making under partial observability requires agents to balance reducing uncertainty (exploration) against pursuing immediate objectives (exploitation). In this paper, we introduce a novel policy optimization framework for continuous partially observable Markov decision processes (POMDPs) that explicitly addresses this challenge. Our method casts policy learning as probabilistic inference in a non-Markovian Feynman--Kac model that inherently captures the value of information gathering by anticipating future observations, without requiring suboptimal approximations or handcrafted heuristics. To optimize policies under this model, we develop a nested sequential Monte Carlo (SMC) algorithm that efficiently estimates a history-dependent policy gradient under samples from the optimal trajectory distribution induced by the POMDP. We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks, where existing methods struggle to act under uncertainty.

Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
Hany Abdulsamad- 1,2
Sahel Iqbal 22
Simo Särkkä²
University of Amsterdam iqpael.iqbal, simo sarkka]@aala fi
Aalto University
h. abdullamadduuv....
Abstract
Optimal reciiion-makinge under partial observability requires agents to balance reducing uncertainty (exploration) against pursuing immediate objectives (exploita- tion). In this paper, we introduce a novel policy optimization framework for continuous partially observable Markov decision processes (POMDPs) that explic- itly addresses this challenge. Our method casts policy learning as probabilistic inference in non-Main Markovvan Feynman Feynman-Kac model that inherently captures the value of information gathering by anticipating future observations, without requir- ing suboptimal approximations under this model, we develop a nested sequential Monte Carlo (SMC) algorithm that efficiently estimates a history-dependent policy gradient under samples from the optimal trajectory distribution induced by the POMDP. We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks, where existing methods struggle to act under uncertainty.
handcrafted heuristics. To optimize policies
1 Introduction
Optimal decision-making under uncertainty central to building autonomous agents capable of operating in real-world environments. The ability to quantify and act on uncertainty enables agents exhibit greater autonomy and robustness. allowing them react effectively beyond controlled laboratory environments
a hallmark of adaptive systems [Feldbaum, 1963). In practice, sensory limitations or environmental factors often conceal information required for optimal decision-making. This setting has been a long-standing challenge in the field of decision theory and the canonical framework to deal with such scenarios is that of partially observable Markov decision processes Sondik 1971).
(POMDPs) [strrm 1965
Aoki 1967
The primary challenge of POMDPs lies in the entanglement of probabilistic inference and sequential planning. At each step, agent must anticipate how its actions will shape future observations, what those observations may reveal about the latent state, and how this information can influence future decisions. Effective decision- making in POMDPs hence necessitates of all possible
exhaustive consideration
trajectories over long horizons. While exact solutions exist for discrete POMDPs with small spaces [Cassandra al.) 1997 Porta et al., 2005 Poupart al.) 2006). continuous POMDPs pose far greater challenges, as the beliet state becomes an infinite- dimensional object. Tractable solutions are restricted linear-Gausser Gaussian problems with quadratic rewards [Aoki] 1967 Stengel 1994], special setting where inference and control decouple without loss of optimality [Bar-Shalom and Tse 1974). In more general continuous settings, existing methods often rely on problematic simplifications, such as invalid decoupling of inference from control [Tse and Bar-Shalom 1975 Li and Todorov 2007), assuming -axinum-likelihood future observations |Platt et al., 2010 applying local Gauss-nn approximations [Van Den Berg
cet al. 2012 Indelman et al. 2015).
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Simulation- -based methods have recently advanced learning in POMDPs by considering only random subset of possible paths the space of states, observations, and actions, thereby mitigating the curse
of histories [Pineau et al., 2003]. This includes both trajectory- -baeed planners (Hafner et al. 2019b
Wang et al. 2020] and deep reinforcement learning (RI.) algorithms (Wierstra et al. |007] Ig et al. 2018 Hafner et al., 2019a Han et al., 2019 Lee et al. 2020) Meng et al., / et al. [023] However, a weakness shared by many of these RL approaches is their adoption of the so-called QMDP approximation [Littman al. 1995] to optimize policies based on latent-state dynamics, rather than reasoning over the space of beliefs. In QMDP, the latent state is assumed to be fully observable after single step, thereby ignoring the impact of future observations As a result, information gathering becomes incidental, rather than intentional.
decision-making
We propose a novel policy optimization algorithm for learning in continuous POMDPs, By directly reasoning over belief trajectories that account for future observations, our approach results in deliber- ate information gathering. It builds on the well-established connection between optimization and
probabilistic inference [Dayan and Hinton| 1997| Attias 2003 Kappen 2005] Toussaint and Storkey 2006 Todorov 2008 Rawlik et al.) 2012 Levine 2018 Watson al., 2020]. This framework is
particularly well-suited for POMDPs, as it casts the coupled interaction between belief propagation
and pociiionn-mkinn as unified statistical inference problem. This perspective enables the use of state-of-the the-art inference techniques, avoiding the common approximations that often limit traditional approaches. Our key contributions are:
A Feynman Kac model that captures the adaptive nature of decision-making in POMDPs and naturally incorporates the value of future observations.
A nested sequential Monte Carlo (SMC) method that simulates adaptive decision-making by sampling from the optimal trajectory distribution of a POMDP.
A policy optimization algorithm framed maximum likelihood estimation within the Feynman-Kac model, resulting in a novel policy gradient method for POMDPs.
We begin by introducing the notation and objective of POMDPs. We then present our nested non- Markovian Feynman-Kac model, which captures the coupling between inference and decision- making in POMDPs. Building on this, we develop efficient sampling scheme for policy optimization. Finally, we relate our approach to prior work and evaluate it on standard POMDP benchmarks.
2 Partially Observable Markov Decision Processes
A partially observable Markov decision process is defined by the tuple (S,A, Here, S C Rdd., A C Rds, and Z C Rd= are the sets of allowed states, actions, and observations, respectively. At any time E N, the transition model f S x A S R20 defines the likelihood of moving to a state St+11 after taking action a; in state st, denoted as f(3++1 8t, 4z). The initial state follows Sq ~ p(sy). The reward function R4 S A R assigns a reward to a state-action pair (st, 1 and the observation model g S x Z + Rgo is the probability of observing 2n when the state is st. written as ( st). In a POMDP, the true state s is latent. The agent instead maintains a belief state p(s,1 Z0:t, a0x-1) given a history of observations 2p: and actions ap.-- 1, with ag:-1 = 0 by convention. Finally, a policy a mapping from observation-action histories to actions. We consider stochastic policies (at
20:s, a0:t-1) parameterized by d.
Modern approaches for POMDPs [Igl et al. 2018 Lee et al. 2020 Hafner al. 2019a Meng
ct al. 2021 Chen et al. 2022] commonly adopt a learning objective o" argmax, J J(Ø), where
Eps ΣT11 [s a)-1) is the expected cumulative reward under the joint density (o Zorr, 11) 1) of all random variables:
[()
po(.) = p(so)
II
f(st41 st, ar) appla| 20:0, 00:-1) (=0
II g(Eu ss))
}
We refer to the objective [Ø) as the state-space objective, as it is defined in terms of the state-action reward (.) and the generative process (.).) induced by the state transition dynamics f(-). We
For notational convenience, and without loss of generality, we consider transition based reward definition Hs(se, Oe-1, 8;-1) Sutton and Barto 2018 in which the dependency on st-1 is omitted.

---

denote this underlying process as the state-space generative process. While valid, the objective J() obscures the adaptive nature of the decision-making process by not explicitly conditioning on intermediate observations. This makes it difficult to reason about how the agent's s beliefs evolve over time, an essential aspect in POMDPs [Kelbling et al., 1998 Thrun et al. 2005]. In practice, optimal decision-making in POMDPs invol two intertwined components, as described by the dual control framework [Bar-Shalom and Tse 1974]: a backward-looking inference process, which updates the belief state p( 20:z, 00:r-1) 1) using realized observations, and forward-looking planning process, which selects actions to maximize the expected future reward (T: under the predictive distribution Po( (Serr, a::r-1; 2++1-11 20:z, dg:--1). Crucially, accounting for future observations 2(+1:2'' enables exploration, as it encourages the agent to consider how future, -o--teeeeeeen information will affect its beliefs and, consequently, its actions.
To model adaptivity, we depart from paradigms that rely on state-space generative processes. Instead, we draw on the work of Thrun [1999] and construct an adaptive Monte Carlo simulation procedure which we denote as the belief-space generative process. We demonstrate that the state-space objective [()) is equivalent to a belief-space objective (()), which is governed by the belief-space generative process that explicitly tracks and updates the belief.
Proposition 1 (Belief-Space Objective). The expected cumulative reward objective I() defined over a sequence of state-action reward functions (8 at-1) evaluated according to a state-space generative process po (O(T, To:T, 01:T-1) is equivalent to
T
( = E (FoTT Go:T 1)
Σ
le le(20ct oox--1)
where & (20:, 0m--11 = B p(se (e/e z0:,, 20:,,00: a0 (st is the expected reward under the belief p(st Zoas -1) and po 41:T-1) characterizes the belief-space generative process:
T-11
Po(zo:: 00:--1) = p(zo) II t=0
p(ze-1 1 2a:s, ag:)) no(as) 20st: a0:t-1);
with p(zo) = E p((s)) 19(60| so) and
p Zazts an:)) = /15 g( s4+1) f(s++1 8t, ae) p(se zoa, a0:-1) ds, d8++1
The proof is in Appendix A. Proposition]] T defines objective that factorizes according to the causal structure of decision making: an agent's 's belief p(s; I.) depends only on past quantities. This reformulation expresses the sequential decision-making problem explicitly in terms of histories (Zo.t, 00 and highlights how agents are incentivized to explore by steering the belief towards informative observations that improve the expected utility 4-(-) (Kaelbling et al. 1998). Next, we leverage this belief-space generative process and objective within a probabilistic inference framework.
3 POMDPs as Keynman-Kac Models
We present novel technique to simulate adaptive decision-maken and learning in POMDPs. We leverage the connection between optimization and inference (Toussaint and Storkey 2006 Levine 2018) construct a Feynman-Kac (FK) model that emulates the decision-making process in the belief space (T]. and derive a policy gradient approach for policy optimization. We start with a brief introduction to Feynman-Kac models and sequential Monte Carlo.
3.1 Kay Models and Sequential Monte Carlo
Let Forr be a sequence of random variables taking values in ((++1), Their joint probability density (xo.)) can be decomposed as (x = Mo( II-_ Mx xo.t-1), where Mi is the prior law of xo and M, are transition kernels. Now, let Go X + R+ and G. : ((+++) + R for t {I,..., T3 be so-called potential functions. Then the sequence of distributions
(c(x))) = Moo(zo) Go(zo) II (* (z) Gu(Zu)). t (O.....Ty T). (3)
L.
3

---

defines a Feynman-Kaa Kac model Del Moral 2004 on X((++1), with L, being the normalizing constant.
Sequential Monte Carlo (SMC) algorithms or particle filters provide discrete approximations to Feynman-Ko Kac models [Chopin and Papaspiliopoulos, 2020). Here we present the bootstrap particle filter from Gordon et al. Go (a²) for n. E [1,...,
[993]. We start by drawing i.i.d. samples zi M. with weights wß x > 1, and alternate between the following steps for & E ......T):
N N
]N 1)= 1. Resample: Sample A Categorical {w (0p-1)==1/
2. Mutate: Generate importance samples =² Mr(-) 10tt11).
A
3. Reweight: Assign importance weights w/ = WT Σx W where WT G.(x
4" Here, :ie is defined recursively as = (- 2/)) The algorithm returns a discrete approxima- tion En= Wx ((ro: -0000) x Q (Toi)) at each time step where 5 is the Dirac delta function. also provides an unbiased estimate of the marginal likelihood increment Le/ L + En= WT For t == T, samples from the particle filter approximate the terminal smoothing distribution Qe (Zo:T).
3.2 Probabilistic Inference for POMDPs
Proposition [ defines a sequential decision decision-makken making problem over observation-action histories, resem- bling objectives from stochastic control and reinforcement learning [Aoki] 1967. Sutton and Barto 2018). This allows us to draw on established techniques for solving such problems, particular, since inference and
are inherently coupled in POMDPs, we adopt the control-as-inference
perspective (Toussaint and Storkey] 2006 Levine, 2018 and construct a Feynman-Kac model over (Zo:r, (0:1-1) ) trajectories. This probabilistic formulation yields a surrogate of the belief-space objective that serves as the foundation for our policy learning approach.
Let {Or be sequence of auxiliary binary random variables with history-dependent likelihood
p(O: = 1 20:1, (0:--1) 1) x exp {1) (z(20 1)) t E (I,...,,,
(4)
where fe(-) is the expected reward function specified in Proposition T and > is a constant. We assume the reward function RS (and hence &,) is bounded. {O, = 1) represents the event that the pair
(ze, 41-1) is optimal given the partial trajectory (Zo:t-1, agx--2), ), and higher probability of optimality corresponds higher expected reward ég. The exponential function strictly positive and monotonie. Positive potential functions are required by the Feynman-Kac formalism and monotonicity guarantees that the potential function maintains the same maximizer as the reward function. For brevity, we write O to denote the event (O, 1} and O1 == - (O =1) to denote optimality of an entire
horizon. With the dynamic process (2 and potential functions we construct an FK model
V (F a0:--1 1i ②) = po (zu:e ap:e-1 O1:)
{
(0())) p(20)
II
p(=x+ 2ocks aoak)
(5)
A=
Ta(a: Zua, aod-1) II 1=0
II m=1
p(O. 20.m ap:m- 00:m-1)
for (1...., T}. V. represents the joint distribution over observation-action trajectories condi- tioned on optimality until time f. The normalizing constant
(0 = p(O1a Zogt, 00:t-1) (zon, 404-1) d'ag dap:-1
quantifies the probability of generating optimal trajectories under the policy Tg. making a natural optimization target in the cnntrol-as-infeeee framework (Toussaint and Storkey 2006
This alternative objective serves as a surrogate for the POMDP objective B(x) and can justified
from multiple perspectives. Optimizing the evidence lower bound [ELBO, Blei et al. 2017 of this
objective is equivalent to solving maximum entropy RL. problem (Ziebart al. 2008 Rawlik et al. 2012 Levine 2018). Alternatively, directly maximizing log p (O1z)) corresponds to optimizing an

---

Algorithm 1: Particle POMDP Policy Optimization (P30).
Initial policy parameters do. step size sequence {ox) and number of samples N. output: Optimal policy parameters "²
input:
Set k 0.
while not converged do
Sample [ - approximately distributed as vr((Φb). // Algorithm
Estimate the policy gradient 9K < 1
N Et Vo log To a/s ab. 1))==)
Update parameters 0x41 t- Φb cr 9k and set + k +1
optimistic risk-sensitive objective Marcus et al.
1997 Rawlik 2013
Watson et al.
2020
1 (o) log (O log L po 11---1) exp
(z a0c-1)
(6)
17
1)
t=1
The hyperparameter 7) controls the degree of risk, with the risk-neutral objective (()) in the limit 1 - 0. In this work, we maximize the objective (Ø) directly using a Monte Carlo policy gradient algorithm, which we describe next.
recovered
3.3 Particle POMDP Policy Optimization (P30)
The risk-sensitive helief-space objective (Ø) is proportional to the log-normalizing constant log (..T).).. As noted Section 3.1 particle filters provide unbiased estimates of the normaliz- ing constant, hinting at a natural approach to policy learning: obtain gradients by differentiating through a particle filter and use gradient ascent. However, the gradient cannot be obtained this way because standard resampling schemes are discontinuous. While differentiable resampling schemes exist [Corenflos et al. 2021]. they do not easily extend non- Mon-Markovian systems. We circumvent this issue by estimating the policy gradient directly using Fisher's 's identity (Cappé et al. 2005].
Proposition 2 (POMDP Policy Gradient). Consider the risk-sensitive belief-space objective B, from ) and the distribution V/T(2O,T, U0:T-1; Φ) from 5 Under suitable regularity conditions, the puthwise policy gradient of (,( satisfies
V (d) E ( -1:
V log To(as 20:e av.-11)
The proof is available in Appendix A.2 gradient of the objective (o) with respect to the policy parameters á by integrating Vg. log ne. under the Feynman Kac distribution Ty This integral is intractable, we propose constructing a particle filter targeting V and using the resulting Monte Carlo samples to estimate the policy gradient (Kantas et al. 2013). The meta procedure outlined in Algorithm
Proposition 2 provides principled way to estimate the
While a Monte Carlo estimate of (7) resembles the REINFORCE estimator [Williams) 1992],
differs significantly. REINFORCE is the gradient of the risk-neuiral objective B(Ø) given by:
VΦ (()) x E. Po (0)
Σ
Σ (z ao:k-1)
Ve log (at Zost: a0
where Po(20:T, (0:T-1) 1 is the prior policy-induced joint distribution. Monte Carlo estimates of this expectation typically suffer from high variance, as the prior distribution p=(-) is often uninfor- mative with respect to the reward and rarely samples from high-reward regions. In contrast, our method samples trajectories directly from the weighed posterior distribution (r(#)), which concentrates probability mass on high-reward trajectories due to conditioning on the optimality events O 1:T (4) As a result, our estimator benefits from lower variance relative to REINFORCE. When using sequential Monte Carlo approximations of (= do:T Φ). which we describe in the upcoming section, this variance reduction arises naturally from an importance (re)-sampling correction, yielding more efficient estimator for the policy gradient.
5

---

Algorithm 2: Particle filter to sample from (rr(z 11
input: Number of history particles N, number of belief particles M, and temperature
output: Weighted particle set (=0 - u7 = approximating (z Qu:r-1: 1))
notation: Operations indexed by n and m are run over n = I,...,N N and m = I,...,... M.
Sample belief particles shnh po(su) and set with 1/M.
// Initialize
Sample observation 2i ~ Em= "h (zo s)m) and set og + 1/N.
for t 4 to 7' do
Sample history ancestor indices: AT Categorical (N, [oe] In=1). ).
// Resample
Sample belief ancestor indices: B, Categorical [M, {w, (um M
Jm=1)
Replace history particles: {=0te: 10:t-1; v0²) f- (20x0
A7 A
Go:- 1/N].
Replace belief particles: {s? ww! {s(
Ar 87
1/M3.
Sample action: af ~ TT(- Zibee (b:
// Mutate
Propagate belief particles: stti ~ f(-1sq (.m a.)) Sample observation: 2 ~ Em= w g(- stt)).
10
Weight and normalize belief particles: witi with g( s41)). Estimate expected reward: 4+ (=0:++1 (ūa) ) s Em=l w Ret1 o.))
// Reweight
12
13
Weight and normalize history particles: oft1 + oa exp {1) 47 (20 4
Implementing a particle filter for V requires sampling from the sequence of target distributions V defined by the Feynman-Kac model in (S). A critical challenge in POMDPs is that the construction of these target distributions depends on the intermediate belief distributions p( F0:t, a0:t-1). 1). as both the utility function (z(z0 d0:t-1) and the predictive process p(z++1 20:, ag:r) are implicitly functions of the belief. To address this dependency, we propose nesting two particle filters. The outer filter targets the trajectory-level distribution defined by the Feynman-Kac model, while the inner filter maintains optimize the parameters & via the policy gradient method in Algorithm [] particle filter in detail in the next section.
estimate of the belief states required at each step. We use the outer samples to
We describe the nested
Remark 1.
The target distribution VT
5 admits the following decomposition:
T-1
p(O Zoce, ao:t)
VT 11) p(zo) O1r))
II t=
p(= R0: doa, Oz+ p(O,+1:T 20:t Qp:t-1)
(8)
The terms log p(O | [2o.t: On 11) and log (lgg((00+11 [Zunt 11 az) stand for the soft value functions Vi (2 00.--1) and Qe 00:t-1l. at) in the control-as-inferene framework, as they describe probabilities of future optimality given partial trajectories [Levine] [0188) Soft actor-critic methods learn parametrized approximations of these functions [Haarnoja al.) 2018 Lee et al. 2020 Zhang el al. (2033). In contrast, we sample from Vr directly using a particle filter, bypassing the need learn explicit value functions. This decomposition is detailed in Appendix A.3
3.4 A Particle Filter for the Distribution !
Following the policy optimization algorithm described in the previous section, the final component required by our method is procedure to generate samples from the target trajectory distribution (a 10:T-1::0 o). This section introduces a novel nested particle filtering scheme that enables such sampling efficiently, thereby supporting the implementation of Algorithm I
As illustrated in Section 3.1
particle filter approximates Feynman Kac models defined by a sequence of transition kernels M, and potential functions G.- Comparing our target distribu- tion in (5) to the model 3 we observe that V. corresponds to a Feynman-Kac model with M. = Po P2. Zz: 0t-1 20:--11----- 20: and G, = exp (n -0) However, direct application of particle filtering is not feasible because here both M, and G, require computing expectations with respect to the belief distribution p( 20:,, 40:-1), which is generally not available in closed form.

---

To address this, and inspired by the success of nested sequential Monte Carlo algorithms in pa- rameter estimation problems [Chopin et al.] 2013 Crisan and Miguez 2018], we propose nesting two (bootstrap) particle filters with interleaved updates. The first particle filter, denoted as the fiter, maintains set of M weighted partieles {&(" wgy ] m=1 approximating the posterior
p((r] 20a, d0:t-1) RJ EM = Wi 0(8; s(")) according to weights updated recursively using the observation model, w. X wi-1 (ze (s")). The second filter, denoted as the Feynman-Kar filter, targets Mr. while maintaining set of N weighted particles {z0 ay: "0:t-1. U. {s} w/ 1 ) m== with the associated weights updated according to the expected reward i, o o/2 (=(=0 (0:01))00111))) 1). Notice that the state of the Feynman Kac filter contains the full states of N unique and independent belief filters, each associated with a history particle (z0:,. -0x-1) These belief filters are used within
the Feynman Kac filter to approximate the reward l a0:--1) R Σm= wp Re(s? .ap-1) and predictive observation distribution p(2 20:4; 00:z) x > ii " 9(2+11 |$(41) according to their definition in Proposition Algorithm 2 outlines the detailed filter operations.
This interleaved construction of particle filters yields set { oi:r- u7 ] of weighted samples that are approximately distributed according to Vr,, and can be used to estimate the policy gradient in Algorithm| I Nonetheless, such samples may suffer from a problem known as path degeneracy: as T increases, all trajectories tend to coalesce and share a common ancestor at the initial time step [Del Moral and Miclo, 2001]. This can lead to higher variance of the policy gradient (7) To mitigate this, an additional backward sampling step [Godsill et al.) 2004] can be applied after performing Algorithm 2 yielding more diverse trajectories. We detail this procedure in Appendix C
Algorithm 2 has computational complexity O(N which increases to ONN when em- ploying backward sampling (Appendix C.. For long-horizan problems, we therefore recommend avoiding backward sampling and instead tuning the parameter 17 to mitigate particle degeneracy.
Related Work
A substantial body of literature addresses online planning in POMDPs. These algorithms plan from the current belief, execute the chosen action, incorporate the new observation, and then repeat the cycle (Silver and Veness 2010 Ross et al. 2008 [Somani et al. 2013 Kurniawati and Yadav] 2016 Ye ad.[2017] Although this loop makes them flexible, it also imposes significant computational cost, since the planner is invoked after every observation. This procedure may violate the real-time requirements of certain systems. Moreover, most online planners are designed for discrete POMDPs and rely on Monte Carlo Tree Search (MCTS) variants (Kocsis and Szepesvári] 2006, [Browne] jet al. 2012). While these MCTS-based solvers have been successfully extended to continuous domains (Suneeg and Kochenderfer 2018 Lim et al., 2021]. such approaches rely on heuristics, like progressive widening, to artificially limit the branching factor of MCTS.
In contrast to online planners, separate class of offline algorithms Iearn value functions and corre- sponding policies to seleet optimal actions without replanning (Kaelbling al. 1998 Thrun [999]. Estimating value functions in continuous POMDPs is difficult because they are defined over infinite- dimensional belief spaces. Therefore, Littman et al. 1995) introduced the QMDP approximation. which replaces belief-space Q-values with the expectation of fully observable state-space Q-values under the current belief, see Appendix B.I and B.2 for more details. This simplification ignores the value of future information, preventing the agent from performing directed exploration, and is therefore sub-optimal [Ross et al. 2008). Despite this, QMDP underpins many popular modern deep-RL algorithms [Hafner et al. 20199, et al., 2019 Wang et al. 2020 Lee et al.) 2020 Chen cct al. 2022 Singh et al. [021 Zhang et al. al.] 2023]. A handful of approaches, such as those proposed by [gl et al. 2018 Meng al. 2022 and Yang and Nguyen| 2021]. avoid the QMDP approximation, yet they still deviate from Bellman's 's optimality principles for POMDPs. Their learning pipelines interact directly with the environment and estimate the belief state and the returns via Monte Carlo rollouts driven by the state dynamics, rather than the belief dynamics. We discuss this difference in further detail in Appendix B.I and B.3
Our approach shares technical aspects with prior work. Particle filters have been used in several POMDP solvers [Thrun] 1999 Coquelin et al. 2008 Igi et al. 2018 Wang et al. 2020 Ma et al.
2020 Deglurkar et al.] 2023), primarily to approximate the belief state. Our method differs in its use of nested particle filtering procedure with interleaved updates that not only tracks the evolving
7

---

Averago Return
Tr(Cor\V,, (,)))
20
20
40
10
60
P30
80
LQG
0
10
20
30
40 50
10-3
10
10
10
10
Training Epoch
Temperature
Figure 1: Benchmarking P3O on a Gaacs-nn quadratic problem. Left: LQG provides tractable optimal baseline and P3O converges to a similar performance under general assumptions. Right: the influence of the temperature v the variance of the gradient, higher leads to smaller variance.
belief but also samples optimal observation action trajectories for policy learning. While Lee et al. [2020]. Wang et al.] [2020), Zhang et al. (2023) also adopt a control-as-infent -nnfeereneeee perspective, they rely the sub-optimal QMDP approximation, which undermines directed exploration.
Numerical Evaluation
For empirical validation, we compare our method, particle POMDP policy optimization (P3O). against several popular baselines. Stochastic latent actor-critic ISLAC, Lee et al. 2020) combines the QMDP approximation with the soft actor-critic [SAC, Haamnoja
2018 algorithm learn state-action value function and history-dependent policy. Deep variational reinforcement learning for POMDPs [DVRL., [gl et al. 2018) avoids QMDP and learns belief-depent delief-dependent policy and value function. We replace its original advantage actor-critic [A2C, Mnih et al. 2016) training procedure with SAC ensure parity. Finally, Dual sequential Monte Carlo [Dua Wang al. 2020) learn a state-action value function and belief-dependent
employs the QMDP approximation policy via SAC, but also includes a planning step that selects actions based on their estimated advantage. To isolate algorithmic differences in policy learning, equip all four methods with the same particle filter belief tracker, which has oracle access to the true observation models. Complete experimental details are given in Appendix D and implementations of all algorithms are available at https://gi www.uu.ocooo//eee com/ Sahe btaheel3// htep111/ppaaiiiiic cle-pomdp
Our P30-framework is agnostic to the policy representation: the policy can be explicitly history- dependent, Te (at 20:t, 0p:--),, or belief-dependent, Te. (at b.) where b; == p(st Zu:t, a0:t-1)- policies are theoretically appealing, yet training long-horizon horizon recurrent networks often suffers from vanishing gradients [Pascanu al.) 2013]. Belief-dependent policies sidestep this is- sue by acting on the instantaneous belief, which is sufficient statistic of the entire observation-action history in POMDPs [Sondik] 1971 Kaelbling et al. 1998]. However, finite-sample approximations of belief have serious drawbacks [Thrun 1999] the same belief state can be approximated by different sets of particles, and neural networks usually cannot respect this invariance. Although this mismatch can introduce spurious effects, particle-based belief inputs have been empirically effective [Ig et al. 2018 consequently, we explore both variants of the policy.
Tractable LQG:
We validate our approach against a closed-fore form linear-quadrati Gaussian (LQG) benchmark. this setting. belief propagation is computed exactly using a Kalman filter, and the optimal policy is an affine function of the Gaussian belief mean [Aoki, 1967]. We test P3O two-dimensional system with linear-Gaussian transition and observation models and quadratic reward, where IQQ provides reference for the achievable cumulative reward. Figure [eture[[compare this bascline with P3O, which operates without exploiting any analytic structure. As training progresses, P3O converges toward the optimal performance of LQG. We also use this problem to illustrate the role of the temperature parameter in controlling the variance the policy gradient in Figure Larger values of 17 reduce variance but introduce bias through the risk-sensitive objective, while + O recovers the unbiased REINFORCE gradient at the cost of high variance.
Here we evaluate on two classical control tasks: stochastic, partially-observe variants the pendulum and cart-pole swing-up tasks. both settings, the agent cannot observe velocity components of the state. These tasks are representative of standard control problems, where active information gathering not critical. Hence, it expected that QMDP approximation should not be detrimental to learning. This intuition is confirmed by the results in Figure 2 The differences
Control Tasks:

---

Light Dark
Trangulation
1000
100
Average Retum
2000
-200
2001
Interacttons: x10*
Interactions
100
P50 (history
PSO (belief impuls):
SLAC:
DVRL:
Dual-SMC
Figure 2: Experiment results on various benchmarks. We report the average return using 1024 trajectory rollouts and plot the mean and standard error over 10 training seeds. P3O and DVRL consistently outperform the QMDP-based solvers, SLAC and DualSMC, which especially struggle in the light-dark and triangulation tasks that require deliberate exploration.
2.0
100
80
Egeevalue
1.0
40
0.0
100
2000
Time Step
Figure 3: Example trajectories from the light-dark (left) and triangulation (middle and right) tasks. In the light-dark environment, the agent learns to steer towards low-uncertainty regions to localize itself, then moves to the target. We visualize the agent's belief at every time step. In the triangulation task, the agent executes a specialized maneuver to estimate its position using only heading measurements. We plot the largest eigenvalue of the agent's belief covariance over time. The red dashed lines represent the shortest path the agent could follow no information gathering was required.
convergence speed reflect how frequently gradient steps are performed. P3O and SLAC, which use recurrent policies full histories, require complete trajectory rollouts before each update, unlike DVRL., DualSMC, and P30 with a belief-dependent policy. While belief-dependent policies can learn faster, their performance
be constrained by the quality of the belief representation, and (bootstrap) particle filters are known to produce degenerate approximations in certain cases Bickel This trade-off is evident the more challenging cart-pole environment.
al. 2008
Light-Dark: Next, we consider a continuous light-dark navigation task (Platt et al. 2010 Van Den Berg al. 2012), where agent must reach a goal state in wwo-dimensional plane while relying on observations with location-dependent noise. Outside a light region, the observation noise very large, whereas near the light the agent can localize accurately. The optimal strategy requires first navigating towards the light to reduce uncertainty before committing to the goal. QMDP- baselines, DualSMC and SLAC, are doubly handicapped in this task. Their disregard for information- gathering results in persistently high-variance observations. This in turn leads to high-variance bootstrap targets their Q-learning procedures, inflating gradient noise and stalling learning, as Figure In contrast, methods that explicitly reason over the belief space, such as P3O, learn to gather information, then exploit, achieving better performance. Figure 3 depicts an example trajectory generated by a policy learned with P3O. The policy intentionally steers away from the target to reduce uncertainty before moving towards the goal state.
Triangulation: our final experiment, we consider an active triangulation task, in which the agent must reach the origin in two-dimensional plane relying solely heading measurements [Tse and Bhal-Sh 1975). This task is particularly difficult, as accurate distance estimation to the target requires the agent to perform special maneuver collect heading measurements from varied positions in the state space. As with the previous experiment, this environment has significant noise both in transitions and observations. P3O successfully learns to solve this task, see Figure 2 A representative trajectory from policy learned with P3O is shown in Figure exhibiting characteristic S-shape oscillation around the line of sight.

---

Cust-Pole 256)
Cart-Pole (M - 32)
Light Dark LV 200)
Light Dark (2d 32)
-100
-100
-10
Avenge Retum
2100
2000
-20
-20
3000
-00
128
-40
11
-01
32
16
5000
-000
-50
0.0
Inseractions X10
x10"
Interactions
x010
Figure 4: Influence of the number of particles in the Feynman Kec filter (N) and belief filter (M). Increasing N and M improves the approximation of the target distribution Vr and the policy gradient, resulting in better learning performance. Plots depict the mean and standard error over 10 seeds.
Finally, we conduct ablation studies to disentangle the effect of P3O's hyper- parameters. In Figure 4 we vary the number of particles used the Keynman-Kac and be-
Ablation Studies:
Pole
Light-Dak
100
Avenge Setura
-200
lief filters. Increasing the particle count yields more accurate approximations of the target dis- tribution , and accelerates learning, though with diminishing returns. In Figure 5 we exam- ine the impact of the temperature parameter v/. Proper tuning of 17 is essential: it controls gra- dient variance and determines the effectiveness of policy updates, although too high values of v) can exacerbate path degeneracy in SMC.
-300
100
-000
0.0G
-60
5000
-80
0.0
0.25
0.5
0.75
1.0
1.0
2.0
4.0
5.0
Internations
x
Interaations X100
Figure 5: The effect of the temperature v) on policy learning. Lower values of lead to high-variance policy gradients and slower convergence.
6 Discussion
Optimal decision making in partially observable environments presents fundamental challenges, as agents have to reason in the space of histories and beliefs. While many existing methods simplify this complexity using approximations such as QMDP, they typically sacrifice the ability to actively gather information and perform directed exploration. optimization (P30), a principled policy learning algorithm that operates directly on the belief space. By formulating the POMDP learning problem within a Feynman-Kac framework, our approach integrates belief tracking with reward-guided trajectory sampling, enabling principled and efficient policy optimization under partial observability in continuous domains.
this work, we introduced particle POMDP policy
Our experiments, while focused on relatively low- www-dimensional tasks, highlight a fundamental limi- tation of several sophisticated algorithms that fail to learn in the light-dark and triangulation tasks. These tasks expose a key distinction between dual-effect systems, where actions influence both state transitions and the agent's uncertainty, and neutral systems, where actions do not affect state estimation [Feldbaum] 1963, Bar-Shalom and Tse. 1974]. In dual-effect settings, P3O demonstrates clear advantage by explicitly incorporating exploration. In contrast, in neutral systems, QMDP- algorithms like SLAC and DualSMC are likely to scale better.
Our method is not without limitations. First, the bootstrap particle filter used in our implementation may not scale well in high- sith--ddmnsiinnll settings due to increased discrepancy between the proposal and target distributions. Using more sophisticated proposals that take the likelihood into account (Doucet et al., 2000] or learning better proposals [Guarniero et al. 2017 Naesseth et al. 2018) may be necessary to ensure satisfactory performance for high- nigh-dimensional problems. Second, the particle belief-tracker (though not the policy learning loop) requires access to the oracle of the observation likelihood. Incorporating a learning- -based state estimator that can operate from raw interaction alone is an important direction for future work. Third, quantifying the bias in the score estimate in Algorithm 2 is a challenging problem that we do not address. While standard SMC methods have well-understood bias properties (O(1/N) with N particles for additive functionals (Del Moral 2004)), our nested structure creates more complex bias behavior. Lastly, tuning the risk parameter 1j is an open problem. We refer the reader to Watson and Peters 2022 for potential calibration strategies.
10

---

7
Contribution Statement
The original idea and motivation due to HA. The methodology, proofs, and experiments were developed jointy by HA and SI, with equal contribution. SI and HA drafted the manuscript. SS provided regular feedback through weekly meetings, contributed valuable insights the setting and background literature, and proofread the manuscript.
Acknowledgments
Part of this work was conducted while HA was a postdoctoral researcher Aalto University and the Finnish Center for Artificial Intelligence (FCAI). HA also acknowledges funding from the Amsterdam Machine Learning Lab (AMLAB) the University of Amsterdam. SI gratefully acknowledges funding from the Research Council of Finland. The authors thank Adrien Corenflos for proofreading the manuscript and providing valuable comments.
References
M. Aoki. Optimization of Stochastic Systems: Topics in Discrete-tine Systems. Mathematics in science and engineering. Academic Press, 1967.
K. Åström. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174-20) 205, 1965.
H. Attias. Planning by probabilistic inference. In International Workshop Artificial Intelligence and Statistics, 2003.
Y. Bar-Shalom and Tse. Dual effect, certainty equivalence, and separation in stochastic control. IEEE Transactions on Automatic Control, 19(5):494 500, 1974.
Bickel, B. Li, and T. Bengtsson. Sharp failure rates for the bootstrap particle filter in high dimensions. In Pushing the limits contemporary statistics: Contributions in honor of Jayanta K. Ghosh, volume 3, pages 318-330. Institute of Mathematical Statistics, 2008.
D. M. Blei, A. Kucukelbir, and D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859--85) 877, 2017.
C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton. A survey of Monte Carlo tree search methods. IEEE Transactions Computational Intelligence and Al in Games, 4(1):144 2012.
P. Bunch and S. Godsill. Improved particle approximations to the joint smoothing distribution using Markov chain Monte Carlo. IEEE Transactions on Signal Processing, 61(4)9956-963, 2013.
O. Cappé, E. Moulines, and T. Rydén. Inference in Hidden Markov Models. Springer Series in Statistics. Springer, 2005.
A. Cassandra, M. L. Littman, and L. Zhang. Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In Conference Uncertainty in Artificial Intelligence, page 54-61. Morgan Kaufmann Publishers Inc., 1997.
X. Chen, M. Mu, P. Luo. Li, and J, Chen. Flow-based recurrent belief state learning for POMDPs. In International Conference on Machine Learning, 2022.
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
N. Chopin and O. Papaspiliopoulos. An Introduction to Sequential Monte Carlo. Springer Cham, 2020.
N. Chopin, P. E. Jacob, and O. Papaspiliopoulos. SMC²: An efficient algorithm for sequential analysis of state space models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 75(3):397-426, 2013.

---

P.-A. Coquelin, R. Deguest, and R. Munos. Particle filter-based policy gradient in POMDPs. In Advances in Neural Information Processing Systems, 2008.
A. Corenflos. Computationally Efficient Statistical Inference in Markovian Models, PhD thesis, Aalto University, 2024.
A. Corenflos. J. Thornoon, G. Deligiannidis, and A. Doucet. Differentiable particle filtering via entropy-regulariize optimal transport. In International Conference on Machine Learning, 2021.
D. Crisan and J. Míguez. Nested particle filters for online parameter estimation in discrete-time state-space Markov models. Bernoulli, 24(4a):3039-3085 2018
H.-D. Dau and N. Chopin. On backward smoothing algorithms. The Annals of Statistics, 51(5):2145 -2169, 2023.
P. Dayan and G. E. Hinton. Using expecaaionn-maxiizzzioo for reinforcement learning. Neural Computation, 9(2):271-278, 1997.
Deglurkar, M. H. Lim, J. Tucker, N. Sunberg, A. Faust, and C. Tomlin. Compositional learning- based planning for vision POMDPs. In Learning for Dynamics and Control Conference, pages 469-482. PMLR, 2023.
Del Moral. Feynman-Kan Kac Formulae: Genealogical and Interacting Particle Systems with Applica- fions. Springer, 2004.
Del Moral and L. Miclo. Genealogies and increasing propagation of chaos for Feynman-Kac and genetic models. The Annals of Applied Probability, ((4):1166-119)) 2001.
A. Doucet, S. Godsill, and C. Andrieu. On sequential Monte Carlo sampling methods for Bayesian filtering Statistics and Computing, 10:197-208, 2000.
Elman. Finding structure time. Cognitive Science, 14(2):179-211, 1990.
A. Feldbaum. Dual control theory problems. IFAC Proceedings Volumes, 1()):411-500 1963.
C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax A differentiable physics engine for large scale rigid body simulation, 2021. URL http: wwt.ta..ooooooooo//// comgg///oo/.ccooooggooo//oo google/ brax
S. Godsill, A. Doucet, and M. West. Monte Carlo smoothing for nonlinear time series. Journal of the American Statistical Association, 156461) 2004.
N. Gordon, D. Salmond, and F. Smith. Novel approach to DeaiinearnoonGauusian Bayesian state estimation. In IEE proceedings F (Radar and Signal Processing), volume 140, pages 107-113, 1993.
Guarniero, M. Johansen, and Lee. The iterated auxiliary particle filter. Journal of the American Statistical Association, 11(6520):633-1661) 2017.
T. Haamoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. 2018.
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint ArXivv/912.011 2019a.
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamies for planning from pixels. In International Conference on Machine Learning, 2019b.
D. Han, K. Doya, and Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint arXiv: 10703, 2019.
S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. CleanRI: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18. 2022.
12

---

M. Igl, L. Zintgraf, T. A. Le, Wood, and S. Whiteson. Deep variational reinforcement learning for POMDPs. In International Conference on Machine Learning, 2018
V. Indelman, L. Carlone, and F. Dellaert. Planning in the continuous domain: A generalized belief space approach for autonomous navigation in unknown environments. The International Journal of Robotics Research, 34(7):849-882, 2015.
Iqbal, H. Abdulsamad, Pérez- Vieite,, S. Särkkä, and A. Corenflos. Recursive nested filtering for efficient amortized Bayesian experimental design. In NeurIPS Workshop on Bayesian Decision- making and Uncertainty, 2024.
L. P. Kaclbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence. 101(1):99-134, 1998.
N. Kantas, A. Doucet, S. S. Singh, J. Maciejowski, and N. Chopin. On particle methods for parameter estimation in state-space models. Statistical Science, 30(3), 2015.
J. Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Sstatistical Mechanics: Theory and Experiment, 2005(1 1):P11011, 2005.
L. Koesis and C. Szepesvári. Bandit based Monte Carlo planning. In European conference on machine learning, pages 282-293. Springer, 2006.
H. Kurniawati and V. Yadav. An online POMDP solver for uncertainty planning dynamic environ- ment. In Robotics Research: The 16th International Symposium ISRR, pages 61 1-629, Springer, 2016.
A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances Neural Information Processing Systems, 2020.
Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for attention-based permutation-invariann neural networks. In International Conference Machine Learning, 2019.
S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv preprint arXiv: 00909, 2018
W. Li and E. Todorov. Iterative linearization methods for approximately optimal control and estimation of non- non-linear stochastic system. International Journal of Control, 80(9):1439-144 1453, 2007.
M. H. Lim, C. J. Tomlin, and Z. N. Sunberg. Voronoi progressive widening: Efficient online solvers for continuous state, action, and observation POMDPs. In Conference on Decision and Control, pages 4490-4500. IEEE, 2021.
F. Lindsten and T. B. Schön. Backward simulation methods for Monte Carlo statistical inference. Foundations and Trends Machine Learning, 6(1):1-143, 2013.
M. L. Littman, A. R. Cassandra, and L. P. Kaelbling. Learning policies for partially observable environments: Scaling up. In International Conference Machine Learning, pages 362-370. Elsevier, 1995.
X. Ma, P. Karkus, D. Hsu, W. Lee, and N. Ye. Discriminative particle filter reinforcement learning for complex partial observations. arXiv preprint arXiv:2002.09884, 2020.
I. Mareus, E. Fernándezz-Guuchrrandd D. Hernández- Heaenndezz-eeadddz... Coraluppi. and Fard. Risk sensitive Markov decision processes. In Systems and Control in the Twenty-Fine First Century, pages 263-279. Birkhäuser Boston, 1997.
I. Meng, R. Gorbet, and D. Kulić. Memory- based deep reinforcement learning for POMDPs. In 202/ IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619-5626. IEEE, 2021.
13

---

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016.
S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih. Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
C. Naesseth, S. Linderman, R. Ranganath, and D. Blei. Variational sequential Monte Carlo. In International Conference Artificial Intelligence and Statistics, 2018.
R. Pascanu, T. Mikolov, and Bengio. On the difficulty of training recurrent neural networks. In International Conference Machine Learning, 2013.
J. Pineau, G. Gordon, S. Thrun, et al. Point-based value iteration: An anytime algorithm for POMDPs. In IJCAI, volume 3, pages 1025-1032, 2003.
R. Platt, R. Tedrake, L. Kaelbling, and T. Lozano-Perez. Belief space planning assuming maximum likelihood observations. In Robotics: Science and Systems, volume 2, 2010.
J. M. Porta, M. T. Spaan, and N. Vlassis. Robot planning in partially observable continuous domains. In Roboties: Science and Systems. Massachusetts Institute of Technology, 2005.
P. Poupart, N. Vlassis, J. Hoey, and K. Regan. An analytic solution to discrete Bayesian reinforcement learning. International Conference on Machine Learning, pages 697-704, 2006.
K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Robotics: Science and Systems, 2012.
K. C. Rawlik. On probabilistic Inference Approaches Stochastic Optimal Control. PhD thesis, University of Edinburgh, 2013.
S. Ross, J. Pineau, S. Paquet, and B. Chaib-Draa Online planning algorithms for POMDPs. Journal of Artificial Intelligence Research, 32:663-704, 2008.
Särkkä and L. Svensson. Bayesian Filtering and Smoothing. Cambridge University Press, 2nd edition, 2023.
D. Silver and J. Veness. Monte Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems, volume 23, 2010.
G. Singh, S. Peri, J. Kim, H. Kim, and S. Ahn. Structured world belief for reinforcement learning in POMDP. In International Conference on Machine Learning, 2021.
A. Somani, Ye, Hsu, and W. S. Lee. DESPOT: Online POMDP planning with regularization. Advances in Neural Information Processing Systems, 26, 2013.
E. J. Sondik. The Optimal Control of Partially Observable Markov Processes. Stanford University, 1971.
R. F. Stengel. Optimal Control and Estimation. Courier Corporation, 1994.
Z. Sunberg and M. Kochenderfer. Online algorithms for POMDPs with continuous state, action, and observation spaces. In International Conference on Automated Planning and Scheduling, volume 28, pages
2018
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018.
S. Thrun. Monte Carlo POMDPs. In Advances in Neural Information Processing Systems, volume 12, 1999.
Thrun, W. Burgard, and D. Fox. Probabilistic Robotics. MIT Press, 2005.
E. Todorov. General duality between optimal control and estimation. In Conference on Decision and Control, pages 4286-4292. IEEE, 2008.
14

---

M. Toussaint and A. Storkey. Probabilistic inference for solving discrete and continuous state Markov decision processes. In International Conference on Machine Learning, pages 945-952, 2006.
E. Tse and Y. Bar-Shalom. Adaptive dual control for stochastic nonlinear systems with free end-time. IEEE Transactions Automatic Control, 20(5):70-677,, 1975.
J. Van Den Berg, S. Patil, and R. Alterovitz. Efficient approximate value iteration for continuous Gaussian POMDPs. In Conference on Artificial Intelligence, volume 26, pages 1832-1838, 2012.
Y. Wang, B. Liu, J. Wu, Y. Zhu, S. S. Du, L.. Fei-Fei, and J. B. Tenenbaum. DualSMC: Tunneling differentiable filtering and planning under continuous POMDPs. In Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, 2020.
J. Watson and J. Peters. Inferring smooth control: Monte Carlo posterior policy iteration with Gaussian processes, 2022.
Watson, H. Abdulsamad, and Peters. Stochastic optimal control as approximate input inference. In Conference Robot Learning, pages 697-716. PMLR, 2020.
D. Wierstra, A. Foerster, J. Peters, and J. Schmidhuber. Solving deep memory POMDPs with recurrent policy gradients. In International Conference on Artificial Neural Networks, pages 697-706. Springer, 2007.
R. Williams. Simple statistical gradient-followin algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, 1992.
7. Yang and H. Nguyen. Recurrent off-policy baselines for memory-based continuous control. arXiv preprint arXiv:2110 12.1100.220 12628, 2021.
N. Ye, A. Somani, Hsu, and W. S. Lee. DESPOT: Online POMDP planning with regularization. Journal of Artificial Intelligence Research, 58:231-266, 2017.
D. Zhang, A. Courville, Y. Bengio, Q. Zheng, A. Zhang, and R. T. Q. Chen. Latent state marginaliza- tion as a low-cost approach for improving exploration. In International Conference on Learning Representations, 2023.
M. Zhang, S. Vikram, Smith, Abbeel, M. Johnson, and Levine. SOLAR:
Deep structured representations for model- based reinforcement learning. In International Conference on Machine Learning, pages
2019.
B. D. Ziebart, Maas, J. Bagnell, and K. Dey. Maximum entropy inverse reinforcement learning. In Conference on Artificial Intelligence (AAAI), 2008
15

---

A Proofs and Derivations
A.1 Proof of Proposition 1
Proof. We begin with the original definition of the objective function J (():
T Σ
I(6) P (ZO:T, 40:7-1, 80:T)
(s:
which defines the expected cumulative reward over fixed horizon, where the expectation is over full trajectories (Zu.T, ag:T-1, So:T) given a policy Tg. By linearity of expectation, we can interchange the expectation and the sum,
T
I(0)
= Σ 1=1
E Po (ZO,T, a0,r-1, sO.T) Re (8 at- 11-1)]
Now, consider the term E (p(()) [Re(se, a)] The reward RG depends only on the state and the
previous action 0ę-1. We can thus marginalize out variables corresponding to future times >> and past states 80:--1:
[(0)
B p (=0 co:
a0:--1 1, st) Re (sc, a(1)
> E p(se 20:t, 40: (2 aga-
Re(ss, a-11):
where p(st1 Ru:,, 00:--1) is the belief state and p& (0:1, 40:-11) is the marginal probability of the history. Now, we can apply the law of total expectation to decouple the expectation over the history (zost 00.-1) from the expectation over the state 8, leading to:
((Φ)= = E Pp (ZO:T, 0o:T-1)
Bp(ss zoie op:- Re [(((st)a-ii)]
The outer expectation with respect to (=(=0 40:2-1) 1) averages over all possible histories of obser- vations and actions generated under the policy т&. The inner expectation with respect to p(st averages the reward R, over the possible states 8,, given specific history (Zo:t: 10 This ob- jective is consistent with the definition of POMDP objectives according to |Kelbling al. 1998].
To simplify the expression, we define an auxiliary function l, representing the expected immediate reward time given the history up to that point:
Ce(20:t -0:--1 Pp( 88 #0:t; 10 Re Ree(st, (((8),)411))
This definition effectively integrates out the dependency on the latent state 8, by averaging according to the belief state distribution. Substituting this definition back yields
((O) pe (Zo,T, ao:r-1)
Σ l Ea 00:t,)
with po(20:7, -1 being the belief-space generative process
T-1
Po(2orT, (o:T-1) = 2(2))
II
p(z-11 20:1, aoa) Ta(ar 20:1; a0:-11); 1=0
where p(zu) = E p((so) [0(20 su]] and
g( s441) f(se41 st, ar) p(se zua, a0a-1) ds; d$ p(z+11 Zuss aw:t) //
required.
16

---

A.2 Proof of Proposition 2
Proof. We begin with the definition of the risk-sensitive belief-space objective
(() = log PO(O):))
Since > O is just a scalar, we focus on the gradient of the log-marginal likelihood log P& (O1r)
V (Ø)
Va Vologpo(O1or)
VoPo(O1T))
Po(OOiT)
PO() Vo f po( (0:--1) p(O1: 20:r, a0:r-1) džp:r da
1
Under suitable regularity conditions, et al. 2020] for details on when this is admissible)
interchange differentiation and integration (see Mohamed
/ V (o F0:T, (0:T-1) dag:r dao:T-1-
V (b) & PO(OI.T)
90:T-1)
Next, using the log-ratio trick,
V Volo f. the expression becomes:
/ Po 10.T (.)) Po dzo:T dag:T-1
V (o) x (O()):T)
Now, let's recall the definition of polzaor 00:T-1) from 2
T-1
Po Po(zorr, -0)
II
p(z1-1 20:t, an:) mo(as) 20st: a0:--1);
and the definition of y from 5
(F N0:T Φ) (0: :0
00:T-1, O1r)) (o(Or:))
Plugging them back into the gradient expression, we get:
Vg (o) x ETT
Vo log
p(z))
II
p(#+1 Rous, aoa) (T(as Zoser (on--1)
E=U
T-1
T-1
= ETA
V log
II
(as zoits a0:
Vo log
p(20)
II
p(z+41 zore, ao:e)
t=
= EAT
Σ Vo log (a(l 20:t, 10)
as required.
A.3 Derivation of Equation
We omit the dependence ó. and start from the definition of y in (5)
(r 1 1) = p(zO.T, d0:7 O1.)
T-1
= p(zo Owr)
II
p(az
2ost; a0:-11 O1:7) (1100:(p(p:()) p(=++1 zo:t, ag:t Owr))
t=
= =((20 O
p(2
20xt Pp:t O++1:T) P(() 20:,, 4p:1- O
0

---

For the final step, we used the fact that 24+1 and ds are conditionally independent of Ora given (Zoct: 00:--1). Using Bayes' rule, we have
p(0++1.7) 20:e: do:t)
p(ar] Fo:z, 403-1, O++1:)) (a F0:2, dg:-11
ppO (1:7 Foct, 40:t-1
where π (ag) 20:e: 10:t-1) is the prior (unconditioned) policy, Putting everything together, get
p(O 20:2, ao::) p(O++1:T)
(r(=0:T,00:T]) x p(z O1:)) II
p(z++1 2oxt, 00:t; 12+1:T)
E0:t; 00:t-1)
t=0
as required.
B Decision-Making in POMDPs
B.1 Bellman's Optimality Equations
In this section, we briefly review the fundamentals of optimal decision-making in partially observable Markov decision processes. We use [zoxe, 00:r-1] to denote the history up to time as a stacked vector. The solution to POMDP can be formulated in terms of optimal value functions, V. and Qt, over beliefs and actions ((=0:2, (0:-11);;;) 0z) as follows (Thrun al. 2005 Chapter 15]:
Vi([201,000a arEA
max 2c((201e00aa-11.00
Qi(lzon, a0u- = /
p(ze+1 E0ce, ao:)
(10)
[e+1l200e11100 an:) Ve+1 ([z0e+1, dozl) d2
where (1+1 (0.z) = -I. p( 20341, (o:t) | at) is the expected reward defined in
Proposition I and p(s++1 P0:t+1; an:e) is the updated belief after following action at and observing 2+41, retrieved according to Bayes filter (Särkkä and Svensson 2023
p(s++1 20.++1, (n:t) x g( (6+1) f s(++11 8t, at) p(#t Eq:1, ao:--1) dst-
Unlike in fully observable Markov decision processes (MDPs), where value functions are defined over ditte--imensional spaces, POMDP value functions Vi and Qr are defined over the infinite- dimensional space of histories. This makes solving Bellman's 's optimality equations significantly more challenging. Crucially, the value of information gathering is implicitly encoded into the value function through the predictive distribution over future observations
/ʃ g( (841) (8 8t, at) p(se 20:t, a0: ds; ds
p(2e+ 20:t, ao:t)
so that the expected utility of an action depends not only on immediate rewards but also its effect on future belief refinement. This way, an agent can account for the long-term value of observations that improve its understanding of the latent state and thus influence subsequent decisions.
B.2 The QMDP Approximation
To simplify Bellman's 's optimality equations POMDPs. Littman al. [9955 introduced what is known as the QMDP approximation, which replaces (10) with the approximation
Qe a0.-11), a.) = E EEp(s stl Zois ao:t- (Q(((()
= If p(s; zoit: 1 ( st: at) R44 (541,)) Ve-1(stt1) t#ssh ds
where Qr(s,, a.) is the state-space Q-function of the corresponding fully observable MDP. While this approach is algorithmically simpler, it relies on a strong, crude assumption: action values are estimated as if the latent state will be perfectly observable from the next time step onwards. This
18

---

effectively ignores the need for, and potential value of, future information gathering, as it fails to account for how future observations 2t+1 might refine the agent's knowledge of the latent state and improve decision-making. Consequently, this method decouples the procedures of probabilistic inference and decision-making, which are inherently intertwined in POMDPs, making any reduction in belief uncertainty merely incidental, rather than directed outcome of the policy. For this reason, methods that use QMDP fail to address the core POMDP challenge of actively balancing the need to reduce state uncertainty (exploration) against maximizing expected extrinsic rewards based on the current state of belief (exploitation). Since optimal POMDP policies must achieve this balance,
policies derived via the QMDP approximation are inherently sub-optimal [Ross et al. 2008 State-
of-the-art work on POMDPs such as [Hafner et al. 2019b a either explicitly or implicitly assume the QMDP setting.
Lee et al. 2020 Wang et al. 2020
B.3
The Belief-Space Generative Process
The Bellman equations in (9) and (10) are computationally tractable only for a narrow class of continuous POMDPs. Consequently, state-of-the-ar methods typically rely on sampling- -based approximations. A critical distinction arises between reinforcement learning in MDPs and POMDPs, namely in the generative process used to obtain Monte Carlo samples.
In an MDP, Bellman's optimality principle defines the value of a state, V (8c+1), as the ex- pected sum of future rewards. This expectation is taken over the known statetrraniiiinn dynamics [st+1 st, az). Hence, MDP algorithms interact directly with the transition dynamics to generate the state-space Monte Carlo tuples of the form (st, dg. St+1, r++1), where r++1 Re+1 (8:41, at st). These samples are then used to approximate the expectation underpinning the state-value.
In contrast, optimal decision-making in POMDPs must operate in the space of histories beliefs. According to Bellman's principle in POMDPs (10), the value function, V4+1 ([zua+1, aq:l). is eval- uated in expectation over the observation predictive distribution p(z++1 Zoot; (0:r). Additionally, the corresponding reward signal is the expected reward, 4+1 1 (Z0++1, 4gx),, given the agent's belief. This implies that theoretically sound sampling-based approach must simulate from this predictive process and collect the tuples of the form ([=ou, -1), (22 #411, (411) to arrive at valid Monte Carlo estimations of the corresponding value functions.
However, many state-of-the-art reinforcement learning methods for POMDPs appear to deviate from this simulation process (Igl et al.) 2018 Meng et al.] 2021 Yang and Nguyen 2021 and instead optimize the state-space objective via a state-space sampling process as described in Sec- tion 2 Instead of simulating from p(z1+1 20:t, aozt), these methods sample observations by inter- acting with the transition dynamics f(+,+1 8,, 0₂) and effectively drawing from the conditional g(2++1 (++1),
| where E1+1 is the true, latent state. Furthermore, they use the raw reward signal, I++1 = Ret1 (8 0r, (t) associated with that specific latent state transition, rather than the expected reward, 444 l (#0:++,, do:z), associated with agent's belief. This leads to learning procedure that evolves with state-space dynamics, rather than belief-space dynamics.
C Backward Sampling
C.1 Overview
Backward sampling is an algorithm used to generate more diverse samples from the smoothing distribution [Godsill et al., 2004 Lindsten and Schön 2013). While the smoothing distribution can be represented by the ancestral lineage of the trajectories generated by the particle filter, backward sampling prescribes an additional step: compute the likelihood that each particle from the previous time step could have led to given current particle, and sample an ancestor from this distribution. As result, backward sampling can generate trajectories that were not explicitly formed during the forward filtering pass.
We use the notation for Feynman Kac models from Section 3.1 and follow the presentation in Corenflos [202]] Chapter 3]. The formula for backward sampling is based on the trivial identity
Qr(wo m(41r) xc (x = (r(o)))) Qu (Zo:z)
Q (2ont)
19

---

V.V After performing particle filtering, we will have weighted particles { wt Jn=1 that form a discrete approximation to the filtering distribution Qu (Z):)) for all E {O...., ...,, T}. We can use these to form an approximation to (For 1e+1:)) as
Qr(aaa (+ITT) R
E nml
wy W/d (Io:t);
where the smoothing weights Wf are given by
N wi, wi = WT"
W Qr Qr(():)
Σ W?"
(11)
Thus, we need compute the ratios Q7 (a T++1-r Q (x in addition the filtering weights wT. The full backward sampling procedure is as follows:
Sample a particle at the final time step: ST Categorical {wi )==1) and ar = Sr
2.
Jn=1) and set Then for E (T- - 1,....00 0}, sample an ancestor S ~ Calegorical (N, {w]])==)
PEr = (xt IIt1::), with the smoothing weights ri computed using
C.2 Details for the Nested SMC Algorithm
We now discuss how to compute the smoothing weights for our nested SMC algorithm (Algorithm) 2
Observe that "particle" of the outer particle filter is the set {2z, at-1: B.- 8t for every t E {i..... }. When doing backward sampling, at time t, we will have already sampled
2.:M4 } θ defined
trajectory Q++1:7, and we need to sample an ancestor 0b; for some n E {l...., ........V. .V. Denoting the
distribution of O%: by V.M. the fraction we need to sample an ancestor for time is given by
/ (O O++1T) M.M (0%:)
V.ll (=D:, C0:-11 P0:-11 *0:
Ba1:M sn1-M
1:M 2++1:T GET--, B1:M - 8+1.7)
M (zi
oi BM1-M snl:M)
T-1
X
ne (a.
Fo: 40: -1: 1 zt l:s: Ot:s-1)
n 1t
(12)
Probability of sampling future actions
Transition probability for the state
M
II
B
f 8t .a))
nB,"
m=1
Probability sampling TT"
Note that we have only written down the terms that are unique to each ancestor O as the terms that are common for all n will be normalized away. By using (12 and the stored filtering weights, we can now perform backward sampling for Algorithm 2 as described in Appendix C.I
The algorithm developed thus far, while correct, can be improved in two major ways. First, when we compute the probability of transitioning from belief state {s!mm, wnti amam) J.M I==1 to a belief state m=1- the expression in (12) looks at the pairwise alignment of individual particles (i.e.,
{s7t 1/M) M
the probability of sampling s from BB"", This is likely to yield very low probabilities for all n except the previously traced ancestor index A:-1, because even if the particle sets represent similar posterior distributions, their pairwise alignment can be arbitrarily poor. To fix this, we use solution proposed in Iqbal et al. (2024) and integrate over the the resampling indices B B.EM, yielding the following transition probability for belief states
/
M
I:M p(s p(t+1
PI: aa)
II f(s st nB," ..) d (BI:M)
m=1
II
f(s
I St sp ai) duw. (B!:M)
m=1
M Σ
nk f(s 1 an)
II
we
,a)
(13)
m=1
k=1
20

---

In the second line, we have used the fact that the resampling indices are sampled independently from a categorical distribution. We replace the second line (22 with (3)
The second improvement we make to the algorithm concerns its computational complexity. Com- single time step has complexity (MM2T), leading to the full backward sampling algorithm having complexity ON (M2 to sim- ulate single trajectory. We improve this ((M²72) by using modified version backward sampling from Bunch and Godsill [2013]. in which the ratio in (12) needs to be computed for only
puting the smoothing weights with (12) and 13 for single
two, rather than N. possible ancestors. For details, we refer to Dau and Chopin 2023
D Evaluation Details
D.1 Architectures
gated recurrent unit [GRU, Cho et al. [014], a type of recurrent neural network [RNN, Elman 1990], to encode the history into a fixed-size vector (Table ( similar to Yang and Nguyen [2021] This embedding is passed to multi-
For distory-dependen policies, used in both P3O and SLAC,
layer perceptron (MLP) to get the mean m and standard deviation a of the action distribution (Table2). Actions are then sampled from diagonal Gaussian distribution, ag (mm, diag(o?)).
Table I: The GRU encoder architecture used for P3O and SLAC.
Layer
Size
Activation
Input
dim Z x AA)
Dense
256
ReLU
LayerNorm
Dense
256
ReLU
LayerNorm
Dense
128
LayerNorm
GRU
128
GRU
128
Dense
128
Table 2: The MLP decoder architecture used for the policies of all algorithms except P3O. For P3O, the noise is independent of the input, and hence the output dense layer has size dim(.
Layer
Size
Activation
Input
Variable
Dense 256
ReLU
Dense 256
ReLU
Dense
dim
The policy in DVRL uses the belief state, which is weighted particle set, input. We concatenate the particles and their weights, flatten them, and use a dense layer to encode the belief state as a vector of size 32. This encoded vector is then passed to the decoder in Table to generate the mean and standard deviation of the Gaussian action distribution. The process for DualSMC similar, except we do not include the weights because the particles are resampled before being passed the policy, and we append the mean of the particles to the belief state before flattening [Wang et al. 2020).
For the belief-dependent policy for P3O, use the set transformer architecture from Lee al. (019) to encode the belief state. The set transformer ensures that the network output is invariant to permutations of the particles in the belief state. The encoded belief state is passed to the same MLP decoder in Table 2
Finally, SLAC, DualSMC, and DVRL also have critic networks. The critic networks for SLAC and DualSMC have the same architecture, given Table 3 The critic networks for DVRL use the belief state input (encoded in the same way for the policy), and the architecture is given in Table
21

---

Table 3: Critic network architecture for SLAC and DualSMC. The input is state, action, and time. The input state randomly sampled from the belief distribution.
Layer
Size
Activation
Input
dim(S A)
+1
Dense
256
ReLU
Dense
256
ReLU
Dense
Table 4: Critic network architecture for DVRL. The input is the encoded belief state, action, and time,
Layer
Size
Activation
Input
32 (AA +
Dense
256
ReLUU ReLU
Dense
256
Dense
D.2 Training Details
SLAC, DualSMC, and DVRL all use soft actor-critic [SAC. Haarnoja et al., 2018 as the base RL algorithm. Our SAC implementation and the hyperparameters chosen for training based on the implementations CleanRL (Huang et al.) 2022] and Brax [Freeman et al. 2021 Please see the code for full hyperparameter specification for all reference algorithms.
For all algorithms considered (including P3O), we use particle filter with 32 particles to track the belief state. Additionally, for P3O, the outer particle filter has 128 particles. Unlike in Algorithm [ which would use all 128 trajectories to perform one gradient step, we use mini-batches to perform multiple gradient updates per nested SMC run to improve sample efficiency.
For P3O, action in adjacent time steps. We found that this was necessary to ensure that the trajectories sampled using Algorithm 2 are smooth enough for the GRU networks learn.
additional slew rate penalty in the reward function that penalizes large changes in
All our experiments were carried out on an NVIDIA A100 80GB GPU. For complete details, including the scripts used for the experiments, see the source code at https: com/ Sahel13/particle-pa
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims made are backed up by both the theory develop and the numerical experiments we report.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of method in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification: For the propositions and remark in the paper, we provide full derivations with the necessary assumptions in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss the experimental details in Appendix D addition, the code for our methods and reference methods, including the scripts to produce the plots reported in the paper, is provided as part of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code part of the supplementary material. It is well- documented and contains all necessary instructions for reproducing the results the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
Poolicc/uuides/Coeeetuliiis
catpssioooolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuaaaeeeae
wtweeee//ooSSSeissss.ee.. www.ssssiicogoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include most of these details in Appendix D and complete specifications are available in the code attached as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Figure where we compare the performance of different algorithms, we report the standard error of the mean.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources used reproduce the experiments are reported in Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wurrppss.ccc/pbbb/////ia
Answer: [Yes]
Justification:
The research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper presents algorithmic contribution for learning in POMDPs without addressing specific applications or deployment scenarios, and hence we not anticipate any direct positive negative societal consequences.
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do a potential risk of misuse for our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have provided proper accreditation wherever applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code implementing the method proposed in the paper is well-documented and provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurdie /atps/////oooefeeeeee////////// for what should or should be described.
29


Reviews:
[{"review_id": "I6Lr0eWhIU", "reviewer": "Reviewer_CaZk", "summary": "The paper presents novel techniques for Reinforcement Learning (RL) in *continuous* Partially-Observable Markov Decision Processes (POMDPs)—one of the most central and challenging problems in this area.\nFirst, the paper proposes an innovative technique based on results that have not been so far applied to POMDPs. Specifically, it reframes the RL problem for POMDPs in terms of Feynman–Kac Models, which also provides an alternative conceptual perspective on the problem\nThen, based on the formulation above, the paper proposes a Sequential Monte Carlo (SMC) algorithm, which allows for providing discrete approximations to Feynman–Kac models.\nFinally, the above contributions provide the basis for a *novel policy gradient method for POMDPs*.\nThe method is evaluated empirically on four standard benchmarks, comparing it against the state-of-the-art methods SLAC, DVRL, and Dual-SMC, *showing significant improvements*.", "questions": "**1.** Do you think finite bounds on performance guarantees (e.g., PAC bounds or regret bounds), or at least convergence guarantees, can be proved?\n\n**2.** If convergence cannot be guaranteed in the general case, do you see any reasonable assumptions that might suffice to ensure convergence? Or in general, conditions under which the problem might be well-behaved from a theoretical and/or practical point of view?\n\n**3.** Do you think your technique could be relevant for Regular Decision Processes (RDPs)* with continuous observations and actions?\n*Reference on RDPs: *Offline RL in Regular Decision Processes: Sample Efficiency via Language Metrics.* Deb et al., ICLR 2025.", "limitations": "No limitations beyond my comments above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n- The paper addresses a **very central problem**. It is in fact one of the most central problems in the POMDP literature. It is a challenging problem, for which no 'ultimate' solution is so far available, and improvements on the state-of-the-art are always important and likely to be impactful, both research-wise and in applications.\n\n- The **technical development is rigorous**.\n\n- The proposed **techniques are innovative** in that they combine several techniques that were not considered before in addressing RL in POMDPs, e.g., non-Markovian Feynman–Kac models.\n\n- The proposes **techniques are empirically shown to be effective**, improving on the state-of-the-art in challenging problems. Notably, a large improvement is observed in an active triangulation task.\n\n\n## Weaknesses\n\n\n**No performance guarantees.** The technique is not accompanied by a theoretical analysis deriving performance guarantees (e.g., PAC bounds or regret bounds) or at least convergence guarantees.\n\n**Presentation style can be improved.** The presentation style focuses on pointing out how exsiting frameworks and results are combined together in order to develop the novel contribution, with few words spent on describing the resulting novel definitions, and constructions.  As an example, in Line 143 and Equation (4), the intuitive meaning and the role of the auxiliary random variables $\\mathcal{O}_t$ is not given. Also, the choice of the exponential distribution in Equation (4) is not discussed.  With this presentation style, I feel that all the reader can do is to mechanically follow the paper as pieces come together, and wait to get to the end of the paper (making a significant effort) to build intuitions and the bigger pictures on how all pieces come together, and why specific choices were made in the constructions. Even worse, the presentation style may make the paper accessible only to an audience that is already familiar with all the employed notions and frameworks. This is far from ideal, as I understand that the paper is rather inter-disciplinary in its nature, as its novelty is largely based on addressing a central (classic) problem of POMDPs by resorting to techniques that have not been used before in this field.", "was_revised": true, "final_justification": "The authors have addressed all my concerns during the rebuttal, and promised to integrate the changes they have described. I will keep my positive score.", "rebuttal": "We thank the reviewer for their positive assessment and valuable suggestions for strengthening the manuscript. We address each comment and question in detail below.\n\n> The presentation style focuses on pointing out how existing frameworks and results are combined together in order to develop the novel contribution, with few words spent on describing the resulting novel definitions, and constructions. As an example, in Line 143 and Equation (4), the intuitive meaning and the role of the auxiliary random variables is not given. Also, the choice of the exponential distribution in Equation (4) is not discussed.\n\nWe thank the reviewer for this feedback on improving the clarity of our presentation. Defining an auxiliary random variable with a Boltzmann/Gibbs distribution is a common trick in reinforcement learning and the control-as-inference literature [1, 2, 3], but we agree that the current presentation assumes too much familiarity with this approach. We have added the following sentences to the text to address your concerns:\n\n  *For time $t$, $\\\\{ \\\\mathcal{O}\\_{t} = 1 \\\\}$ represents the event that the pair $(z_{t}, a_{t-1})$ is optimal given the history $(z_{0:t-1}, a_{0:t-2})$. A higher probability of optimality corresponds to higher expected reward $\\ell_{t}$. The exponential function is strictly positive and monotonic. Positive potential functions are required by the Feynman-Kac formalism and monotonicity guarantees that the potential function maintains the same maximizer as the reward function.*\n\nWe also acknowledge the reviewer's broader point about the presentation style making readers work through all details before seeing the complete picture. We will make small edits throughout the manuscript to improve flow and clarity where possible, while maintaining the technical structure.\n\n---\n\n> Do you think finite bounds on performance guarantees (e.g., PAC bounds or regret bounds), or at least convergence guarantees, can be proved? If convergence cannot be guaranteed in the general case, do you see any reasonable assumptions that might suffice to ensure convergence? Or in general, conditions under which the problem might be well-behaved from a theoretical and/or practical point of view?\n\nProviding convergence guarantees for our method is challenging. Our algorithm employs stochastic gradient ascent to maximize the objective $\\\\mathcal{B}_{\\\\eta}(\\\\phi)$. One of the two requirements for convergence to a local optimum under the Robbins-Monro conditions is an unbiased gradient estimator. However, our gradient estimates are computed as expectations over a particle approximation obtained with an SMC algorithm, which is biased for any finite number of particles [4].\n\nWhile standard SMC methods have well-understood bias properties, $\\\\mathcal{O}(1/N)$ with $N$ particles for additive functionals [4, 5], our nested structure with $N$ outer particles and $M$ inner particles creates more complex bias behavior whose theoretical characterization is still an open problem. Despite this theoretical limitation, the algorithm demonstrates reliable behavior in practice. We observe that increasing both $N$ and $M$ improves performance, and we can control gradient variance through the temperature parameter $\\eta$. To support these claims empirically, we have added ablation studies that highlight the role of $N$ and $M$ on the learning progress, and on the role of $\\eta$ in decreasing the gradient variance. Please see our answer to Reviewer [uahs] for more details on these empirical results.\n\nFormal PAC or regret bounds are beyond the scope of this work. Future theoretical contributions could include quantifying the bias of our nested SMC algorithm as a function of $N$ and $M$ (similar to [6]). Understanding this bias-particle relationship could potentially lead to convergence guarantees under appropriate growth conditions on $N$ and $M$ as learning progresses.\n\n---\n\n> Do you think your technique could be relevant for Regular Decision Processes (RDPs)* with continuous observations and actions? *Reference on RDPs: Offline RL in Regular Decision Processes: Sample Efficiency via Language Metrics. Deb et al., ICLR 2025.\n\nWe thank the reviewer for bringing RDPs to our attention, which we were not aware of. Given our limited experience with the RDP literature [7, 8], we want to be cautious with our claims. However, upon initial review of the RDP framework, it seems that RDPs can be understood as POMDPs [8] with finite observation and action spaces, and dynamics representable by finite-state automata. While we designed our algorithm for continuous POMDPs where the states, observations and actions lie in continuous spaces, we think the approach is general and can accommodate discrete or discrete-continuous POMDPs, and may be applicable to RDPs with continuous observations and actions.\n\n---\n\n## References\n1. Levine, S. (2018). Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909.\n2. Lee, A. X., Nagabandi, A., Abbeel, P., \\& Levine, S. (2020). Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. NeurIPS.\n3.  Lázaro-Gredilla, M., Ku, L., Murphy, K. P., \\& George, D. (2024). What type of inference is planning?. NeurIPS.\n4. Del Moral, P. (2004). Feynman-Kac formulae: Genealogical and interacting particle systems with applications. Springer New York.\n5. Kantas, N., Doucet, A., Singh, S. S., Maciejowski, J., \\& Chopin, N. (2015). On particle methods for parameter estimation in state-space models. Statistical Science, 30(3).\n6. Crisan, D., \\& Miguez, J. (2018). Nested particle filters for online parameter estimation in discrete-time state-space Markov models. Bernoulli, 24(4a).\n7. Deb, A., Cipollone, R., Jonsson, A., Ronca, A., \\& Talebi, M. S. (2025). Offline RL in Regular Decision Processes: Sample Efficiency via Language Metrics. ICLR\n8. Brafman, R. I., \\& De Giacomo, G. (2019). Regular Decision Processes: A Model for Non-Markovian Domains. IJCAI."}, {"review_id": "4EF8Uyjyve", "reviewer": "Reviewer_hAe2", "summary": "This paper proposes a new policy optimization framework for continuous POMDPs, named P3O. It addresses a key problem where many existing reinforcement learning methods, relying on the QMDP approximation, cannot perform deliberate information gathering because they ignore the value of future observations. To overcome this, the study formulates the POMDP as a probabilistic inference problem within a Feynman-Kac mode. This approach inherently incorporates the value of information gathering by anticipating future observations. The framework uses a nested Sequential Monte Carlo (SMC) method for optimization, which efficiently estimates the policy gradient by sampling from optimal trajectories. Experiments confirm that P3O significantly outperforms existing methods, particularly on tasks that require deliberate information gathering.", "questions": "**My primary concern is P3O's dependence on a true observation model**, which significantly limits its applicability to real-world problems. Could the authors elaborate on the feasibility of experimentally demonstrating an extension where this dependency is removed by using a learned, rather than an oracle, observation model?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper presents a coherent theoretical framework by reformulating policy learning in POMDPs as a  probabilistic inference problem within a non-Markovian Feynman-Kac model, based on the connection between optimization and inference.\n- The proposed method, P3O, avoids the QMDP approximation and reasons directly in the belief space, enabling it to incorporate actions aimed at reducing future uncertainty into its plans. This strength is clearly demonstrated in tasks where information gathering is key to success, such as 'Light-Dark' and Triangulation. The experiments show that agents trained with P3O learn to perform intentional exploration—such as first moving toward a light source (\"Light-Dark\") or executing an S-shaped maneuver (\"Triangulation\") to reduce positional uncertainty, rather than moving directly to the goal. This provides strong evidence for P3O's superiority, as these are tasks where QMDP-based methods fail.\n\n## Weaknesses\n\n- **Concerns about Scalability to High-Dimensional Problems.** As the authors acknowledge, the core of the proposed P3O method is a nested Sequential Monte Carlo (SMC) algorithm. This approach is susceptible to the \"curse of dimensionality,\" where the number of particles required increases exponentially as the dimensionality of the state and observation spaces grows, leading to a significant degradation in performance.\n- **Strong Dependence on a True Environment Model (Oracle).** P3O's learning loop uses a particle filter to track the belief state. This filter assumes that the true observation model, g(z|s), which maps states to observations, is known and accessible (i.e., as an oracle). This assumption is not met in many realistic scenarios, such as robotics or finance, where the environment model is often unknown. Therefore, the applicability of the method in its current form is significantly limited. The paper lacks discussion and experiments on how the method's performance would change if the observation model itself were learned from data.", "was_revised": true, "final_justification": "The authors' rebuttal successfully addressed my concerns regarding scalability. The proposed method is interesting, but its fundamental dependence on a true (oracle) observation model remains a significant limitation for practical, real-world applications. While the work has merit, this key constraint makes it a borderline case.", "rebuttal": "We thank the reviewer for their evaluation. We provide responses to the concerns raised below.\n\n> Concerns about Scalability to High-Dimensional Problems. As the authors acknowledge, the core of the proposed P3O method is a nested Sequential Monte Carlo (SMC) algorithm. This approach is susceptible to the \"curse of dimensionality,\" where the number of particles required increases exponentially as the dimensionality of the state and observation spaces grows, leading to a significant degradation in performance.\n\nThe reviewer raises an important point about scalability. They are correct that SMC methods, including our nested SMC approach, can suffer from the curse of dimensionality in high-dimensional problems due to its reliance on importance sampling. We appreciate the opportunity to clarify our perspective on this important limitation.\n\nIn many practical problems with high-dimensional state/observation spaces, the likelihood is concentrated in lower-dimensional manifolds, and the effective dimension governing particle requirements can be much smaller than the ambient dimension [1]. The number of particles required is also influenced by the discrepancy between the target and proposal distributions [1, 2], and hence can be reduced by either constructing or learning better proposals that are adapted to the target [3, 4, 5]. Algorithm 2 admits the use of any proposal distribution, and our choice of the transition dynamics as the proposal was made to to simplify the exposition. Future extensions of this work can focus on integrating proposal learning into the policy learning loop.\n\nWe have modified the discussion section in the paper as follows to incorporate these ideas: \n\n*The bootstrap particle filter used in our implementation may not scale well in high-dimensional settings due to increased discrepancy between the proposal and target distributions. Using more sophisticated proposals that take the likelihood into account [3, Chapter 10] or learning better proposals [4, 5] may be necessary to ensure satisfactory performance for high-dimensional problems.*\n\n---\n\n> My primary concern is P3O's dependence on a true observation model, which significantly limits its applicability to real-world problems. Could the authors elaborate on the feasibility of experimentally demonstrating an extension where this dependency is removed by using a learned, rather than an oracle, observation model?\n\nAs we acknowledge in the draft, we agree with the reviewer that the dependence on a true observation model limits our algorithm's applicability. However, we would like to highlight that there are many real-world POMDP settings with known observation models. For example, in robotic systems, like manipulators and drones, only position measurements are available while velocity cannot be measured. Another example is bearings‑only active target tracking in radar systems. In both cases, observation models can be extracted analytically from physical relationships. \n\nNonetheless, in principle, it is feasible to postulate an extension of our method that learns an observation model, similar to [6, 7]. One can parameterize a conditional density estimator with a variational auto-encoder or a normalizing flow and alternate between policy optimization and observation-density learning in Algorithm 2. Unfortunately, the exact details cannot be reasonably explored in the scope of this rebuttal.\n\nWe deliberately focused on the case with known models to isolate and address the fundamental challenge of POMDP optimization without resorting to the suboptimal QMDP approximation typically used in RL frameworks.\n\n---\n\n## References\n\n1. Agapiou, S., Papaspiliopoulos, O., Sanz-Alonso, D., \\& Stuart, A. M. (2017). Importance sampling: Intrinsic dimension and computational cost. Statistical Science, 405-431.\n2. Chatterjee, S., \\& Diaconis, P. (2018). The sample size required in importance sampling. The Annals of Applied Probability, 28(2), 1099-1135.\n3. Chopin, N., \\& Papaspiliopoulos, O. (2020). An introduction to sequential Monte Carlo. Springer.\n4. Naesseth, C., Linderman, S., Ranganath, R., \\& Blei, D. (2018). Variational sequential Monte Carlo. AISTATS.\n5. Heng, J., Bishop, A. N., Deligiannidis, G., \\& Doucet, A. (2020). Controlled sequential Monte Carlo. The Annals of Statistics, 48(5), 2904-2929.\n6. Lee, A. X., Nagabandi, A., Abbeel, P., & Levine, S. (2020). Stochastic latent actor-critic:\nDeep reinforcement learning with a latent variable model. NeurIPS.\n7. M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson (2018). Deep variational reinforcement learning for\nPOMDPs. ICML."}, {"review_id": "HRWq9HF8z8", "reviewer": "Reviewer_uahs", "summary": "This paper presents a nested particle filtering framework for policy optimization in partially observable Markov decision processes (POMDPs). The authors formulate a risk-sensitive belief-space objective using a Feynman–Kac distribution and derive a novel policy gradient estimator (Proposition 2) based on Fisher’s identity. The proposed method performs sampling over belief trajectories using a two-layer particle filter—one for observation-action histories and another for latent state inference. The framework is validated on continuous-control and visual POMDP benchmarks.", "questions": "1. Discuss the bias–variance tradeoff in the nested sampling procedure. How does the number of history particles (N) and belief particles (M) affect performance and stability?\n2. Provide empirical or theoretical insights into whether the gradient estimator is prone to local minima due to limited expressiveness or the nested approximation.\n3. I am not an expert in the policy learning area. Please clarify whether this method supports online or only offline training, and what implications this approach may have for real-time control.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Clear and Coherent Framework: The paper presents a well-motivated and mathematically coherent approach to optimizing policies in POMDPs via nested sequential Monte Carlo (SMC). The derivation of the Feynman–Kac-based trajectory distribution is sound.\n2. Main Innovation in Proposition 2: The derivation of the policy gradient using Fisher’s identity over the Feynman–Kac belief-trajectory distribution appears novel in the context of POMDPs. This formulation offers a clean way to estimate gradients under a risk-sensitive trajectory objective.\n3. Use of Risk-Sensitive Objective: The incorporation of a soft-optimality reward structure (Equation 6) is relatively uncommon in belief-space POMDP literature and can be viewed as a meaningful design contribution.\n4. Algorithm Design: The nested structure in Algorithm 2 is a practical realization of the theoretical framework, clearly leveraging a two-level particle system to handle both history uncertainty and belief uncertainty.\n\nWeaknesses\n1. Limited Novelty Outside Proposition 2: While Proposition 2 is the core contribution, most other components rely on known techniques. For example, Proposition 1 simply reformulates the standard POMDP objective in belief space, and the use of Feynman–Kac models in SMC is well-established.\n2. Lack of Discussion on Optimization Challenges: The method relies on gradient ascent over a non-convex and highly stochastic objective. The paper does not discuss whether the optimization suffers from local minima or high variance in gradient estimates, especially under particle approximation and finite horizon.\n3. Scalability and Cost: Although the algorithm is elegant, the nested particle structure introduces significant computational cost (O(NM) per time step). There is no discussion on scaling the method to long-horizon or high-dimensional settings.\n4. Missing Comparisons: While comparisons to some baselines are provided, the paper omits relevant prior works that also perform sampling-based policy learning in belief or latent space (e.g., VIREL, Variational Policy Search, SMCP, PO-SMC).\n5. No Ablation or Variance Analysis: It would be valuable to show the effect of η (risk level), particle count (N, M), or nesting on gradient variance and convergence.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and evaluation. We provide responses to the concerns raised below.\n\n>Scalability and Cost: Although the algorithm is elegant, the nested particle structure introduces significant computational cost (O(NM) per time step). There is no discussion on scaling the method to long-horizon or high-dimensional settings.\n\nThe reviewer raises a valid point. We previously discussed time complexity in the appendix in Section C.2. However, given its importance, we agree it should be highlighted in Section 6. We have now added the following text to Section 6:\n\n*Algorithm 2 has computational complexity $\\mathcal{O}(NMT)$, which increases to $\\mathcal{O}(NM^2T^2)$ when employing backward sampling (Appendix C). For long-horizon problems, we therefore recommend avoiding backward sampling and instead tuning the parameter $\\eta$ to mitigate particle degeneracy while maintaining computational efficiency.*\n\n---\n\nWe also agree with the reviewer that scalability to high dimensions is an issue. Our draft briefly raises this point in Section 6. However, given the reviewer's comment, we have decided to expand that discussion to highlight the point further. Section 6 now includes the following text: \n\n*The bootstrap particle filter used in our implementation may not scale well in high-dimensional settings due to increased discrepancy between the proposal and target distributions. Using more sophisticated proposals that take the likelihood into account [3, Chapter 10] or learning better proposals [4, 5] may be necessary to ensure satisfactory performance for high-dimensional problems.*\n\nPlease also see our answer to Reviewer [hAe2] for a more detailed discussion on scaling to higher dimensions.\n\n---\n\n> While comparisons to some baselines are provided, the paper omits relevant prior works that also perform sampling-based policy learning in belief or latent space (e.g., VIREL, Variational Policy Search, SMCP, PO-SMC).\n\nWe thank the reviewer for pointing out potentially relevant prior work that we may have overlooked. To our knowledge, VIREL [1] and Variational Policy Search [2] are algorithms that target MDP settings and are not applicable to POMDPs. As for SMCP and PO-SMC, despite our efforts, we were unable to identify papers with these specific acronyms in the POMDP literature. We would be grateful if the reviewer could provide full citations or author names, as these works may use different terminology than we anticipated.\n\n---\n\n> No Ablation or Variance Analysis: It would be valuable to show the effect of $\\eta$ (risk level), particle count (N, M), or nesting on gradient variance and convergence.\n\nFollowing the reviewer's recommendation, we have now conducted ablation studies examining the impact of key hyperparameters on algorithm performance. Specifically, we performed ablation analyses in the Cart-Pole and Light-Dark experiments, varying the following hyperparameters:\n\n| **Cart-Pole** | **Light-Dark** |\n|---------------|----------------|\n| Outer particles: $N \\in$ \\{32, 64, 128\\}  | Outer particles: $N \\in$ \\{32, 128, 256\\} |\n| Inner particles: $M \\in$ \\{32, 128, 512\\}  | Inner particles: $M \\in$ \\{16, 64, 256\\} |\n| Tempering: $ \\eta \\in$ \\{0.005, 0.01, 0.05, 1\\} | Tempering: $ \\eta \\in$ \\{0.001, 0.005, 0.01, 0.05\\} |\n\nIn this ablation study, we plot the learning progress for the listed hyperparameters, averaged over 10 seeds. The plots show an expected behavior: increasing $N$, $M$, leads to better approximations of the distribution $\\Psi_{T}$ and contributes to faster training. The parameter $\\eta$ controls the variance of the gradient: lower values of $\\eta$ lead to higher-variance gradients and cause learning to stall and increase the variability across seeds.\n\nTo isolate the role of $\\eta$ even further, we run an additional experiment to analyze the variance of the gradient estimate in a linear-quadratic-Gaussian (LQG) control problem. The table attached below clearly shows that lower values of $\\eta$ lead to higher variance of the policy gradient as measured by the trace of the gradient covariance matrix. It is important to remark that the REINFORCE policy gradient is recovered for $\\eta \\to 0$, corresponding to high-variance policy gradients:\n\n| Tempering $\\eta$   | Trace |\n|----------------------|----------------------------------------|\n| $ 1.0 \\times 10^{-3} $ | $ 21.2627 \\pm 0.6912 $ |\n| $ 5.0 \\times 10^{-3} $ | $ 20.0137 \\pm 0.7931 $ |\n| $ 1.0 \\times 10^{-2} $ | $ 19.7993 \\pm 0.6831 $ |\n| $ 5.0 \\times 10^{-2} $ | $ 15.5139 \\pm 0.3298 $ |\n| $ 1.0 \\times 10^{-1} $ | $ 13.3801 \\pm 0.3931 $ |\n| $ 5.0 \\times 10^{-1} $ | $ 6.3718 \\pm 0.2818 $ |\n| $ 1.0 \\times 10^{0} $  | $ 4.2700 \\pm 0.1835 $ |\n| $ 5.0 \\times 10^{0} $  | $ 1.7832 \\pm 0.0647 $ |\n| $ 1.0 \\times 10^{1} $  | $ 1.2546 \\pm 0.0505 $ |\n\nThese ablation studies provide important practical guidance for hyperparameter selection. The detailed results and figures have been added to Section 5 of the revised manuscript. We are grateful to the reviewer for this suggestion, which has improved the quality and completeness of our experimental results.\n\n**Unfortunately, the NeurIPS rebuttal guidelines do not allow us to provide a modified PDF with the highlighted changes.**\n\n---\n\n>Discuss the bias–variance tradeoff in the nested sampling procedure. How does the number of history particles (N) and belief particles (M) affect performance and stability?\n\nIn standard particle filters, both bias and variance decrease at $\\mathcal{O}(1/N)$ rate with the number of particles $N$ [3], so there is no traditional bias-variance tradeoff with respect to the particle count. We expect the same property to hold for our nested particle filter: increasing either $N$ or $M$ reduces both bias and variance.\n\nThe primary bias-variance tradeoff in our method arises from the temperature parameter $\\eta$, where lower values provide unbiased but high-variance gradient estimates, while higher values reduce variance at the cost of introducing bias with respect to the risk-neutral objective. This behavior is standard in decision-making objectives that incorporate risk.\n\nFor a more detailed comment of how $N$ and $M$ affect performance and stability, please see our response to your comment on ablation analysis above.\n\n---\n\n> Provide empirical or theoretical insights into whether the gradient estimator is prone to local minima due to limited expressiveness or the nested approximation.\n\nWe assume the reviewer meant \"limited expressiveness *of* the nested approximation\".\nOur nested SMC algorithm introduces bias in our gradient estimates, but it is difficult to draw conclusions on the quality of the solutions based on that. Biases appear in most policy optimization techniques, for example, in bootstrapped TD learning that powers most state-of-the-art algorithms, or in approaches that rely on iterative linearizations, or variational Gaussian approximations, which are popular approaches in POMDPs. Like most policy gradient methods, our approach can converge to local optima, mainly because the underlying decision-making problem is non-convex, but this non-convexity is inherent to the general POMDP structure. There is no indication that the nested SMC structure exacerbates this issue compared to the other POMDP solution methods we compare to.\n\n---\n>Please clarify whether this method supports online or only offline training, and what implications this approach may have for real-time control.\n\nOur approach involves directly interacting with the environment during training so it is online in that sense. We generate new trajectories using the current policy, perform updates and iterate over this procedure. However, once a policy is learned, deployment is very cheap. Obtaining actions only requires a forward pass through a neural network, leading to minimal computational requirements, and making our approach practical for real-time control applications.\n\n---\n## References:\n\n1. Fellows, M., Mahajan, A., Rudner, T. G., Whiteson, S. (2019). VIREL: A variational inference framework for reinforcement learning. NeurIPS.\n\n2. Levine, S., Koltun, V. (2013). Variational policy search via trajectory optimization. NeurIPS.\n\n3. Chopin, N., \\& Papaspiliopoulos, O. (2020). An introduction to sequential Monte Carlo. Springer.\n\n4. Naesseth, C., Linderman, S., Ranganath, R., \\& Blei, D. (2018). Variational sequential Monte Carlo. AISTATS.\n\n5. Heng, J., Bishop, A. N., Deligiannidis, G., \\& Doucet, A. (2020). Controlled sequential Monte Carlo. The Annals of Statistics, 48(5), 2904-2929."}, {"review_id": "x7RL4FXXVU", "reviewer": "Reviewer_6N23", "summary": "The paper addresses the problem of solving planning problems in POMDPs. For this ideas from particle filtering and control via inference are combined. The authors present a nested particle filtering approach to solve the planning problem under partial observability. The paper evaluates the problem on a set of benchmarks problems.", "questions": "How could you deal with unknown models in this work, such as, reward functions, dynamics or emission probabilities?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is overall well-written and presents an interesting method that demonstrates its performance on benchmark tasks. The related work is integrated nicely and provides a solid foundation for the reader. The explanation of related concepts is thorough, although in some cases—such as the detailed description of the bootstrap filter or control via inference—it may be more elaborate than necessary for the main narrative.\n\nThe method itself is compelling and shows promise, particularly in its ability to handle complex scenarios. However, several areas could benefit from further development. For instance, the derivation of the core methodology is largely relegated to the appendix; incorporating more of this into the main text would enhance clarity and accessibility. Additionally, a deeper discussion of computational complexity is warranted, especially given that the nesting approach implies an exponential increase in the number of particles required.\n\nEmpirical results are promising, but the evaluation could be strengthened. More extensive results for the filter would be valuable, particularly to demonstrate the absence of variance collapse. Including reference results for standard models such as LQG would also help contextualize performance. Furthermore, it would be beneficial to show experimentally how the proposed estimator achieves lower variance, as claimed.\n\nThe manuscript raises important questions that are not fully addressed, such as how the method scales to high-dimensional settings, what specific requirements are imposed by the approach, and how it might be adapted to handle unknown models—potentially through learning-based techniques like the ideas presented within DVRL.\n\nDespite these points, the paper makes a meaningful contribution to the field and introduces a novel approach that is both technically sound and practically relevant. The strengths of the work outweigh the current limitations, and I believe it will spark valuable discussion and further research.", "was_revised": true, "final_justification": "I have read the rebuttal and thank the authors for their comments. I appreciate the inclusion of the LQG benchmark and further discussion on the algorithms details. I will keep my score and recommendation.", "rebuttal": "We thank the reviewer for their time and evaluation. We provide responses to the concerns raised below.\n\n> A deeper discussion of computational complexity is warranted, especially given that the nesting approach implies an exponential increase in the number of particles required.\n\nThe reviewer is correct to highlight the computational complexity issue. We would like to clarify, the nesting does not necessarily imply exponential growth in particle number, instead it leads to complexity $N \\times M$ per time step. Nonetheless, based on the comments of multiple reviewers, we believe the issue of complexity is important and has to be addressed within the main text. We have added the following text to the discussion in Section 6: \n\n*Algorithm 2 has computational complexity $\\mathcal{O}(NMT)$, which increases to $\\mathcal{O}(NM^2T^2)$ when employing backward sampling (Appendix C). For long-horizon problems, we therefore recommend avoiding backward sampling and instead tuning the parameter $\\eta$ to mitigate particle degeneracy while maintaining computational efficiency.*\n\nAdditionally, in our answer to Reviewer [hAe2], we discuss further details on how the number of particles scales with dimensionality in SMC methods in general.\n\n---\n\n> Including reference results for standard models such as LQG would also help contextualize performance. Furthermore, it would be beneficial to show experimentally how the proposed estimator achieves lower variance, as claimed.\n\nBased on the reviewer's recommendation, we have now included an LQG baseline comparison. The baseline uses LQG to optimally control a linear-Gaussian dynamical system with quadratic rewards accompanied by a Kalman filter belief-tracker. The resulting plot shows the optimal performance of LQG and how our policy learning method gradually converges to the same level over learning iterations, thus empirically validating against the closed-form LQG baseline. \n\nAdditionally, we run an experiment to analyze the variance of the gradient estimate for the LQG problem. The table below clearly shows that lower values of $\\eta$ lead to higher variance of the policy gradient as measured by the trace of the gradient covariance matrix. It is important to remark that the REINFORCE gradient is recovered at the limit of $\\eta \\to 0$, corresponding to high-variance gradients:\n\n| Tempering $\\eta$   | Trace |\n|----------------------|----------------------------------------|\n| $ 1.0 \\times 10^{-3} $ | $ 21.2627 \\pm 0.6912 $ |\n| $ 5.0 \\times 10^{-3} $ | $ 20.0137 \\pm 0.7931 $ |\n| $ 1.0 \\times 10^{-2} $ | $ 19.7993 \\pm 0.6831 $ |\n| $ 5.0 \\times 10^{-2} $ | $ 15.5139 \\pm 0.3298 $ |\n| $ 1.0 \\times 10^{-1} $ | $ 13.3801 \\pm 0.3931 $ |\n| $ 5.0 \\times 10^{-1} $ | $ 6.3718 \\pm 0.2818 $ |\n| $ 1.0 \\times 10^{0} $  | $ 4.2700 \\pm 0.1835 $ |\n| $ 5.0 \\times 10^{0} $  | $ 1.7832 \\pm 0.0647 $ |\n| $ 1.0 \\times 10^{1} $  | $ 1.2546 \\pm 0.0505 $ |\n\nFinally, we run more ablation studies to highlight the role of the number of particles $N$ and $M$ on the learning procedure. The plots show an expected behavior: increasing $N$, $M$, leads to better approximations of the distribution $\\Psi_{T}$ and contributes to faster learning and better quality solutions.\n\n**Unfortunately, the NeurIPS rebuttal guidelines do not allow us to provide a modified PDF with the highlighted changes.**\n\n---\n\n>How could you deal with unknown models in this work, such as, reward functions, dynamics or emission probabilities?\n\nThis is an interesting question. On the one hand, our approach does not require full access to reward and dynamics functions. These can be treated as black-box quantities to sample transitions and receive reward signals. However, our method currently requires oracle access to the observation model in order to evaluate emission probabilities while tracking the belief. \n\nIn principle, it is feasible to postulate an extension of our method that learns model, similar to [1, 2]. One can parameterize a conditional density estimator with a variational auto-encoder or a normalizing flow and alternate between policy optimization and density-learning in Algorithm 2. This may require formulating and optimizing a lower-bound on the marginal log-likelihood, but the details of such an approach are difficult to explore in the scope of this rebuttal.\n\nWe appreciate the reviewer recognizing our contribution nonetheless. We deliberately focused on the case with known models to isolate and address the fundamental challenge of POMDP optimization without resorting to the suboptimal QMDP approximation typically used in RL frameworks. Our future work will target making this approach as general as possible.\n\n---\n\n## References:\n\n1. Lee, A. X., Nagabandi, A., Abbeel, P., & Levine, S. (2020). Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. NeurIPS.\n2. M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson (2018). Deep variational reinforcement learning for POMDPs. ICML."}]

Author rebuttal:

Meta-review: This paper introduces a novel formulation of RL in continuous POMDPs via Feynman–Kac models, leading to a Sequential Monte Carlo algorithm and a new policy gradient method. The approach is original and shows empirical gains over QMDP baselines.

Reviewers agree that importing Feynman–Kac models into RL is both interesting and relevant. Continuous POMDPs are a challenging problem, and this new perspective is promising.

Concerns were raised about scalability to high-dimensional problems, reliance on a true observation model, and the thoroughness of the experimental evaluation. The rebuttal provided additional results that alleviated some of the experimental concerns. Theoretical and scalability aspects were addressed, but there is still clear scope for further work.

Thus, I recommend acceptance, while encouraging the authors to strengthen both theoretical guarantees and practical scalability to realistic high-dimensional problems moving forward.
Decision: Accept (poster)