Title: Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: POMDP

Policy Optimization

Sequential Monte Carlo

Feynman--Kac

Control-as-Inference

Decision-Making under Uncertainty
Abstract: Optimal decision-making under partial observability requires agents to balance reducing uncertainty (exploration) against pursuing immediate objectives (exploitation). In this paper, we introduce a novel policy optimization framework for continuous partially observable Markov decision processes (POMDPs) that explicitly addresses this challenge. Our method casts policy learning as probabilistic inference in a non-Markovian Feynman--Kac model that inherently captures the value of information gathering by anticipating future observations, without requiring suboptimal approximations or handcrafted heuristics. To optimize policies under this model, we develop a nested sequential Monte Carlo (SMC) algorithm that efficiently estimates a history-dependent policy gradient under samples from the optimal trajectory distribution induced by the POMDP. We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks, where existing methods struggle to act under uncertainty.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of solving planning problems in POMDPs. For this ideas from particle filtering and control via inference are combined. The authors present a nested particle filtering approach to solve the planning problem under partial observability. The paper evaluates the problem on a set of benchmarks problems.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is overall well-written and presents an interesting method that demonstrates its performance on benchmark tasks. The related work is integrated nicely and provides a solid foundation for the reader. The explanation of related concepts is thorough, although in some cases—such as the detailed description of the bootstrap filter or control via inference—it may be more elaborate than necessary for the main narrative.\n\nThe method itself is compelling and shows promise, particularly in its ability to handle complex scenarios. However, several areas could benefit from further development. For instance, the derivation of the core methodology is largely relegated to the appendix; incorporating more of this into the main text would enhance clarity and accessibility. Additionally, a deeper discussion of computational complexity is warranted, especially given that the nesting approach implies an exponential increase in the number of particles required.\n\nEmpirical results are promising, but the evaluation could be strengthened. More extensive results for the filter would be valuable, particularly to demonstrate the absence of variance collapse. Including reference results for standard models such as LQG would also help contextualize performance. Furthermore, it would be beneficial to show experimentally how the proposed estimator achieves lower variance, as claimed.\n\nThe manuscript raises important questions that are not fully addressed, such as how the method scales to high-dimensional settings, what specific requirements are imposed by the approach, and how it might be adapted to handle unknown models—potentially through learning-based techniques like the ideas presented within DVRL.\n\nDespite these points, the paper makes a meaningful contribution to the field and introduces a novel approach that is both technically sound and practically relevant. The strengths of the work outweigh the current limitations, and I believe it will spark valuable discussion and further research.\n\nQUESTIONS:\nHow could you deal with unknown models in this work, such as, reward functions, dynamics or emission probabilities?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a nested particle filtering framework for policy optimization in partially observable Markov decision processes (POMDPs). The authors formulate a risk-sensitive belief-space objective using a Feynman–Kac distribution and derive a novel policy gradient estimator (Proposition 2) based on Fisher’s identity. The proposed method performs sampling over belief trajectories using a two-layer particle filter—one for observation-action histories and another for latent state inference. The framework is validated on continuous-control and visual POMDP benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Clear and Coherent Framework: The paper presents a well-motivated and mathematically coherent approach to optimizing policies in POMDPs via nested sequential Monte Carlo (SMC). The derivation of the Feynman–Kac-based trajectory distribution is sound.\n2. Main Innovation in Proposition 2: The derivation of the policy gradient using Fisher’s identity over the Feynman–Kac belief-trajectory distribution appears novel in the context of POMDPs. This formulation offers a clean way to estimate gradients under a risk-sensitive trajectory objective.\n3. Use of Risk-Sensitive Objective: The incorporation of a soft-optimality reward structure (Equation 6) is relatively uncommon in belief-space POMDP literature and can be viewed as a meaningful design contribution.\n4. Algorithm Design: The nested structure in Algorithm 2 is a practical realization of the theoretical framework, clearly leveraging a two-level particle system to handle both history uncertainty and belief uncertainty.\n\nWeaknesses\n1. Limited Novelty Outside Proposition 2: While Proposition 2 is the core contribution, most other components rely on known techniques. For example, Proposition 1 simply reformulates the standard POMDP objective in belief space, and the use of Feynman–Kac models in SMC is well-established.\n2. Lack of Discussion on Optimization Challenges: The method relies on gradient ascent over a non-convex and highly stochastic objective. The paper does not discuss whether the optimization suffers from local minima or high variance in gradient estimates, especially under particle approximation and finite horizon.\n3. Scalability and Cost: Although the algorithm is elegant, the nested particle structure introduces significant computational cost (O(NM) per time step). There is no discussion on scaling the method to long-horizon or high-dimensional settings.\n4. Missing Comparisons: While comparisons to some baselines are provided, the paper omits relevant prior works that also perform sampling-based policy learning in belief or latent space (e.g., VIREL, Variational Policy Search, SMCP, PO-SMC).\n5. No Ablation or Variance Analysis: It would be valuable to show the effect of η (risk level), particle count (N, M), or nesting on gradient variance and convergence.\n\nQUESTIONS:\n1. Discuss the bias–variance tradeoff in the nested sampling procedure. How does the number of history particles (N) and belief particles (M) affect performance and stability?\n2. Provide empirical or theoretical insights into whether the gradient estimator is prone to local minima due to limited expressiveness or the nested approximation.\n3. I am not an expert in the policy learning area. Please clarify whether this method supports online or only offline training, and what implications this approach may have for real-time control.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new policy optimization framework for continuous POMDPs, named P3O. It addresses a key problem where many existing reinforcement learning methods, relying on the QMDP approximation, cannot perform deliberate information gathering because they ignore the value of future observations. To overcome this, the study formulates the POMDP as a probabilistic inference problem within a Feynman-Kac mode. This approach inherently incorporates the value of information gathering by anticipating future observations. The framework uses a nested Sequential Monte Carlo (SMC) method for optimization, which efficiently estimates the policy gradient by sampling from optimal trajectories. Experiments confirm that P3O significantly outperforms existing methods, particularly on tasks that require deliberate information gathering.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper presents a coherent theoretical framework by reformulating policy learning in POMDPs as a  probabilistic inference problem within a non-Markovian Feynman-Kac model, based on the connection between optimization and inference.\n- The proposed method, P3O, avoids the QMDP approximation and reasons directly in the belief space, enabling it to incorporate actions aimed at reducing future uncertainty into its plans. This strength is clearly demonstrated in tasks where information gathering is key to success, such as 'Light-Dark' and Triangulation. The experiments show that agents trained with P3O learn to perform intentional exploration—such as first moving toward a light source (\"Light-Dark\") or executing an S-shaped maneuver (\"Triangulation\") to reduce positional uncertainty, rather than moving directly to the goal. This provides strong evidence for P3O's superiority, as these are tasks where QMDP-based methods fail.\n\n## Weaknesses\n\n- **Concerns about Scalability to High-Dimensional Problems.** As the authors acknowledge, the core of the proposed P3O method is a nested Sequential Monte Carlo (SMC) algorithm. This approach is susceptible to the \"curse of dimensionality,\" where the number of particles required increases exponentially as the dimensionality of the state and observation spaces grows, leading to a significant degradation in performance.\n- **Strong Dependence on a True Environment Model (Oracle).** P3O's learning loop uses a particle filter to track the belief state. This filter assumes that the true observation model, g(z|s), which maps states to observations, is known and accessible (i.e., as an oracle). This assumption is not met in many realistic scenarios, such as robotics or finance, where the environment model is often unknown. Therefore, the applicability of the method in its current form is significantly limited. The paper lacks discussion and experiments on how the method's performance would change if the observation model itself were learned from data.\n\nQUESTIONS:\n**My primary concern is P3O's dependence on a true observation model**, which significantly limits its applicability to real-world problems. Could the authors elaborate on the feasibility of experimentally demonstrating an extension where this dependency is removed by using a learned, rather than an oracle, observation model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents novel techniques for Reinforcement Learning (RL) in *continuous* Partially-Observable Markov Decision Processes (POMDPs)—one of the most central and challenging problems in this area.\nFirst, the paper proposes an innovative technique based on results that have not been so far applied to POMDPs. Specifically, it reframes the RL problem for POMDPs in terms of Feynman–Kac Models, which also provides an alternative conceptual perspective on the problem\nThen, based on the formulation above, the paper proposes a Sequential Monte Carlo (SMC) algorithm, which allows for providing discrete approximations to Feynman–Kac models.\nFinally, the above contributions provide the basis for a *novel policy gradient method for POMDPs*.\nThe method is evaluated empirically on four standard benchmarks, comparing it against the state-of-the-art methods SLAC, DVRL, and Dual-SMC, *showing significant improvements*.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper addresses a **very central problem**. It is in fact one of the most central problems in the POMDP literature. It is a challenging problem, for which no 'ultimate' solution is so far available, and improvements on the state-of-the-art are always important and likely to be impactful, both research-wise and in applications.\n\n- The **technical development is rigorous**.\n\n- The proposed **techniques are innovative** in that they combine several techniques that were not considered before in addressing RL in POMDPs, e.g., non-Markovian Feynman–Kac models.\n\n- The proposes **techniques are empirically shown to be effective**, improving on the state-of-the-art in challenging problems. Notably, a large improvement is observed in an active triangulation task.\n\n\n## Weaknesses\n\n\n**No performance guarantees.** The technique is not accompanied by a theoretical analysis deriving performance guarantees (e.g., PAC bounds or regret bounds) or at least convergence guarantees.\n\n**Presentation style can be improved.** The presentation style focuses on pointing out how exsiting frameworks and results are combined together in order to develop the novel contribution, with few words spent on describing the resulting novel definitions, and constructions.  As an example, in Line 143 and Equation (4), the intuitive meaning and the role of the auxiliary random variables $\\mathcal{O}_t$ is not given. Also, the choice of the exponential distribution in Equation (4) is not discussed.  With this presentation style, I feel that all the reader can do is to mechanically follow the paper as pieces come together, and wait to get to the end of the paper (making a significant effort) to build intuitions and the bigger pictures on how all pieces come together, and why specific choices were made in the constructions. Even worse, the presentation style may make the paper accessible only to an audience that is already familiar with all the employed notions and frameworks. This is far from ideal, as I understand that the paper is rather inter-disciplinary in its nature, as its novelty is largely based on addressing a central (classic) problem of POMDPs by resorting to techniques that have not been used before in this field.\n\nQUESTIONS:\n**1.** Do you think finite bounds on performance guarantees (e.g., PAC bounds or regret bounds), or at least convergence guarantees, can be proved?\n\n**2.** If convergence cannot be guaranteed in the general case, do you see any reasonable assumptions that might suffice to ensure convergence? Or in general, conditions under which the problem might be well-behaved from a theoretical and/or practical point of view?\n\n**3.** Do you think your technique could be relevant for Regular Decision Processes (RDPs)* with continuous observations and actions?\n*Reference on RDPs: *Offline RL in Regular Decision Processes: Sample Efficiency via Language Metrics.* Deb et al., ICLR 2025.\n\nLIMITATIONS:\nNo limitations beyond my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}