Title: Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability
Venue: NeurIPS 2025
Accepted: yes
Keywords: POMDPs

Robustness

Multi-environment
Abstract: Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty. ME-POMDPs represent a finite set of POMDPs that share the same state, action, and observation spaces, but may arbitrarily vary in their transition, observation, and reward models. Such models arise, for instance, when multiple domain experts disagree on how to model a problem. The goal is to find a single policy that is robust against any choice of POMDP within the set, *i.e.*, a policy that maximizes the worst-case reward across all POMDPs. We generalize and expand on existing work in the following way. First, we show that ME-POMDPs can be generalized to POMDPs *with sets of initial beliefs*, which we call *adversarial-belief POMDPs* (AB-POMDPs). Second, we show that any arbitrary ME-POMDP can be reduced to a ME-POMDP that only varies in its transition and reward functions or only in its observation and reward functions, while preserving (optimal) policies. We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs, and thus ME-POMDPs. We demonstrate that we can compute policies for standard POMDP benchmarks extended to the multi-environment setting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses finding a robust policy when there are a finite different models for a POMDP, where each model can have different transition, observation, and reward models. To solve this, they are optimizing for the policy that maximizes the expected value of the worst-case model. They show that solving this problem is equivalent to playing a partially observable adversarial game where the fully observable agent’s policy is to minimize the performance of the partially observable agent by selecting the model. They also provide a process of converting the set of models into a Multi-Environment POMDP, where the set of models is encoded into the problem and belief state. As such, the adversarial agent is selecting a belief state. They introduce an algorithm, Adversarial-Belief HSVI, for solving for the robust policy and evaluate this algorithm on two different domains.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* The main contribution of this paper is the framework and theoretical results that are clear and sound, to the best of my knowledge.\n* The paper clearly distinguishes itself from the existing work on robust policies, providing a convincing argument on its novelty.\n* The methodology of the empirical evaluation is clear and reproducible. \n\n**Weaknesses:**\n\n* The paper currently does not have a proof or empirical evaluation showing that the AB-HSVI algorithm converges. It is clear that it does, just that explicitly including it would strengthen the paper.\n* While the empirical evaluation effectively shows the behavior of the AB-HSVI algorithm and solving for the robust policy using the ME-POMDP, it is hard to say when solving for the robust policy is feasible.\n* It would have been nice if a more substantial ablation study had been performed to evaluate the approximate computational complexity of the individual variables (number of experts, amount of disagreement between the models, etc.) \n\n**Other Comments:**\n\n* One assumption that I think needs to be mentioned is that the reward function needs to use the same units between models. Otherwise, comparing the expected value between the models is not a meaningful metric. For example, an expert could increase the survival cost in Rocksample such that their model dominates the set of models, causing the robust policy to have the agent exit as fast as possible.\n* The performance and the magnitude of the differences between the models would impact the quality of the robust policy. For example, if the POMDP encodes a multi-objective problem where experts disagree on the objective, the robust policy is unlikely to achieve either objective.\n* Table 4’s comparison of the robust policy compared to the optimal policy for each individual model is interesting but causes more confusion than benefits for the main paper. Since the robust policy solves a different problem, the only conclusion that can be drawn is that solving the optimal policy for an individual model will outperform the robust policy. Rather, it creates more confusion, suggesting that the robust policy is somehow one of these optimal policies.\n\nQUESTIONS:\n1.\tDo you have a proof or empirical evaluation for showing that AB-HSVI converges over time?\n2.\tHave you evaluated any other problems or done additional evaluation on the computation cost of changing the individual factors?\n\nLIMITATIONS:\nThe main limitations are discussed; however, having a limitation section would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses robust policy planning in ME-POMDPs, where multiple POMDPs defined over the same state, action, and observation spaces differ in their dynamics and reward functions. The authors extend the theoretical framework by introducing AB-POMDPs, which generalize ME-POMDPs by allowing the initial belief to be adversarially selected from a set. The main theoretical contribution is showing that any ME-POMDP can be efficiently reduced to an AB-POMDP, and both can be formulated as special cases of one-sided POSGs. The paper develops both exact and approximate (point-based) algorithms for robust policy computation, notably extending HSVI with linear programming to handle adversarial initial beliefs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: This paper demonstrates notable theoretical depth and clarity by providing a rigorous formalization of ME-POMDPs and AB-POMDPs, including precise definitions, clear reductions, and systematic connections to POSGs. The foundational theoretical results are well-motivated, and the approach of reducing robust planning problems to standard constructs such as POSGs and restricted POMDP variants enables both conceptual clarity and algorithmic leverage.\n\n**Weaknesses**\n1. The paper does not provide a theoretical analysis of time or space complexity with respect to key parameters (e.g., state/action space size, number of environments, planning horizon).\n2. Results are reported on randomly generated environments without variance, error bars, or confidence intervals, limiting the interpretation of empirical robustness.\n3. There is no in-depth discussion of how randomness in benchmark generation may affect results, nor is there a systematic exploration of whether variations between random seeds could lead to significantly different conclusions.\n\nQUESTIONS:\n1. The experimental tables show that convergence time can grow rapidly with the number of environments and problem size, sometimes resulting in incomplete runs. Can the authors provide theoretical or empirical guidance on the expected runtime as a function of the number of environments, states, and actions?\n2. How well does the approach scale as the state, action, or environment space increases?\n3. How do the proposed algorithms perform empirically compared to robust POMDP methods that use continuous or convex uncertainty sets, or alternative robust optimization frameworks?\n4. On line 19, the equation \"$H+1=H$\" requires clarification. On line 99, the superscript $\\star$ is not defined.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about multi-environment POMDPs (ME-POMDPs), a version of POMDPs, in which the transition model, observation model or reward function is uncertain, in that there are a finite number of possible models. It is concerned with the setting without prior knowledge about which environment is active and tries to find policies that maximize the value in the worst case. This is achieved using an algorithm that transforms ME-POMDPs into so-called adversarial-belief POMDPs (AB-POMDPs) and solves the resulting AB-POMDPs using linear programming. The algorithm is evaluated on two benchmarks and compared to a baseline that solves the individual POMDP models without taking robustness into account.\n\nSTRENGTHS AND WEAKNESSES:\nI appreciate that the authors have tried to present very technically dense material in a readable fashion. I found the introduction well-written and it seems to cover relevant related work. The section on POMDPs and alpha vectors provides useful background for readers not familiar with these concepts. However, after the introduction, the paper often feels less clear and I was at times confused about some points that seem quite central to the paper (see questions below), concerning both the results and the method. Furthermore, the paper lacks visualizations, intuitive explanations, or defnitions of some of the notation. For these reason, the rest of my review consists mostly of suggestions for how the paper could be made clearer and more accessible and of clarification questions.\n\n1. The paper contains no figures at all. This is in some sense understandable for a theory paper, but I think the authors miss a few opportunities to present their work in a more accessible fashion. For example, the results would be more easily digestible if they were shown using a few clear figures instead of a whole page of tables. Also, a graphic illustrating the relationship between ME-POMDPs, PO-MEMDPs, MO-POMDPs and AB-POMDPs would be highly appreciated, as it was easy to lose track of how the concepts fit together.\n2. I did not check the proofs in detail, as that is outside my area of expertise, but it seems to me that the theorems in Section 4 and 5 could benefit from some more intuitive descriptions. Before or after going into mathematical jargon, it would be nice to provide some intuition about what is actually being stated in natural language. \n3. Symbols or equations are sometimes used before being properly defined or introduced. I will give a few examples, but this is not an exhaustive list. In Theorem 1, some of the symbols are not explained (e.g. ⊥, ⊤). On page 7, $y$ is used before being defined. The linear programs (3) and (4) are already referenced in Section 5.1, but are defined only Section 5.2, which in turn refers to \"the LPs of Section 5.1\".\n4. The discussion of the results is very sparse and not easy to follow. See also my questions below.\n    a. Section \"Q1. Scalability.\" offers no discussion of the three different times of models (PO-MEMDP, MO-POMDP, and ME-POMDP).\n    b. Statements like \"the randomly generated environments have a great effect on the difficulty of the problems\" would benefit from concrete pointers to numbers particular tables.\n\nMinor points:\n- The notation (A x Z)* for the history of states and observations makes sense but could be made more explicit that that is what is means, because in other contexts the authors use the star to indicate an optimal policy.\n- Ordering the references by occurence would make for easier reading\n\nQUESTIONS:\n- Can't we write any ME-POMDP as a single POMDP, which includes an unobserved discrete variable indicating the true model, which does not change during an episode? If that is the case: What makes the current approach more efficient than the naive approach of solving that POMDP directly? And why was the naive approach not considered as a baseline?\n- I understood the solution technique is to turn an ME-POMDP into an AB-POMDP and then solve that using AB-HSVI. If that is the case, then how do I interpret the comparison in Table 3 between ME-POMDPs and AB-POMDPs, both solved with AB-HSVI?\n- In a POSG, why does the fully observing player's policy take the history of states, both players actions and observations as the input?\n- In response to checklist item 7 (statistical significance), the authors state \"The algorithm is deterministic. It does not use any randomization during runtime, and hence will produce the same result upon running it twice. Note, we do use randomization to generate the models that we run our algorithm on, but then the models are fixed.\" This argument is not convincing to me. Wouldn't we still want to know if the algorithm performs well on some distribution of models rather than on a single randomly generated model?\n\nLIMITATIONS:\nThe paper does not discuss any limitations or potential societal impacts. In my opinion, a discussion of societal impacts is not necessary for this kind of basic theoretical research, but a discussion of limitations would be appreciated to put the approach into context.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of robust planning in POMDPs under discrete model uncertainty, i.e., Multi-Environment POMDPs (ME-POMDPs), where the goal is to compute a single policy that maximizes the worst-case expected reward across a finite set of possible models. They make a theoretical contribution by introducing Adversarial-Belief POMDPs (AB-POMDPs), proving that ME-POMDPs can be reduced to this instance. Algorithmically, they propose the Adversarial-Belief Heuristic Search Value Iteration (AB-HSVI), which integrates linear programming into the popular HSVI point-based solver to adversarially select the belief for value function updates. The effectiveness and scalability of this approach are demonstrated through experiments on POMDP benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper addresses a well-defined and practically important problem. The assumption that exact model parameters for a POMDP are known is often a bottleneck for real-world applications. The ME-POMDP formulation is an intuitive and relevant model for handling discrete model uncertainty, and the paper's focus on robust, worst-case optimization is appropriate for high-stakes decision-making.\n2. The paper presents theoretical analysis. The introduction of AB-POMDPs as a unifying framework looks sound. The theorems that formally establish reductions and equivalences between ME-POMDPs, AB-POMDPs, and one-sided POSGs (Theorems 1-4) can be a good contribution.\n3. The proposed AB-HSVI is a technically sound approximate solution method, which builds upon the foundation of HSVI, a popular and effective POMDP algorithm. The integration of a linear program to find the worst-case belief is a direct and elegant way to handle the adversarial nature of the problem within the value iteration framework.\n\n\nWeaknesses\n1. While the experiments analyze scalability, they also confirm that it is a primary limitation. The computational time increases significantly with the number of environments or the problem size. For problems with large state/action space or a large number of potential environments, the current approach may be intractable.\n2. The proposed method is rooted in traditional POMDP solvers that rely on maintaining an explicit PWLC value function (a set of alpha-vectors). This representation can become a bottleneck in POMDPs with very large state or observation spaces, which is where modern deep reinforcement learning methods are often applied. The paper does not discuss how these ideas might be extended to work with nonlinear function approximators (e.g., neural networks), or online planing (e.g., POMCP), which limits its applicability to problems outside the scope of classic point-based solvers.\n3. They correctly note in the checklist that their algorithm is deterministic. However, the problem instances themselves are generated via randomization. The empirical results would be stronger if, for each configuration (e.g., RS_3,1,3), the authors had generated a small number of random instances (e.g., 5-10) and reported the mean and standard deviation of the results. This would provide a better sense of the method's performance consistency across different models of the same size, rather than just on a single random instance.\n\nQUESTIONS:\n1. The scalability of AB-HSVI with respect to the number of environments n and the state-space size |S| appears to be the main bottleneck. While the current work provides a crucial theoretical and algorithmic foundation, have you considered any techniques to mitigate this for larger problems? For instance:\n- Could one potentially prune redundant or clearly dominated environments from the set before solving to reduce n?\n- Could the core idea of adversarial belief selection be integrated with more scalable online planning methods, such as POMCP, where a complete value function is not maintained?\n2. The paper focuses on a classic, point-based solver. The theoretical formulation of ME-POMDPs and AB-POMDPs, however, seems general and valuable. Do you see a path to applying these core ideas in settings where PWLC value functions are intractable? For example, could this be formulated as a two-player zero-sum game solvable with deep reinforcement learning methods, where one agent (the policy) plays against another (an adversary selecting the environment/belief)?\n3. The experimental results are reported on a single, randomly generated instance for each problem configuration. This makes it difficult to assess the general performance and variance of the algorithm. Could you comment on how representative you believe these single runs are? For the final version of the paper, I would encourage reporting results averaged over multiple random seeds (e.g., 5-10 runs per configuration), including standard deviations, to provide a more robust empirical validation.\n4. The paper provides a proof connecting AB-POMDPs to one-sided POSGs. Does this theoretical connection provide more than just a formal link? For instance, could specific insights or algorithms from the rich POSG literature be leveraged to develop alternative or more efficient solution methods for AB-POMDPs in the future?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-16"}