Title: On Evaluating Policies for Robust POMDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robust POMDPs

Policy evaluation

Planning under uncertainty
Abstract: Robust partially observable Markov decision processes (RPOMDPs) model sequential decision-making problems under partial observability, where an agent must be robust against a range of dynamics. RPOMDPs can be viewed as a two-player game between an agent, who selects actions, and nature, who adversarially selects the dynamics. Evaluating an agent policy requires finding an adversarial nature
policy, which is computationally challenging. In this paper, we advance the evaluation of agent policies for RPOMDPs in three ways. First, we discuss suitable benchmarks. We observe that for some RPOMDPs, an optimal agent policy can be found by considering only subsets of nature policies, making them easier to solve. We formalize this concept of solvability and construct three benchmarks that are only solvable for expressive sets of nature policies. Second, we describe a new method to evaluate agent policies for RPOMDPs by solving an equivalent MDP. Third, we lift two well-known upper bounds from POMDPs to RPOMDPs, which can be used to efficiently approximate the optimality gap of a policy and serve as baselines. Our experimental evaluation shows that (1) our proposed benchmarks cannot be solved by assuming naive nature policies, (2) our method of evaluating policies is accurate, and (3) the upper bounds provide solid baselines for evaluation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper constructs infrastructure for RPOMDP research. In particular, the authors design benchmarks for testing RPOMDP algorithms, baselines to compare those algorithms against, and a method for computing the performance of arbitrary policies (which is non-trivial in *R*POMDPs, because one has to find the POMDP that is worst for the policy in question). In order to do the latter efficiently, they prove a result showing that the worst-case POMDP must be an extreme element of the convex set of POMDPs that compose the RPOMDP. They show why the benchmarks they propose are better, in the sense of being less easily solved by naive methods, than common existing benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper lays out it its aims clearly, those aims are compelling and important, and it accomplishes those aims. In my view, the prevailing feeling in RL that it's enough to study MDPs has hamstrung the field; studying RPOMDPs seems necessary for the field of RL to (safely) make contact with reality.\n\nWeaknesses: There were a few points where I was confused by the mathematical presentation.\n\nIt seems that the RPOMDP field is immature enough that we cannot really complex environments as benchmarks. I suppose that weakens this paper, since everything is still fairly toy. However, I suppose we build to harder benchmark environments when we have better solvers, and we get better solvers by having better benchmark environments. \n\nConfusions with the presentation:\n\n> \"decision variables v ∈ Vars,a only affect transitions and observations in the state-action pair ⟨s, a⟩\"\n\nI don't understand what this means.\n\n> Lastly, we assume P is convex, which is a common assumption for tractability reasons \n\nP is a function from U to probability distributions. Absent a total order over probability distributions, what does it mean to say P is convex? Do you mean the set {P(u) : u \\in U} is convex? Likewise, is Extremes(P) really Extremes({P(u) : u \\in U})?\n\n> V n = minθ∈Θ maxπ∈Π V π,θ \n\nWhy do you ever define this quantity or talk about the Nash equilibrium? In the rest of the paper, as far as I can tell, the min over environments is always inside the max over policies.\n\n> Intuitively, if a model is Θ-bar-solvable, then we can find an optimal agent policy against Θ by finding the optimal policy against only Θ-bar ⊆ Θ. \n\nI don't follow. If it's theta bar solvable, then there exist policies are that are optimal both against theta and theta bar, but it doesn't mean that all policies that are optimal against theta bar are also optimal against theta. Do you mean \"we can potentially find an optimal policy\"?\n\n> Intuitively, an RPOMDP with a stationary nature policy represents a POMDP. \n\nIsn't the objective different? If you represent an RPOMDP with a stationary nature policy as a POMDP, then a solver should attempt to optimize by taking an expectation over the uncertain feature of the state that corresponds to the nature policy. In an RPOMDP, the agent is not taking an expectation over the uncertain feature of the state (the fixed nature policy), but rather minimizing it.\n\n> Definition 3.\n\nShould R_n(<s, x, a>) = *negative* R(s, a)?\n\n> Def. 4\n\n\"b\" is not defined (although I knew what it meant).\n\nQUESTIONS:\nYou consider the setting where nature is adversarially picking environment parameters, and nature can change those parameters every timestep, after looking at the agent's past actions. How does that differ from the following setting: nature looks as the agent's whole policy, and then picks (permanent) environment parameters once?\n\nLIMITATIONS:\nYes, but I would also propose adding a note that the benchmarks are fairly toy. You can certainly defend this, but it may be good to mention.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

comments: SUMMARY:
In this work, authors present a new evaluation pipeline for RPOMDPs that itself has been an understudied problem. As authors describe RPOMDPs are naturally described as a two-agent games where agent policy picks and action and then nature policy adversarially picks the dynamics. Resulting in worst possible dynamics for a given policy. Authors describe three solvability classes for nature policy: trivial, naive and stationary. They define that such nature policies to be too easy and describe benchmarks that are _not_ in those two solvability classes. Best response (BR) is found to be a good substitute worst-case nature policy. This results in a nature MDP and is solvable using off-the-shelf tools.

STRENGTHS AND WEAKNESSES:
Strengths:
- Understudied problem is put into a strong theoretical and practical footing. 
- Idea of defining naive nature policies as too simple and showing that benchmark policies are not in the naive category gives theoretical grounding on the using benchmark policies instead of naive policies in comparisons. 
- Connection from RPOMDP to MARL (via best response natural policy) is brilliant, even though obvious idea. 
- Experiments also gave new information about the relative strengths of different environments and the usefulness of RFIB as an adequate baseline. 

Weaknesses: 
- How we can say that the only categories of naive solvability in Section 3.1 are these three? Can there be possible other possible policies for \theta. And if it is possible to have other policies that can be considered naive, then does it somehow interact with the Theorems in Section 3.2?

QUESTIONS:
-

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a set of intuitions, evaluation methods, algorithms and results on the problem of evaluating policies of agents in Robust POMDPs. They propose new environments to use as benchmarks that satisfy some intuitive (and some formal) properties desirable for the evaluation of policies, they justify why it should be enough to evaluate the policies against nature best response (and not agains nature’s optimal policy) and provide two extensions of existing value iteration algorithms applied to RPOMDPs. Finally they run the algorithms in the proposed benchmarks and discuss the results and implications.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The problem is well motivated, and the contributions are clear.\n- The paper is novel and covers an existing gap in the RPOMDP literature.\n- The paper is clear, easy to follow and introduces interesting ideas.\n\n## Weaknesses\n- Some decisions or results in the paper are not well justified.\n- Some more discussion is needed to add nuance to the relevance of the contributions.\n- The implications and scalability of the work are not really clear.\n\nQUESTIONS:\n1. The intuition behind solvability is not very obvious. While I see that it provides an argument for finding optimal policies in smaller subsets of environment parameters and ensuring global optimality, I do not really see how to use it to derive these policies, and I also do not follow how does it apply to the idea of evaluating RPOMDP policies against best response parameters. Can the authors expand on this, and justify the need for considering the given solvability definition?\n2. Similar to the question above, the idea of naive solvability seems arbitrary (which the authors seem to agree as per their own assessment). Furthermore, the authors propose this taxonomy of solvability in terms of the nature policies, but provide results about solvability for specific nature policies that are not necessarily the best response. How does the concept of solvability link to the proposed evaluation method?\n3. The authors touch slightly on the topic, but I would like to further understand the utility in practice of RPOMDPs. As in other minmax, zero sum game frameworks, obtained policies can be extremely conservative as some (very unlikely) particular nature parameter may force the optimal agent policy to act very suboptimally in other regions of the parameter space. As this work provides benchmarking and evaluation baselines for RPOMDPs, what problems do the authors consider it can be useful for?\n4. Regarding the scalability of solving the nature MDP, the authors state that MCTS fails to provide valid nature MDP solutions in larger RPOMDPs. Is there a way of addressing this, or is it an artefact of the complexity of the resulting MDPs?\n5. Is the fact that Tiger and HeavenOrHell are 'naively solvable' a problem for general evaluation of RPOMDPs?\n\nLIMITATIONS:\nThe authors address briefly the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper addresses the evaluation of agent policies in Robust Partially Observable Markov Decision Processes (RPOMDPs), a framework for sequential decision-making under partial observability and model uncertainty. RPOMDPs are conceptualized as two-player games where an agent selects actions, and "nature" adversarially chooses the system dynamics to minimize the agent's value. The core challenge lies in evaluating an agent policy, which necessitates finding a worst-case adversarial nature policy – a task that is computationally demanding.

STRENGTHS AND WEAKNESSES:
Strengths

1. Interesting take on RPOMDP research, focussing primarily on evaluation of policies. 
2. Considering different uncertainty sets, rectangular and convex.

Weaknesses

1. Impact of the approximate nature of Montecarlo approaches is not mentioned. Given that approximate policies are found with MCTS type approaches, aren't there gaps in evaluation?
2. It would seem that the current approach can have scalability issues especially with larger environments (e.g., ALOHA, HEALTHDETECTION)

QUESTIONS:
1. Would help if the authors can provide inputs on scalability?
2. Can the authors provide inputs on generalisability of benchmarks beyond solvability, especially in real world problems?
3. Are there any real world problems where this work will be useful?
4. Can the provided approaches be extended beyond rectangular uncertainty sets?
5. Theorem 3 and its discussion mention that best response nature policy is simpler than optimal nature policy. Can authors provide more quantitative or concrete examples of how this simplicity is important in practice?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-04