Title: value: Belief-State Query Policies for User-Aligned POMDPs
Authors: value: Daniel Richard Bramblett
Siddharth Srivastava
Venue: NeurIPS 2024
Keywords: value: POMDPs
sequential decision making
user-preferences in POMDPs

Abstract: value: Planning in real-world settings often entails addressing partial observability while aligning with users' requirements. We present a novel framework for expressing users' constraints and preferences about agent behavior in a partially observable setting using parameterized belief-state query (BSQ) policies in the setting of goal-oriented partially observable Markov decision processes (gPOMDPs). We present the first formal analysis of such constraints and prove that while the expected cost function of a parameterized BSQ policy w.r.t its parameters is not convex, it is piecewise constant and yields an implicit discrete parameter search space that is finite for finite horizons. This theoretical result leads to novel algorithms that optimize gPOMDP agent behavior with guaranteed user alignment. Analysis proves that our algorithms converge to the optimal user-aligned behavior in the limit. Empirical results show that parameterized BSQ policies provide a computationally feasible approach for user-aligned planning in partially observable settings.


Review discussion:
In this work, to ensure compliance with user's preference, the policies are directly parameterized by the preferences (formalized by boolean formulas). This approach is quite different from the more well-known approach of including preferences in the objective function, and it is worth investigating.
(1) Readability: This paper introduces many heavy notations. The definitions (of BSQ, policies, etc.) are pretty abstract.
(2) The policies are defined using BSQ. The expressiveness of such a policy class is not well-justified: it is not clear whether this policy class includes all policies that comply with the agent's preferences.
(3) The experiments are performed on several different environments, but the uniformly random baseline (RCompliant) is probably too weak.
N/A
N/A
No ethics review needed.
Thank you for your feedback.
**Weakness 2:** We assume that a policy must comply with a BSQ preference to align with the user’s intentions.
Therefore, we can prove that all aligning policies are included in the policy class.
**Weakness 3:** The baseline RCompliant is there to (1) show the BSQ preferences alone are insufficient for solving the evaluation problems and (2) to evaluate how much the solution of PRS can improve the solution.
We did run experiments searching for a competitive baseline that we excluded from the paper. We found that a brute-force grid search requires significantly more time than PRS because evaluating the expected cost of a point in the parameter space requires numerous samples. For smarter parameter optimization algorithms (Nelder Mead and particle swarm), we found that the sampling cost, along with optimizing over a non-convex continuous parameter space that lacks a gradient, made these algorithms too unreliable. POMDP solvers would be an unfair comparison to use as a baseline since PRS receives additional information on the problem.
Otherwise, there is no existing work that we can directly compare against. We will add experiments and analyses on this into the supplementary for the final version.
Thank you for your response. I will keep my score and remain positive about this paper.
Thank you again for your review and being optimistic about our paper.
- The problem of user-preference compliant policy computation in a partially observable setting is interesting. It allows specifying additional objectives for the agent that cannot be directly expressed with LTL, as the agent cannot observe the complete state of the model.
- The definition of BSQs allows the specification of user preferences in more natural language, making it accessible to non-experts.
- The paper removes limitations from previous works: the difficulty of guarantees in reward calibration approaches, the need for past traces in the works of Mazzi et al., and the lack of a formal analysis in the work of Srivastava et al.
-- I raise my score after the rebuttal of the authors with an additional evaluation of the method --
- The experimental comparison is limited. It would have been interesting to see how the approach compares to the reward calibration approaches that do not guarantee preference compliance or the referenced works of Mazzi et al. There is no mention of why this comparison would not be possible.
Another interesting comparison would be to compare to a non-BSQ-compliant policy and analyze the difference in probability of reaching the goal states and satisfying the BSQ. Such a comparison would give insight into the gains and losses of being BSQ-compliant.
- Information about the sizes of the problems used for the experiments is missing, making it difficult to estimate the strength of the approach.
- The definition of the goal-oriented POMDPs is hard to follow. It is still unclear to me where the constant symbols are used, why the states S are defined as valuations for Vf, how the set of state variables for a function F follows from instantiations of F, and what the type of F is. The second part, starting with the definition of the transition function, is clear. The part before that is not.
I list a couple of questions related to these weaknesses, and I am happy to reconsider the score depending on the answers.
Minor comments:
- The paper mentions relational goal-oriented POMDPs at the start of section 3, but afterward, only goal-oriented POMDPs are explained.
- Grounded state is mentioned without explanation (L59). This term is not familiar to me.
- Figure 2a contains a lot of information introduced in different parts of the text. Perhaps this can be split into two iterative figures for clarity.
- I think the input theta for the compound BSQ in lemma 1 (L220) should be a capital theta.
- Definition 9 (L204) contains a curly P to many.
- The R with superscript 1 and subscript 0 (L503) was not introduced.
- The union between the expected cost of a partition and the expected cost of a leaf in Algorithm 1 line 10 seems a bit weird. Is this the correct operator? If so, a small explanation would be helpful.
1. What is the motivation for minimizing the cost instead of maximizing the probability of reaching the goal states?
2. Is there a reason for not comparing your approach to other existing approaches? Can you give some intution on how the approaches would compare to each other?
3. What are the sizes of the problems used for the experimental evaluation? What makes them challenging (L327)?
4. Why is the comparison sign not an input for the BSQ?
5. Definition 1 (goal-oriented POMDPs) uses unintroduced symbols (the non-curly F, O, and T). Is this correct? If so, what do they mean?
6. Can you clarify Definition 1 (goal-oriented POMDP)? See weaknesses for the unclear parts.
The authors addressed the limitations of their work, discussing the impact on their results.
No ethics review needed.
Thank you for the feedback and questions.
**Q1:** Maximizing the probability of reaching the goal does not factor in the time required for goal completion.
Therefore, the optimal strategy would be to use the maximum allowed time for gathering information before choosing to reach the goal. By minimizing the time to achieve the goal, we balance the probability of goal achievement and the average goal completion time without reward engineering.
**Q2:**  We experimented with various algorithms for comparing against PRS that are not included in the paper, which can be categorized into parameter search and unconstrained POMDP solvers. The main challenge with parameter search is that we are optimizing over a non-convex continuous search space that does not have a gradient.
Additionally, we observed that evaluating the expected cost accurately requires numerous samples of the same point and is computationally expensive.
For example, evaluating a parameter value in Lane Merger by taking 200 samples on average requires 22.07 seconds, and the average coefficient of variation of the expected cost is 135.88%, meaning more samples are needed.
Therefore, we explored using more advanced parameter search algorithms like Nelder Mead and particle swarm optimization.
However, they were inconsistent, with Nelder Mead having a high expected cost coefficient of variation of 78.8% on a toy example. PRS is taking advantage of the structure of the expected cost function, allowing it to consistently and quickly converge on a solution.
For unconstrained POMDP solvers, we evaluated POMCP [Silver and Veness, 2010] and DESPOT [Somani et al., 2013]. POMCP failed to converge. DESPOT was unable to run on Lane Merger. On Graph Rock Sample, DESPOT only reached the goal 0.5% of the time due to a farther goal and sparse rewards. On Spaceship Repair, DESPOT achieved a 7.57% higher goal achievement rate and 13.3% lower expected cost. On Store Visit, DESPOT achieved a 0.58% higher goal achievement rate and 20.0% lower expected cost. We evaluated the alignment with the BSQ preferences and found that DESPOT was compliant 7.3% and 0% of the time on Spaceship Repair and Store Visit, respectively.
Therefore, we can conclude that (1) BSQ preferences can help solve problems that unconstrained solvers cannot and (2) the unconstrained solver can perform better on some problems, but their policies are unlikely to comply with the user’s intentions.
Mazzi et al.'s work solves a different problem of normalizing behavior, so we could not compare our approach to theirs.
We will include these experiments and analyses in the supplementary for the final version of the paper.
**Q3:** The size of the evaluation problems are as follows:
* Spaceship Repair: 44 states, 3 actions, 4 observations
* Lane Merger: 307,040 states, 4 actions, 32 observations
* Store Visit: 30 states, 5 actions, 27 observations
* Graph Rock Sample: 128 states, 13 actions, 3 observations
These problems are challenging due to the long horizon and safety risks. Evaluating for a horizon of 100 makes it intractable to explore the entire strategy tree (Definition 6). Each problem contains potential risks, such as merging into another car or entering unsafe areas, which would prevent the agent from completing the goal. As these risks are partially observable and relate to the goal, no policy guarantees goal completion.
Therefore, the agent must balance the trade-off between taking longer to reach the goal (to gather more information) and the associated risks.
**Q4:** We excluded the comparison operator as an input into a BSQ since it is constant after initialization. Note only the belief state and the parameter values vary after. To remain consistent, we will either remove the first-order logic formula as an input for the same reason or add the comparison operator as an input. Thank you for pointing this out.
**Q5:** In Definition 1, non-curly $F$ and $T$ are syntax errors and have been corrected. Non-curly $O$ is the set of objects, which we will clarify in the next version of the paper. Thank you for pointing these errors out.
**Q6:** Our gPOMDP definition draws upon existing literature on expressive languages for specifying POMDPs [Sanner, 2010, Srivastava, 2012]. This representation allows us to specify large POMDPs by reusing functions from the set of functions $\mathcal{F}$ and instantiating them from the set of objects $O$.
For example, whether each rock is safe to sample in Graph Rock Sample can be represented as $safe(rock_i)$. To build the set of states $\mathcal{S}$, we first construct all the state variables $\mathcal{V}_f$, which are all the combinations of function parameters using $O$. Each state in $\mathcal{S}$ represents a set of values setting each state variable.
**References:**
Sanner, Scott. "Relational dynamic influence diagram language (rddl): Language description." Unpublished ms. Australian National University 32 (2010): 27.
Silver, David, and Joel Veness. "Monte-Carlo planning in large POMDPs." Advances in neural information processing systems 23 (2010).
Somani, Adhiraj, et al. "DESPOT: Online POMDP planning with regularization." Advances in neural information processing systems 26 (2013).
Srivastava, Siddharth, et al.
First-order open-universe POMDPs: Formulation and algorithms. Technical report, EECS-2013-243, EECS Department, UC Berkeley, 2012.
I thank the authors for their efforts in answering my questions. I urge the authors to add part of the additional evaluation to the main body of the paper. Trusting that this will happen, I will raise my score to 6.
Thank you for your support and increasing the score. We are glad our response helped answer your questions. We will add further evaluation to the final version of the main paper, as you suggested.
BSQ polices haven't received attention for some time, so it's good to see a paper developing the theory and practice of BSQ policy evaluation and optimization. My concerns about the framing and motivation around "preferences" aside (see Weaknesses), the paper is technically sound, and elucidates the structure of the BSQ policy value function in a much deeper way than previously studied by Srivastava et al (2012). Given the piecewise constant nature of this value function, the partition-based policy optimization algorithm is a natural but principled approach. The proofs that I checked appear to be correct, and the experiments indicate that convergence to the optimal policy is indeed achieved. Insofar as BSQ policies provide a tractable and interpretable alternative to more direct approaches to solving POMDPs (though to me this remains to be seen), this paper is a step towards improving their practical applicability, and is likely of interest to others working on POMDPs.
**References:**
Srivastava, S., Cheng, X., Russell, S., & Pfeffer, A. (2012).
First-order open-universe POMDPs: Formulation and algorithms. Technical report, EECS-2013-243, EECS Department, UC Berkeley.
My concerns about this paper center around two main issues (i) motivation and framing around "preferences" (ii) the strength of the evaluation.
**Motivation and Framing**
Building upon the concept of BSQ policies introduced in Srivastava et al (2012), this paper introduces the concept of "BSQ preferences", which are basically BSQ policies without fixed probability thresholds $\theta$.
In other words, a "BSQ preference" is just a parameterized family of BSQ policies -- a *policy class* -- which can alternatively be interpreted as a *policy constraint* (Hasanbeig et al, 2018), a *policy sketch* (Andreas et al, 2017; Verma et al, 2018) or a *partial (policy) program* (Andre & Russell, 2002; Hahn et al, 2022) where branch conditions take the form of belief-state queries.
Given these other interpretations of what a "BSQ preference" is -- which, in my opinion, are much more natural -- it is strange to me that the paper adopts the terminology of "preferences" instead. This leads to a motivation in the Introduction that feels quite at odds with the actual technical content: While it's undoubtedly the case that reward engineering is difficult, especially in partially observable settings, it's far from obvious that specifying BSQ policy constraints actually address these specification challenges.
Indeed, the paper does not study whether humans find it easier to specify BSQ policy constraints rather than (a) rewards or (b) the more common notion of preferences as pairwise comparisons (used in e.g. RLHF).
I understand that the authors are building upon the literature of "planning with preferences" in the way they use the term "preferences", but IMO that choice of terminology was always unfortunate*, and people should really have called stuff like LTL specifications "constraints" rather than "preferences". This is especially important if the authors are hoping to learn BSQ policies from human preference data in the future (as they suggest in the Discussion) --- it will be very confusing for the term "preference" to both refer to a class of (BSQ) policies, and also the more standard notion of a pairwise comparison between trajectories / choices.
At the very least, I think the authors should discuss the alternative interpretations of a "BSQ preference" that I have listed above -- and cite the relevant literature -- though I think it would be ideal to just move away from the "preference" terminology altogether.
(*Note that even in the "planning with preferences" literature, a preference specification is understood to introduce an ordering relation over plans, in accordance with the standard notion of preferences. This notion of an ordering relation isn't discussed by the definition of a "BSQ preference" in this paper. I would be happier with the "preference" terminology if ordering relations were actually discussed, and if the paper said something about how to handle *multiple* BSQ policy constraints, each with different priorities, since this would introduce a richer ordering relation than the trivial one that prefers all policies that satisfy the constraint.)
**Strength of Evaluation**
While the Partition Refinement Search (PRS) algorithm is well explained and motivated, the evaluation is unfortunately quite limited, only comparing PRS variants to a baseline where the probability thresholds are sampled at random, with no policy optimization at all. I think minimally, there should be some kind of grid-search baseline similar to the one in Srivastava et al (2012), which more clearly shows the benefits of a search strategy based on partition refinement, compared to one which enumerates over the full space of thresholds until the time limit is up. I think there should also be a random search baseline --- i.e. one which repeatedly samples random threshold values, evaluates the resulting policies, and returns the best threshold combination so far. An even stronger baseline would be some kind of coordinate-ascent or Gibbs sampling algorithm, which optimizes one threshold at a time (via gridding or sampling) while keeping the other thresholds fixed.
The above suggestions only pertain to policy optimization under the assumption that we know for sure we want to satisfy the constraint induced by the BSQ policy class.
But if the authors really want to show the benefits of BSQ policies as either (i) a way of specifying user constraints or (ii) a way of avoiding downsides of unconstrained policy optimization in POMDPs, then I think more experiments are in order.
For example, to show that BSQ policy constraints might be a more concise way of communicating user requirements than alternatives, it would be great to see a comparison with reward engineering, or with fitting and then optimizing a reward model learned from pairwise trajectory preferences (as in RLHF). I think this would be especially important if the authors want to keep the "preference" oriented framing of the paper.
And to show that BSQ policies have practical engineering value over direct POMDP solution approaches like POMCP or PBVI, it would be good to find some way of comparing the solution quality of PRS vs. those other approaches --- for example, it might be the case that BSQ policy optimization actually produces better outcomes in less solution time because it allows users/developers to constrain the space of policies (i.e. provide "policy guidance"), similar to the benefits of HTN planning over classical planning.
**References:**
Srivastava, S., Cheng, X., Russell, S., & Pfeffer, A. (2012).
First-order open-universe POMDPs: Formulation and algorithms. Technical report, EECS-2013-243, EECS Department, UC Berkeley.
Hasanbeig, M., Abate, A., & Kroening, D. (2018). Logically-constrained reinforcement learning. arXiv preprint arXiv:1801.08099.
Andreas, J., Klein, D., & Levine, S. (2017, July). Modular multitask reinforcement learning with policy sketches. In International conference on machine learning (pp. 166-175). PMLR.
Verma, A., Murali, V., Singh, R., Kohli, P., & Chaudhuri, S. (2018, July). Programmatically interpretable reinforcement learning. In International Conference on Machine Learning (pp. 5045-5054). PMLR.
Andre, D., & Russell, S. J. (2002, July). State abstraction for programmable reinforcement learning agents. In AAAI/IAAI (pp. 119-125).
Hahn, E. M., Perez, M., Schewe, S., Somenzi, F., Trivedi, A., & Wojtczak, D. (2022). Recursive reinforcement learning. Advances in Neural Information Processing Systems, 35, 35519-35532.
POST-REBUTTAL: I have adjusted my score to a 6 given the additional baseline comparisons in the author's rebuttal, which show that PRS outperforms less naive parameter search methods, as well as unconstrained policy optimization methods like POMCP and DESPOT.
In light of the issues with framing and evaluation, I am currently recommending a paper score of 4. If only the framing issues existed, I would give a score of 5.
But unfortunately the combination of issues leads me to think this paper isn't ready for publication in NeurIPS yet, even though the theory portion is interesting and sound.
To address these issues, here are my recommendations / questions:
- Consider renaming "BSQ preference" to "BSQ policy class/sketch/constraint". Minimally, please discuss these alternative interpretations of a "BSQ preference".
- If the "BSQ preference" terminology and framing is kept, there should be more rhetorical and empirical justification for why BSQ policies are a useful way of specifying user preferences, and more discussion of the notion of preferences as ordering relations.
- Why is PRS not compared against stronger policy optimization baselines, e.g. grid search, random search, or coordinate ascent?
- Consider comparing PRS against unconstrained policy optimization to more clearly show the benefits of BSQ policy constraints as a form of policy guidance.
- If the "BSQ preference" framing is kept, consider comparing user specified BSQ preferences against user specified reward functions or reward models fitted to pairwise comparison data.
Minor Comments:
- In the description of PRS, the terms "partition" and "optimal partition" seem to be used in place of "interval" / "optimal interval", which led to some confusion when I was reading that section. I think the authors should try to keep to the standard notion of "partition" as a way of splitting a set into mutually exclusive and exhaustive subsets, and use "interval" to refer to one of those (continguous) subsets.
A number of important limitations are discussed in Appendix H. There are other limitations that I've pointed out above that should be either addressed or discussed.
No ethics review needed.
Thank you for your insightful comments and for appreciating the strength of technical contributions in the paper. We agree that the BSQ framework deserves a much-needed deeper analysis, and the framework developed here takes key steps towards it. We believe that presenting this new analytical paradigm and the surprising result about piecewise-constant parameter spaces to a broad community such as NeurIPS will enable faster theoretical and algorithmic development in this direction.
**Q1 and Q2:** We agree with you on both points, that (a) the notion of preferences used in this paper has been established in prior alliteration in AI, and (b) this existing choice of terminology is rather unfortunate due to the reasons you elaborate. We used it to be consistent in situating our work and acknowledging past scholarship.
Partial programs [Andres et al., 2017], Alisp programs [Marthi et al., 2005], and this work represent different evolutions of the idea of using partial specifications, and all these directions draw upon Hierarchical abstract machines [Parr and Russell, 1997], albeit in fully observable settings.
In fact, BSQ templates can express finite-state machine based partial specifications as well. They also specify an implicit ordering induced by the if-then-else structure.  Considering this broader context, we will use the term BSQ guidelines instead of BSQ preferences to better reflect this nature.
We respectfully maintain that terminological choices aside, this work presents a novel technical contribution whose dissemination at this stage will greatly help advance research on the topic.
**Q3:** We extensively evaluated for competitive baselines, including methods such as grid-search on preliminary simpler toy problems. PRS is significantly faster than grid-search making it challenging to compare the two on the same time scale.
For example, constructing the non-convex example of Spaceship Repair shown in Figure 1(c) is equivalent to doing a brute-force search, and it took 159.65 minutes to evaluate all parameter combinations with 0.01 precision.
In comparison, PRS converged on a longer horizon version of Spaceship Repair within 4 minutes (Figure 4).
Furthermore, to match the performance of PRS would require significantly higher precision (e.g., PRS consistently found the solution $0.600000024 < \Theta_1 ≤ 0.600000143$ and $0.49999994 < \Theta_2 ≤ 0.50000006$ for Spaceship Repair).
During our evaluation, we observed that estimating the expected cost is challenging due to the high variability in sampling noise, necessitating numerous samples per point, which is computationally expensive.
For instance, evaluating a random point in the Lane Merger problem with 200 samples takes, on average, 22.07 seconds. The average coefficient of variation for the expected cost for this point is 135.88%, suggesting that even more samples are required.
To address this, we also tried Nelder Mead and particle swarm optimization, which are significantly more advanced than vanilla parameter search. PRS outperformed them both, with the better of the two, Nelder Mead, having an expected cost coefficient of variation of 78.8% on a toy example, 11 times higher than PRS.
These experiments and analyses are not in the paper, but we will add them to the supplementary for the final version.
**Q4:** We evaluated unconstrained policy optimization using both POMCP [Silver and Veness, 2010] and DESPOT [Somani et al., 2013]. POMCP failed to converge. DESPOT could not run on Lane Merger and only reached the goal 0.5% of the time in Graph Rock Sample. This supports your hypothesis that BSQ templates can help guide the solver. On the other two problems, DESPOT achieved lower expected costs (20.0% and 13.3% for Store Visit and Spaceship Repair, respectively).
However, we evaluated the trajectories produced by DESPOT and only 7.3% and 0% of the time do they align with the user’s intentions in Spaceship Repair and Store Visit, respectively. We will add these experiments and analyses to the supplementary for the final version.
**Q5:** This paper focuses on developing a formal framework and key technical results, illuminating the nature of the problem and solution directions for further research in an area that has been rather understudied. Human subject studies constitute a valuable direction for future work beyond the scope of this current paper.
**Partition Terminology:** The PRS algorithm uses partition terminology to emphasize that the set of partitions $X$ is always mutually exclusive and covers the parameter space.
Additionally, a partition can represent multiple disjoint intervals due to the disjunction or the negation of a conjunction of BSQs.
**References:**
Andreas, Jacob, Dan Klein, and Sergey Levine. "Modular multitask reinforcement learning with policy sketches." International conference on machine learning. PMLR, 2017.
Marthi, Bhaskara, Stuart Russell, and David Latham. "Writing Stratagus-playing agents in concurrent ALisp." Reasoning, Representation, and Learning in Computer Games (2005): 67.
Parr, Ronald, and Stuart Russell. "Reinforcement learning with hierarchies of machines." Advances in neural information processing systems 10 (1997).
Silver, David, and Joel Veness. "Monte-Carlo planning in large POMDPs." Advances in neural information processing systems 23 (2010).
Somani, Adhiraj, et al. "DESPOT: Online POMDP planning with regularization." Advances in neural information processing systems 26 (2013).
Thank you to the authors for their response, and for providing additional results showing that PRS is significantly faster than grid search, Nelder Mead, and particle swarm optimization, while also outcompeting unconstrained policy optimization in a number of cases. It would be ideal for these results to be expanded upon (i. e. don't just perform Nelder Mead / particle swarm optimization on a toy example, run it on all domains), and for them to be included in not just the Appendix but in the main paper (including the POMCP and DESPOT results).
Renaming "BSQ preferences" to "BSQ guidelines/templates" is also appreciated, and I think it would be good to reframe the motivation accordingly (perhaps by highlighting how BSQ guidelines can provide solution guidance that render a POMDP solvable even when unconstrained methods would fail to solve the POMDP). If the authors can make all of these changes for the final paper, then I am happy to increase my score to a 6.
In the meantime, I am increasing my score to a 5.
Thank you for your feedback and for improving the score. We are happy to hear that our response addressed your questions. We are currently evaluating Nelder-Mead and particle swarm optimization on our four test problems, as you suggested. The current results show that solutions solved with Nelder-Mead for Spaceship Repair, Store Visit, and Graph Rock Sample have a higher expected cost, lowest goal achievement rate, and significantly higher standard deviation compared to PRS as expected. We will include these results, with the POMCP and DESPOT results, in the final version of the main paper using an extra page. We will also switch the paper over to guidelines-based terminology rather than preferences and adjust our work's motivation appropriately.
- The paper tackles an interesting problem by looking at goal orientation and preferences in combination.
- The framework appears sound and well-rounded.
- The presentation could be improved. There are some (minor) details that need fixing for a final version, should the paper be accepted:
* Please use abbreviations consistently. Introduce an abbreviation once when the term occurs for the first time in the main text and then use the abbreviation only (exception: headings).
At least one abbreviation is introduced several times (BSQ, PRS), at least one is not introduced upon first occurrence of the term (gPOMDP), one is not introduced at all in the main body (POMDP).
* It usually is a sign of a weak text structure if sections have single subsections as seen with 5.1 and 6.1 (as also argued in most style guides). In the cases here, the first part of the sections deserve their own subsection headings. If the space does not permit this, consider making 5.1 and 6.1 part of the section, maybe using \paragraph to further structure the overall section.
* I recognise that space is scarce but the figures are very small, especially the experiments figures are hard to read. Please make sure that a colour-blind person can also read the figures.
* It could be more explicitly stated which parts of the formalisation is new and which part is already existing. The way it is formulated right now, it could be read as gPOMDPs and BSQ being existing concepts used for the first time for preference encoding or that both are new concepts.
* Again a question of available space but wrapping an algorithm in text is highly uncommon in academic AI papers.
(Not sure where this needs to go in the review form; minor comments:
- For the sake of consistency: align convexity / concavity between the abstract and the main body of the paper
- In Def. 1, the $F$ in line 104 is not defined. Should it be $\mathcal{F}$?
- p. 7, line 276: missing space between (PRS)(Algo. 1) // abbreviate Algorithm by Alg. to save the space of the 'o'?
The limitations are discussed in the appendix. It would be great to have at least some discussion / acknowledgement of it in the main paper.
No ethics review needed.
Thank you for your detailed feedback and suggestions. We will incorporate these changes in the final version of the paper. We also plan to use the extra page to update the figures and add a portion of the limitations into the conclusion section.
Our work is the first to formally define a BSQ framework for partially observable environments and to conduct a formal analysis of BSQs. We are the first to prove the surprising result that the parameter space for these preferences is piecewise constant and non-convex. Leveraging these properties, we developed a probabilistically complete algorithm for computing policies that match user intentions that balance goal achievement rate and average goal achievement time.
Thank you for the response. I will keep my score.
Thank you again for your review and for providing valuable feedback. We appreciate your support for our work.
We thank the reviewers for their detailed reviews and comments. We answer the questions posed by the reviewers separately. Please find them in the response below the reviews.

Meta-review:
role: PC
data: comment: value: This paper addresses the use of POMDP policies formulated using belief-state queries (BSQs) and how user constraints or guidance (in the authors’ terminology, preferences) using BSQs can be solved effectively. The paper provides a novel formulation, decent theoretical analysis and empirical validation of the effectiveness of the proposed method. This paper generated considerable discussion, especially around two main weaknesses identified by the reviewers: (1) the motivation and presentation, specifically, the use of the term “preferences” rather than, say, “policy constraints,” “policy class restrictions,” “user (behavior/policy) guidance,” etc.  (2) the inadequate comparison to suitable (albeit natural, easy-to-test) baselines. 

The authors and reviewers engaged in significant discussion, including the authors provided a substantial collection of new empirical results in the rebuttal, and agreeing to recharacterize the framing and motivation of the paper. This resulted in an overall raise in the reviewers’ (aggregate) assessment and scores. As such, I believe this paper makes a valuable contribution to NeurIPS. But it is critical that, if the paper is accepted, the authors’ make all of the revisions discussed (incl. incorporation of the new empirical results, ideally, not stashed away in an appendix).

Decision: Accept (poster)