Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper describes an approach for collaborative decision-making where another agent can make suggestions (in the form of suggested actions) to the subject agent in order to influence their chosen actions, but the subject agent retains autonomy in its own decision-making.  In particular, the subject agents decision-making is modeled as a POMDP and the suggestions from other agents are taken as observations by the subject agent and incorporated in the agent's belief state revision.  Two approaches are provided for representing the stochastic observation function for suggested actions, and empirical evaluation on the Tag and variants of the RockSample domains are employed, demonstrating how incorporating suggested actions is beneficial to the subject agent's utility returns.
strengths_and_weaknesses: The problem of collaborative decision-making is relevant to the multiagent planning community at NeurIPS.  The paper is relatively easy to follow.  I appreciated that it included more than one experimental domain, although the problems are relatively and old (see the POMCP, DESPOT, and other papers for larger, more recent benchmark domains).  The paper was also thorough in testing different parameters to the settings in the experiments (e.g., the \nu, \tau, and \lambda parameters).  I would have appreciated also seeing some theoretical analysis of the approach, such as regret bounds when the two different approaches for estimating the observation function are used.  

I also had some concerns with the approach and its performance.  The Noisy Rational estimate of p(o^s_t | s_t) requires the subject agent to have access to Q(s_t, o^s_t).  However, in POMDPs, the agent typically constructs Q(b, a) not Q(s, a).  So I'm not sure when or how the agent will have access to Q(s, a) to use the approach.  Also, in the original SARSOP paper, the average performance on Tag was -6.13, whereas you've reported -10.7, which calls into question whether (1) your implementation is correct, or (2) you ran it with a reasonable set of hyperparameters (e.g., number of backups).

### Post Rebuttal Period ###

I thank the authors for all of their responses to the reviewers questions and comments, including my own.   I think the idea of treating recommended actions as observations is a clever way to adapt single-agent POMDPs to incorporate advice from another agent (human or artificial), and I think the approaches you proposed for defining an observation function over those new observations are reasonable first approaches (and could inspire additional mechanisms with beneficial properties). The experiments and empirical results were also extensive and used reasonable benchmark problems and baselines. At the same time, without any theoretical analysis (which is increasingly common in papers like this), it is difficult to know how the results might generalize to other domains and situations.  I strongly encourage you to continue this work, and if you were to add some theoretical analysis, I think it would be a valuable contribution to the literature.
questions: 1) How does the agent calculate a Q value for a state instead of a belief (to be used in Noisy Rational)?  Could this approach generalize to algorithms other than point based value iteration derived solutions (e.g., SARSOP)?  Especially since the vast majority of POMDP research is using online planning using Monte Carlo methods, which is vastly different (and maintains no representation for Q values over states)?

2) From what I remember, a key feature of the SARSOP algorithm compared to previous PBVI-based algorithms is its focus on planning only for beliefs that are reachable from the initial belief.  This gave it a computational speedup in that it could prune out unreachable beliefs and not bother planning for them (in its point-based backups).  However, by adding new observations during runtime (i.e., suggestions from the other agent), you've changed the set of reachable beliefs -- a suggestion might cause the agent's new belief to not be near the ones considered during SARSOP.  In that case, your alpha vectors from SARSOP might not be a good approximation of the value function at your new, previously unreachable belief, and thus we have a new source of approximation error.  What impact might this have on the subject agent's performance in your approach, and how might it be prevented or mitigated?

3) Do you have an idea of why your SARSOP ("Normal") performance on Tag is so much lower than originally reported?

4) I appreciate that you included the code of your experiments, but some details of how the domains were changed from their typical implementations would be helpful in the paper.  For example, you state that you changed the opponent behavior in Tag to allow "more movement away from the agent", but what does that mean?
limitations: The limitations of the approach are adequately described.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper looks at the problem of designing a framework to support collaboration between an autonomous system and a human supervisor. In particular, the human supervisor provides action suggestions to the autonomous agent to help overcome any limitations the human could have in its sensor capabilities. The human is assumed to be a potentially suboptimal agent with full observability, while the agent is expected to operate under partial observability. The method proposes to use the action suggestions, by using the human policy as part of the observation model of the underlying POMDP, where the actions suggested by the human would correspond to the full observation. This means that the human suggestions could be used to refine the belief space the agent may hold regarding the current state. However, when implementing the approach, rather than directly using this new observation model during the POMDP planning, they use an approximation where in the human suggestion is used to further refine the belief space only if the suggestion differs from the current policy action. This approximate method is then tested on two benchmark domains (under various conditions), where the paper demonstrates that the human suggestions does make a positive impact on the system performance.
strengths_and_weaknesses: Strengths:
- Developing methods that are able to effectively collaborate with people and incorporate their suggestions into their decision-making process is an important problem.
- The paper presents a relatively clean and clear formulation. Additionally, the fact that the formulation allows the problem to be directly compiled into a standard POMDP means that one could always leverage the latest developments in POMDP planning/RL for the problem.
- While the evaluations are limited they do present some intriguing results. Specifically, the results presented in section 4.4 in regards to robustness of the methods to suboptimal human behavior is interesting.

Weaknesses:
I have two big concerns of the paper:

- Type of Interaction: To me the assumption that a human would be an ever present supervisor ready to provide suggestions to the system seems like the worst way to make use of the human. There are many results from psychology and human-factors (not to mention many high profile accidents) that show that humans are pretty bad in roles where they are expected to constantly pay attention and intervene as required. Humans constantly fall prey to issues like automation bias, where they may overtrust the system when they see that the system is performing as they expected in the beginning. I would argue that it would be better to treat the human as an expensive but highly information source of knowledge about the task. Rather than asking the human to come up with an action, the system could have equally asked the human about the underlying state. To avoid the system bothering the human too much, one could also attach some high cost to such information gathering actions.

- Use of the approximation: It's quite unfortunate that the authors chose not to use the exact formulation for the system evaluation. As the paper itself points out, the human agreeing with the system could also present a lot of useful information. While it is a positive sign that the system is still performing well even while only using a subset of the information, it is not quite obvious to me how the inclusion of this additional information would have impacted the results in section 4.4.


Clarification:
- Claims about human model: In multiple places in the paper, the authors claimed that unlike earlier works the proposed method builds no explicit human model (repeated in both section 2 and 3.3). As far as I can tell this is not true. It is true that the proposed method doesn’t make strong assumptions about the policy and in some sense the rationality of the human, however, you do make strong assumptions about the sensor model and the knowledge the human supervisor possesses. The authors assume that the human’s knowledge about the task dynamics is the same as what the system’s dynamics is, or at least that the system knows this model. So in some sense you don’t have to build a human model, because it's known beforehand. Additionally, to generate the human policy from this model you make additional assumptions about the rationality of the human planner. It’s also worth noting that there are many previous works (like [a] and [b]) that don't assume that the human’s knowledge about the task dynamics is the same as what the system knows.



Other Suggestions
- User Study: While the current experiments are a good starting point, I would strongly urge the authors to perform user studies to see how effectively their system can actually work with real people.
- Other forms of information: There is no reason to believe that the information about the underlying state is the only kind of information the human can provide. In future versions, I would suggest the authors consider cases where the human feedback may also reveal additional information about the task, including information about the task reward or potential information about effective strategies the system could use to solve the task.

[a] Reddy, Sid, Anca Dragan, and Sergey Levine. "Where do you think you're going?: Inferring beliefs about dynamics from behavior." Advances in Neural Information Processing Systems 31 (2018).

[b] Gong, Ze, and Yu Zhang. "What is it you really want of me? generalized reward learning with biased beliefs about domain dynamics." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 03. 2020.
questions: I would appreciate it if the authors could respond to the weaknesses and the clarification I had mentioned in the previous section. To repeat these are,
1. The merit in the assumption that the human would be a supervisor who constantly observes the system as opposed to other interaction paradigms
2. How the choice to only use the human suggestion to update belief states when they differ from the robot’s policy may have affected the results in section 4.4?
3. Please clarify, what you mean by the statement that the method doesn’t build a model of the human  

limitations: I don’t see any specific ethical concerns with the proposed methods. I have listed some of the limitations I have noticed of the proposed method under the weakness and other suggestions section. Most of these relate to practical challenges of interacting with humans. I would strongly urge the authors to consider checking human-factors literature on creating systems that are meant to effectively interact with people.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a model and experiments to demonstrate the usefulness of (possibly intermittent and/or incorrect) human action suggestions on a POMDP-based system, treating the suggestion not as a direct input but rather as an indicator of the partially-observed implicit state. Simulated experiments on a discrete state problem with an offline generated policy show some improvement using (possibly intermittent and/or partially incorrect) action suggestions from a human with an omniscient view (when sending correct information).

strengths_and_weaknesses: Originality
The authors point out much related work; the specific contribution here is that the action is not taken itself as a suggestion to be used or ignored, but rather as another kind of observation that can update the agent’s estimate of the true state, thus potentially wringing more information from the suggestion than merely following it would (and being more helpful when the suggestions are intermittent).

Clarity
The paper is generally clear and easy to read. The details of computation of the scaled rational and noisy rational approaches are a bit unclear, but the authors claim to provide code in supplemental material.

I strongly suggest that the authors mention early-on (e.g. in the abstract!) that the human is the one giving the suggestions (one could also view that multiple agents who have distinct partial views of the global state could also provide such suggestions that then fold into the local estimate of the full state, which is what I originally thought when I read the abstract. The paper *does* make this clear in the last line of paragraph 2 in the introduction.

Last paragraph section 2 “we do not build an explicit model” —> “we do not build an explicit model of the suggester”

I assume the notation “o_{o:t}” means all observations from time 0 through the current time?

Quality
The abstract and introduction talk a lot about failure and brittle or inconsistent behavior of current MDP-variant-based systems, and that human suggestions could help with that. However, the experimental environments (Tag and RockSample) don’t clearly show these issues. Instead, the versions of the system that take into account the *omniscient* collaborator, who has information not available to the agent, do better (even the naive agent that simply uses suggestions directly). The authors do go to pains to demonstrate that the more complex indirect use of the intermittent (but often omnisciently correct) suggestions as observation about true state require fewer communications and more robust in the face of unreliability of the suggester, but how that might help reduce brittle and inconsistent behavior as discussed in the abstract and introduction is somewhat unclear.

[After rebuttal: just downplay this, especially in abstract/intro, so the reader is not looking at this as the primary contribution. Could mention it later in discussion as a possible area to explore]

Aside from that: how many runs of each scenario? The text just says that they “varied” (why not some single, consistent numbers runs?)

Significance
The core idea seems to be a useful one, although the demonstration here is on small, discrete, 2-agent environments without catastrophic failures, brittle behavior, or clear inconsistencies. 

questions: How do these experiments shed light on your original research impetus of reducing brittle behavior via human suggestions ?

Is the assumption that the “suggested action is not influenced by the previous action” really reasonable, given that one might think the human would be especially wanting to suggest an action if the agent does something bad/wrong/ill-advised?

limitations: This work seems theoretical enough to not have significant societal impacts (beyond, perhaps, the use of computational agents, but here they have human supervisors!)
review_title: null

Meta-review: metareview: The paper considers a collaborative decision-making setting in which one agent can suggest actions for a listening agent to execute. The listening agent is not bound to take these suggested actions. The paper models the listening agent's decision-making process as a POMDP, treating the suggestions provided by other agents as observations (i.e., dependent only on the state) that are used to update the belief state. The paper describes two representations of the suggested actions and presents empirical results on simulated domains that demonstrate that the listening agent's performance improves when following the suggestions of a perfect oracle, while it degrades given imperfect suggestions in proportion to the level of noise.

The paper was reviewed by three researchers who read the author response and discussed the paper together with the AC. The reviewers largely agree that the collaborative decision-making problem as formulated is interesting. The reviewers find that the idea of formulating suggestion-following as a POMDP with suggestions modeled as observations is clever, and that the description of the proposed formulation is clear and easy to follow. The two formulations of the observation function are reasonable, albeit very simple initial models, though the paper does not provide much insight into when one should be used over the other. A primary concern with the paper is that it only provides empirical results. The work would have significantly benefited from theoretical results, which would help to understand how the method would generalize to other domains.
confidence: Less certain
Decision: Accept