Collaborative Decision Making Using Action Suggestions
Authors: Dylan Asmar
Mykel Kochenderfer
Venue: NeurIPS 2022 Accept (2022)
Keywords: collaboration
decision making
human-ai collaboration
pomdp
state estimation

Abstract: The level of autonomy is increasing in systems spanning multiple domains, but these systems still experience failures. One way to mitigate the risk of failures is to integrate human oversight of the autonomous systems and rely on the human to take control when the autonomy fails. In this work, we formulate a method of collaborative decision making through action suggestions that improves action selection without taking control of the system. Our approach uses each suggestion efficiently by incorporating the implicit information shared through suggestions to modify the agent's belief and achieves better performance with fewer suggestions than naively following the suggested actions. We assume collaborative agents share the same objective and communicate through valid actions. By assuming the suggested action is dependent only on the state, we can incorporate the suggested action as an independent observation of the environment. The assumption of a collaborative environment enables us to use the agent's policy to estimate the distribution over action suggestions. We propose two methods that use suggested actions and demonstrate the approach through simulated experiments. The proposed methodology results in increased performance while also being robust to suboptimal suggestions.

Collaborative Decision Making Using Action Suggestions
Dylan M. Asmar
Mykel J. Kochenderfer
Stanford Intelligent Systems Laboratory
Stanford Intelligent Systems Laboratory Stanford University
Stanford University
Stanford, CA 94305
Stanford, CA 94305
asmar0stanf ord. edu
mykel&stanfordd
Abstract
The level of autonomy is increasing in systems spanning multiple domains, but these systems still experience failures. One way mitigate the risk of failures is to integrate human oversight of the autonomous systems and rely on the human take control when the autonomy fails. In this work, we formulate method of collaborative decision making through action suggestions that improves action selection without taking control of the system. Our approach uses each suggestion efficiently by incorporating the implicit information shared through suggestions to modify the agent's belief and achieves better performance with fewer suggestions than naively following the suggested actions. We assume collaborative agents share the same objective and communicate through valid actions. By assuming the suggested action is dependent only on the state, we can incorporate the sug- gested action as an independent observation of the environment. The assumption of a collaborative environment enables us to use the agent's 's policy to estimate the distribution over action suggestions. We propose two methods that use suggested actions and demonstrate the approach through simulated experiments. The pro- posed methodology results increased performance while also being robust suboptimal suggestions."
Introduction
Autonomous systems and humans have different strengths. Moravec [1] claimed in 1988 that "It comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult impossible give them the skills of one-year-old when it comes to perception and mobility." Thirty years later, computers have achieved superhuman level per- formance many games like Go and StarCraft II [2], [3]. Despite major strides in perception, algorithms still exhibit brittleness and struggle with consistency in tasks that most would consider trivial for young adult, like reliably generating plausible sentence from list of words [4]-[7).
Humans often make suboptimal decisions; however, they are exceptionally good finding ways to accomplish tasks, balance risk, reason through unstructured problems, and incorporate new infor- mation into decisions without prior experience take advantage of human strengths by using human-on-the-logg structure. This framework allows for humans to use their experience to recognize situations where the autonomy might struggle or failing and to take control [8], [9]. We hypothesize that instead of requiring the human to assume control, an agent can use occasional control inputs from the human-on the-loop in the form action suggestions improve its understanding of the environment.
situation. Systems integrating autonomy often
'Code is available at http://hhhht..ccoooasaacccccctee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we explore a method of collaboration that enables combining benefits of heterogeneous systems like humans and machines through action suggestions versus direct control. Our approach was inspired by the way that we see humans incorporate action suggestions in daily life. A real world example is a pilot flying an aircraft while coordinating with air traffic control (ATC). the United States, there are various services ATC provides to ensure safety. Many of these services provide suggestions to pilots, but do not mandate an action or remove responsibilities from the pilot (e.g. radar assistance to VFR aircraft) [10]. Consider situation where pilot is flying at low altitude, has knowledge of another aircraft to the left, and an ATC controller recommends a turn to the left. This action suggestion would cause the pilot to consider situations where a left turn towards traffic would be optimal (e.g. terrain ahead) and reassess their belief about the environment.
In our problem of interest, we are assuming the agent has the same objective as suggester and use action suggestions as additional information about the environment to increase the quality of the action selection. We avoid having to share exact belief states between agents by treating the suggested action as an observation. Using the assumption that the suggester collaborative, we use the agent's 's policy to infer a distribution over suggested actions and use this distribution to update the agent's belief. This approach allows collaboration between different types of systems and avoids explicit translation of complex belief spaces.
In this paper, we first review related work in section and discuss how our approach differs from previous methods. Our primary contributions are in section 3. We first provide an overview of a partially observable Markov decision process (POMDP) and then outline the problem setting. Section 3.3 demonstrates how we can view action suggestions as independent observations and modify the belief update process. We then provide two methods to estimate the distribution over suggested actions using the agent's policy in order to update the belief in section 3.4. Section 4 presents the experiments and discusses key results demonstrating the effectiveness of our approach.
2 Previous Work
Collaborative sequential decision making between two agents is a form of shared autonomy. The exact definition of shared autonomy and control varies in the literature [11]. A common theme among the approaches is combining inputs between two agents achieve performance better than the agent's s acting alone. The problems often involve one of two main categories: 1) assistance from an autonomous agent to human or 2) assistance from human to the autonomous system. The problem framework we are considering aligns with the second category.
In the first category, an autonomous agent is seeking to perform actions to assist the other agent (often a human). Dragan and Srinivasa [12] discuss key aspects of this category and decompose into prediction of the user's 's intent and arbitration between the user's inputs and the action chosen by the autonomous system. Nguyen et al. [13] propose a method that decomposes a game into subtasks that can be modelled as a Markov decision process. The autonomous agent then infers the subtask the other agent is attempting to accomplish and selects actions to assist. Others model this category as POMDP where the uncertainty is around the goal of the assisted agent [14]-[16].
The second category often involves forms of corrections. Losey and O' Malley [17] use aspects of optimal control to learn from physical human interventions. Nemec el al. [18] first learn from human demonstrations and then refine the policy through incremental learning from kinesthetic guidance of the autonomous agent. Other approaches perform the corrections in an iterative cycle during a planning process. Reardon et al. [19] present a method that allows a human to interactively provide feedback during planning. The feedback is in the form of hints or suggestions and is used to modify the optimization process. This approach is similar to the concept of reward shaping in reinforcement learning.
Cognetti et al. [20] provide method that allows for real time modifications of a path while Hagenow et al. [21] present method that permits an outside agent to modify key robot state variables. These changes are then blended with the original control. The concept of following guidance versus acting independently and how to blend the two was explored by Evrard and Kheddar [22] where they use two controllers and alternate roles of following and leading. Medina et al. [23] present a method that selects between two strategies depending on the level of disagreement between agents.
2

---

Another related area of research is imitation learning. Within this field, there are methods to learn from feedback by querying the expert during training or expert intervention real-time [24], [25]. These methodologies are a form of luman-maching collaboration by using the expert knowledge of the human to train or correct the system and then letting the system perform the task autonomously. These approaches generally rely on strict assumptions or an explicit model of the other agent so the autonomous system can interpret inputs in a way to reason how to integrate them. In the first category, Jeon et al. [26] propose a model structure to map human inputs to different actions based on the robot's is confidence in the goal. Reddy et al. [27] deviate from the use of model based methods by using human-in iattan----tte-ooooo tho- deep reinforcement learning to map from observations and user inputs to an agent action.
Our approach builds on the concept of mapping user inputs to an agent action. However, we differ from other methods in that we do not build an explicit model of the suggester nor learn a mapping that might be suggester dependent. We achieve this mapping by assuming the suggester is collab- orative and shares the same objective when providing inputs. This assumption enables the use of the key insight from Spencer et al. [25]. "... any amount of expert feedback... ... provides information about the quality of the current state, the quality of the action, or both", and we build an implicit model of the suggester using only the agent's ss knowledge of the environment and the agent's 's policy.
3 Action Suggestions as Observations
3.1 Background
A partially observable Markov decision process (POMDP) is a mathematical framework to model sequential decision making problems under uncertainty [28]. A POMDP is represented as a tuple (S, A. O, T, O, R, 7), where S is set of states, A is set of actions, and O is a set of observations. At each time step, an agent starts in state & E S and chooses an action, a E A. The agent transitions from state to state s' based on the transition function T(s, a, 8') p(s' 8, a), which represents the conditional probability of transitioning to state s' from state s after choosing action a. The agent does not directly observe the state, but receives an observation E O based on the observation function O a, o) = p(o s', a), which represents the conditional probability of observing observation o given the agent chose action and transitioned to state s'.
At each time step the agent receives a reward, R(s, a) E E for choosing action a from state s. For infinite horizon POMDPs, the discount factor y E [0, 1) is applied to the reward at each time step. The goal of an agent is to maximize the total expected reward EE /Ee 7 (sx, at)], where 8 and a, are the state and action at time t. One method to solve a POMDP is to infer belief distribution b E over S and then solve for policy ı that maps the belief to an action where B is the set of beliefs over S [29]. Executing with this type of policy requires maintaining b through updates after each time step.
Policies can be generated offline or computed online during execution. In this work, we focus on applying our method policies generated offline and leave the application to online solvers for future work. Many approximate offline solvers involve point-based value iteration. The idea is to sample the belief space and perform backup operations on the sampled points of the belief space, iteratively applying the Bellman equation until the value function converges. PBVI [30], HSV12 [31], FSVI [32], and SARSOP [33] are examples of such an approach, though they differ in the selection of initial belief points, the generation of points at each iteration, and the choice of which points to backup. These algorithms represent the policy as a set of alpha vectors. this work, we used SARSOP to generate the policies; however, any algorithm that produces policy where the utility of a belief can be calculated could be implemented with little change to our methods.
3.2 Problem Formulation
For given problem of sequential decision making under uncertainty, we choose to model the prob- lem as a POMDP and use point-basee solution method. In this work, we assume discrete state, action, and observation spaces but the methods can be generalized to continuous spaces. Our prob- lem involves two entities: an autonomous system using the POMDP policy to perform actions and interact with the environment that we will refer to as the agent, and a suggester providing action suggestions to the agent. The suggester can observe the environment, but not interact or affect the
3

---

state except to provide recommended actions to the agent. The suggester is not required to provide suggestions but can provide a maximum of one action suggestion each time step.
The agent and suggester are collaborative and share the same objective of maximizing the total ex- pected reward. The agent and suggester can receive different information and maintain separate beliefs of the environment. The separation of the actors allows each to process and receive informa- tion independently and capitalize on strengths differently. An example problem is an expert human receiving observations not modeled by an autonomous robot and providing intermittent suggestions.
3.3 Incorporating Action Suggestions
There are different ways for the agent to use action suggestions. One approach would be to model the suggester in the original POMDP. This approach is similar to previous shared autonomy work (section 2). Modeling the suggester within the POMDP would provide a fundamental way of incor- porating suggestions, but would require a priori knowledge of the suggester to develop an explicit model to solve for the policy and also require maintaining a belief over the model during execution. A simple method that does not require model of the suggester would be for the agent to naively follow each suggestion. meets some defined criteria, thus potentially disregarding suboptimal suggestions (c.g. only follow a suggestion if it a specific action). These approaches are simple and can incorporate suggestions; however, they do not benefit from incorporating implicd information contained with each suggested action.
similar and more robust method would to follow a suggestion if
Applying our inspirations we introduced in the flight example to our problem, each action suggestion contains information related to the suggester's belief of the enviromment. We propose treating each action suggestion an observation of the state in order to update the agent's 's belief. This idea enables the suggester to influence the agent while the agent remains autonomous. Our problem assumes the suggester and the agent maintain independent beliefs. If we further assume the suggested action is not influenced by the agent's previous action, the suggested action is only dependent on the state. The independence of the agent's previous action and the suggester allows modification of our belief update process that only involves the probability of receiving the suggested action given the current state. The suggested action is not always independent of the agent's action, but the assumption that it only depends on the current state is often reasonable.
Our belief at time over state E S with observations o E O and action suggestions Oa E A is p( 00:t-1, Outt o. Using Bayes' theorem, we can rewrite our expression as
p(s: ag:t Op:t: daa) x p(o Sts Go.t , Ooa h )poo
St, aott .ott .0:))
p(st ag:t- -0:0
(I)
where the subscript 0:t refers to all instances of that variable from 0 t, and tt =t t-1 1. This expression can be simplified using the independence assumption, the law of total probability, and the Markov property to
p(8: 0p:1 00.1, )or) plo sc)p(e se, a-)) Σ p(se 85-, )p((e- I aa-,00- og ) ES
(2)
Equation (2) is a simple modification to our standard belief update procedure with POMDPs and is an expression of updating belief with two independent observations.
3.4 Inferring the Distribution over Suggested Actions
We can update the agent's belief based on p(o st). However, the agent cannot calculate this distribution directly. We propose using the assumption that the agents are collaborative and estimate p(o st) by using the agent's 's policy. In our flight example, when the pilot receives the suggestion to turn left they would use their experience to reason through what scenarios they would have provided that same suggestion.
Scaled Rational One approach to assume the suggester perfectly rational and has a policy identical to the agent. The suggester would only give actions that would maximize the total expected

---

reward using the agent's policy In other words, p( st) = 1 =((s)) where 1(-) is the indicator function. To relax the assumption of perfectly rational agent and am identical policy, we (0. With a scaling factor, we assume the suggester acts rationally
introduce a scaling factor, and with the agent's policy a fraction r of the time and uses a random policy otherwise. The scaled rational update can be expressed as
T, -A-1
of = (s:))
plop 8t)) ~
(3)
otherwise.
Noisy Rational. Another approach is to assume the likelihood of the suggested action is related to the total expected reward of choosing that action. Shepard [34] developed a noisy rational model from psychological perspective, and this model has been widely used in robotics to model subop- timal decision making [35]-[37]. With this model, the suggester is most likely to choose the action with the highest expected return and less likely to choose suboptimal actions. This model requires calculating the total expected return of each action.
The action value function Q(s,a) returns the expected value of performing action in state and then executing optimally thereafter. We can use the agent's policy and the reward function and perform one step look ahead to calculate the Q-function [29]. Using policy represented with alpha vectors, we can calculate Q(s, a) by performing a one step look ahead using a belief that the agent is in state s. The noisy rational model has the same form the Boltzmann distribution and the softmax function. We can express this model as
exp (AQ(s,, o;))
p st)
Ene A exp (AQ(sr,
where E [0, oo) is hyperparameter often referred to as the rationality coefficient. The distribu- tion approaches uniform distribution À approaches O. As À increases, the model approaches perfectly rational agent.
Experiments
The proposed methodology to incorporate action suggestions as observations was evaluated on two classic POMDP problems, Tag [30] and RockSample [38]. These domains relatively simple but provide a way to evaluate the merits of the approach by removing domain-specifie variables that might influence performance. The simulations
constructed to first evaluate the effective- ness and efficiency of the proposed approach and then to test the robustness to suboptimal action suggestions.
4.1 Environments
The two enviromments chosen to evaluate our approach have discrete action, state, and observation spaces. The structure of the problems allow for the visualization of the belief space while also providing modest scalability problem.
Tag. The Tag environment ment can be seen in fig. 1. The agent and an opponent are initialized randomly in the grid. The goal of the agent is to tag the opponent by performing the tag action while in the same square as the opponent. The agent can move in the four cardinal directions or perform the tag action. The movement of the agent is deterministic based on its selected action. A reward of -1 is imposed for each motion action and the tag action results in +10 for successful tag and - 10 otherwise. The agent's position is fully observable but the opponent's position is unobserved unless both actors are in the same cell. The opponent moves stochastically according to a fixed policy away from the agent. The opponent moves away from the agent 80 % of the time and stays in the same cell other- wise. Our implementation of the opponent's 's movement policy varies slightly from the original paper allowing more movement away from the agent, thus making the scenario slightly more challenging. Section I. provides more details of the differences.
first introduced by Pineau et al. [30]. The layout of the environ-
5

---

RockSample. The RockSample environment consists of a robot that must explore an environment and sample rocks of scientific value [38]. Each rock can either be good or bad and the robot receives rewards accordingly. The robot also receives reward for departing the environment by entering an exit region. The robot knows the positions of every rock and its own location exactly, but does not know whether each rock is good or bad. The robot has noisy sensor to check if rock is good or bad and the accuracy of the sensor depends on the distance to the rock. Upon each use of the sensor, the robot receives negative reward. In the following sections, a RockSample problem will be designated as RockSample(n, K, ST, sp) where n designates a grid size of n x nt, k is the number of rocks, sr is the sensor range efficiency, and sp is the penalty for using the sensor. The reward for sampling a good rock and exiting the environment is +10 and the penalty for sampling a bad rock is 10.
4.2 Simulation Details
The simulation environment was built using the POMDPs.jl framework [39]. The Tag environment used the standard parameters and was simulated until the first of the agent tagging the opponent or 100 steps. The RockSample environment was simulated until the agent exited the environment. The SARSOP algorithm was used to generate policies for the agent. All of the agents used the same policies for the environment but incorporated the action suggestions differently. If the suggested action was the same the selected action with the current belief, modification of the belief was performed. This implementation decision potentially prohibits valuable information to be passed when actions align. However, simulations with the RockSample and Tag environments showed differences were negligible and the slight decrease in computation time allowed for more simulations to be performed. Performing belief updates with aligned suggestions would likely be critical in different scenarios.
In RackSample, the rocks were randomly initialized based uniform distribution. The agent's belief was initialized with a uniform distribution over the state space in Tag and a uniform distri- bution over the rocks in RockSample. The number of simulations for given scenario varied and we provide the 95 % confidence interval for each value reported in the results section. The rewards reported are the mean value over all simulations for a given scenario. The number of suggestions refers to the number of times the suggested action differed from the action initially selected by the agent before considering the suggestion. To simulate a suggester that was not always present or could not communicate actions reliably, the simulation also supported varying the percentage of suggestions passed to the agent.
One suggester was used but the quality and consistency of the action suggestions varied, Different agents were simulated to evaluate our proposed approach and establish baselines in each scenario. Details of each agent are provided below.
Suggester. The suggester is combination of an all-knowing agent and purely random suggester. The rate of randomness is adjusted to scale from purely random to completely all-knowing. If the suggester is sending a non-random suggestion, the action is selected from the POMDP policy using the true state of the environment.
Normal Agent. The normal agent executes in the environment using the policy without considering any action suggestions. This agent provides baseline when action suggestions are not considered.
Perfect Agent. The perfect agent is initialized and executed with perfect knowledge of the state. At each time step action is selected from the policy given the true state of the system. This agent provides and upper bound on the total expected reward executing with the given policy.
Random Agent. The random agent chooses an action from uniform distribution over the action space at each time step. This agent provides a lower bound on the total expected reward when investigating robustness to suboptimal and random action suggestions.
Naive Agent. The naive agent executes in the environment using the POMDP policy. When action suggestions are received, it follows the action suggestions naively, and performs no modifications to its belief state. The rate at which the naive agent follows the suggestions was adjusted to investigate robustness. The rate of following the suggestion depicted by v in the results.

---

Table 1: Simulation results of various agents using an all-knowing suggester.
Tag
RS(7,8, 20,0)
RS(8, 4, 10, -1)
Agent Type
Reward
# Sugg
Reward
# Sugg
Reward
# Sugs
Normal
10077
21.5 $1.5.00 0.6
10.1 0.1 16.7 0.1
Perfect Naive
-1.7
28.1 0.5
V 1.00
0.2
3.1
28.5 1
15.3 = 0.3
16.9 + 0.1
8.4
0.75
0.2
6.1
2.2
0.2
14.6
7.7
0.50
0.3
0.9
23.8
15.1 0.2
12.7 0.2
0.2
Scaled
т 0.99
0.8
3.1
0.5
6.1
0.4
2.8
T = 0.75
0.2
3.3
0.5
6.8
16.2
2.8
0.50
0.2
3.1
2.10
0.1
16.3
3.1
Noisy
0.8
3.1
2..
7.8
1..4
4.6
= 2.0
0.2
3.
0.6
9.1
16.3
4.5
A = 1.0
0.2
3.6
26.8 + 0.6
10.6 + 0.2
16.2 + 0.2
5.2 0.1
Scaled Agent. The scaled agent incorporates the methodology outlined section 3.4 and uses eq. (3) estimate p(ož st). The hyperparameter kept constant for each sim- ulation and the value used is shown with the presented results. The value t was not adjusted to fine-tune performance. It was broadly changed to show overall effects of the hyperparameter different situations.
Noisy Agent. The noisy agent also incorporates the ideas from section 3.4. This agent uses eq. (4) to estimate p(o s.). The hyperparamter À is kept constant for each simulation and the value is shown with the respective results. Like the scaled agent, parameter À was not fine-tuned for performance. Coarse adjustments were made to depict how the hyperparameter influenced the algorithm.
4.3 Effectiveness and Efficiency Results
The different agents were compared using an all-knowing suggester with 100 % message recep- tion rate. Simulations were ran on Tag, RockSample(77, 8, 20, 0), and RockSample(S, 4, 10, -1). RockSample(8, 4, 10)- was designed with the rocks near the corners of the environment. This layout emphasized the importance of each movement direction. The results are summarized in ta- ble 1. We also compared the different agents using suggesters with partial views of the state and provide those results in section A.
As expected, incorporating actions suggestions from an all-knowing suggester improves perfor- mance across all scenarios. Despite not directly following the suggested action, the Scaled and Noisy agents were able to reach near perfect reward. A key metric note with these results is the number suggestions. The naive agent was able to achieve perfect scores by simply following the suggestions; however, it did not integrate any information contained from each suggestion and resulted in requiring more suggestions to achieve similar scores. While the scores for all assisted agents decrease when the hyperparameters change (become less trusting of the suggester) the naive agent's 'ss score decreases at a faster rate. Again, this difference an artifact the benefit of updating the belief of the agent with each suggestion.
visual depiction of the change in the belief state of an agent in the Tag environment after incorpo- rating an action suggestion is depicted in fig. 1. Figure la shows the belief the agent has of where the target located. With that belief, the best action according to agent's policy would to move north However, the agent receives a suggestion from suggester to move west. Figure 1b shows the updated belief after using the noisy rational approach with the west of the agent where the target might be located. The update process was able to incorporate
There are multiple states to
7

---

6/M4
0.00
0.01
301
0.00
1000
0.01
000
0063
0.08
002
0.01
0M0
0.15
305
0.06
0.8
061
0.02
6/2
0004
0.55
0.15
0.3
0.12
0.11
0.04
0m1
0.08
002
0.8
(T)
north
<(b)
west
(a) Before received action suggestion.
(b) After belief update with o² = west.
belief state agent after incorporating recommended action. Using the original belief the agent's policy и returns an action of north. The recommended action was to move west. Figure 1b depicts the updated belief b' after incorporating the suggested action. After the update, the policy produces an action of west. The belief update used noisy rational approach with À
Figure 1:
Changes
Rowed
Naunal
Scaled- 0.59
Noary
Nekey 5.0
05
114
Mosigo Rocetion Rate
Mosage Rocyticn Rxt
(a) Message rate versus mean reward.
(b) Message rate versus suggestions per step.
Figure Tag performance with varying message reception rates.
the implied information the west suggestion by shifting the distribution towards the west side of the grid.
Collaborators often are not capable of continuously providing suggestions an agent. The percent- age of suggestions received was varied to investigate the effectiveness of the agents in the presence of intermittent action suggestions. A suggester with perfect state knowledge was used but the mes- sage reception rate was varied. The results for the Tag environment are shown in fig. 2. The change in reward is shown fig. 2a. Figure 2b shows how the ratio of the number of suggestions to the number of steps required to tag changes. The increased reward and lower suggestion ratio further emphasizes the benefit of our approach.
4.4 Robustness Results
The previous results assumed perfectly rational suggester, but this is not always realistic. We evaluated the robustness of the different agents by adjusting the randomness of the suggester. The chance of a random action suggestion (uniformly picked from the action space) was varied from 0 to representing perfect suggester simulations are shown for both the Tag environment and RockSample($, 4, 10, -1) in fig.
completely random suggester. The results from these
As expected, the naive agent that follows all suggestions performs poorly as the randomness of the suggester increases. Decreasing y increases the robustness, but sacrifices performance. The scaled and noisy approaches perform well even with high trust parameter settings (T and v)) These results demonstrate the value of not naively following suggestions and the benefit of balancing the agent's initial belief state with the information received from the action suggestion.

---

Narnal
Resond
Rondeen
Nave-13
Nave- 0.25
hane -025
Scaad- 1399
Scalal 005
Sodel 0.5
Suggetton
Seggention
(a) Tag
(b) RockSample(S (8, 10, -1)
Figure 3: Robustness suboptimal action suggestions.
Conclusion
We developed new method to increase collaboration between heterogeneous agents through the use of action suggestions. Based on the idea that suggested action conveys information about the suggester's belief, we used the agent's current policy transform the suggested action into distribution that we used to update the agent's belief. We applied this approach in simulation and demonstrated the increased efficiency by requiring fewer suggestions and suggestion rate. We fur- ther demonstrated our approach was robust to suboptimal decisions and could still perform better than an unassisted agent with more than 50 % of the suggestions being random. This methodol- ogy does not rely on knowledge of other agents. The only requirement is a shared objective and communication through the agent's set of actions. This low threshold of coordination increases collaboration opportunities with other systems including humans.
Integrating action suggestions observations requires change to the belief update process of the agent. The independence of the observations allow the update process to occur simultaneously or in any order which allows for asynchronous and unreliable suggestions without added complexity. This approach also scales linearly with the number of suggesters because the agent must only approximate the distribution
action suggestions for each suggester.
critical assumption that enables this process with no knowledge of the suggester is that the sug- gester is collaborative. The scaled and noisy rational approaches do factor in suboptimal decisions but ultimately assume the suggester is providing actions in the best interest of the agent. a sug- gester maleficent, the belief would be skewed towards distributions that would explain the sug- gestions. The proposed approach is designed to be robust to suboptimal suggestions, but not resilient to bad actors.
There are many opportunities of future work expanding on these ideas. As previously discussed, this approach was formalized and applied to offline methods, but the same concept can be extended to online solvers. Online approaches also offer unique opportunities use suggested action to help balance exploration and cxploitation while searching for action. The experiment results demon- strated a trade-off of performance and robustness when changing the hyperparameters. Finding a way to learn the quality of the suggester and adjust the parameters real time is promising idea. The benefit of modifying the agent's 's belief through an action suggestion is not limited to situations when the subsequent, updated belief is more accurate. A modified belief could be beneficial if the resulting actions are optimal regardless the accuracy of the belief. This idea opens up possibili- ties of using action suggestions to increase performance when agent's policy does not match the environment.
References
[1] H. Moravec, Mind Children: The Future of Robot and Human Intelligence. Harvard Univer- sity Press, 1988.

---

[2] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, Lillicrap, and D. Silver, "Mastering atari, go, chess and shogi by planning with a learned model," Nature, vol. 588, no. 7839, pp. 604-609, 2020.
[3] O. Vinyals, I. Babuschkin, W. M. Czarnecki, al., "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, vol. 575, pp. 350-354, 2019.
[4] B. Y. Lin, M. Shen, W. Zhou, P. Zhou, C. Bhagavatula, Y. Choi, and X. Ren, "CommonGen:
A constrained text generation challenge for generative commonsense reasoning," in Findings of the Association for Computational Linguistics, 2020.
[5] J. Su, D. V. Vargas, and K. Sakurai, "One pixel attack for fooling deep neural networks," IEEE IEEE 2019.
Transactions on Evolutionary Computation, vol. 23, pp.
[6] N. Akhtar, A. Mian, N. Kardan, and M. Shah, "Advances in adversarial attacks and defenses
in computer vision: A survey," IEEE Access, vol. 9, pp. 155 161-155 196, 2021.
[7] J. E. Korteling, G. C. van de Boer- Visschedijk. R. A. M. Blankendaal, Boonekamp, and A. Eikelboom, "Human versus artificial intelligence," Frontiers in Artificial Intelligence, vol. 4, 2021.
[8] Tesla "Introducing navigate on autopilot." (2018), [Online]. Available: https : / www. tesla. www.e/ooooooottreeeeeeeeeeeeeeee
wwwtrrotcinnnnnvggatteeeeee navigate-autopee autopilot (visited 05/16/2022).
[9] M. Körber, Baseler, and K. Bengler, "Introduction matters: Manipulating trust in automa- tion and reliance in automated driving," Applied Ergonomics, vol. 66, pp. 18-31, 2018.
[10] FAR/AIM: Federal Aviation
Information Manual, Federal Aviation Administration, Washington, DC: U.S. Department of Transportation, 2022.
[11] D. A. Abbink, T. Carlson, M. Mulder, J. C. D. Winter, Aminravan, T. L. Gibo, and E. R. Boer, "A topology of shared control systems-finding finding common ground in diversity," IEEE Trunsactions on Sitman-Macchnne Systems, vol. 48, no. 5, pp. 509-525, 2018.
formalism for shared control," The [12] A. D. Dragan and S. S. Srinivasa, "A policy-blending
International Journal of Robotics Research, vol. 32, no. 7, pp. 790-805, 2013.
[13] T.-H. D. Nguyen, D. Hsu, W. S. Lee, T.-Y. Leong, L. P. Kaelbling. T. Lozano-Perre Perez, and
A. H. Grant, "CAPIR: Collaborative action planning with intention recognition," in AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 2011.
[14] O. Macindoe, L. P. Kaelbling, and T. Lozano-Perez, "POMCoP: Belief space planning for sidekicks in cooperative games," in AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2012.
[15] S. Javdani, S. S. Srinivasa, and J. A. Bagnell, "Shared autonomy via hindsight optimization," Robotics: Science and Systems, 2015.
[16] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell, "Shared auton- omy via hindsight optimization for teleoperation and teaming." The International Journal of Robotics Research, vol. 37, no. 7, pp. 717-742, 2018.
[17] D. P. Losey and M. K. O'Malley. "Learning the correct robot trajectory in real-time from physical human interactions." ACM Transactions on Hlaman-Robot Interaction (THRI), vol. 9,
no. 1, pp. 2019.
[18] B. Nemec, L. Žlajpah, S. Šlajpa, Piškur, and A. Ude, "An efficient PbD framework for fast deployment of bi-manual assembly tasks," in IEEE-RAS International Conference Humanoid Robots (Humanoids), 2018
[19] C. Reardon, H. Zhang, and J. Fink, "Shaping of shared autonomous solutions with minimal interaction." " Frontiers in Neurorobotics, vol. 12, 2018.
[20] M. Cognetti, M. Aggravi, C. Pacchierotti, Salaris, and P. R. Giordano, "Perception-aware human-assisted navigation of mobile robots on persistent trajectories," IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 4711 1-4188, 2020.
[21] M. Hagenow, E. Senft, R. Radwin, M. Gleicher, B. Mutlu, and M. Zinn, "Corrective shared autonomy for addressing task variability," IEEE Robotics and Automation Letlers, vol. 6,
no. 2, pp. 3720-3727, 2021.
[22] P. Evrard and A. Kheddar, "Homotopy-basee controller for physical human-robot interac- tion," in IEEE International Symposium on Robot and Human Interactive Communication, 2009.
10

---

[23] J. R. Medina, M. Lawitzky, A. Molin, and S. Hirche. "Dynamic strategy selection for physical robotic assistance in partially known tasks," in IEEE International Conference on Robotics and Automation (ICRA), 2013.
[24] S. Ross, G. Gordon, and J. A. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning," in International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
[25] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and S. Srinivasa, "Expert intervention learning." Autonomous Robots, vol. 46, pp- 99-113, 2021.
[26] J. Jeon, D. P. Losey, and D. Sadigh, "Shared autonomy with learned latent actions," in Robotics: Science and Systems, 2020.
[27] S. Reddy, A. D. Dragan, and S. Levine. "Shared autonomy via deep reinforcement learning," in Robotics: Science and Systems, 2018.
[28] D. Smallwood and E. Sondik, "The optimal control of partially observable Markov processes over finite horizon," Operations Research, vol. 21, pp. 1071-1088, 1973.
[29] M. Kochenderfer, T. A. Wheeler, and K. H. Wray, Algorithms for Decision Making, MIT Press, 2022.
[30] J. Pineau, G. Gordon, and S. Thrun, "Point-based value iteration: An anytime algorithm for POMDPs," in International Joint Conference on Artificial Intelligence (IJCAI), 2003.
[31] T. Smith and R. Simmons, "Point-based POMDP algorithms: Improved analysis and imple- mentation," in Conference on Uncertainty in Artificial Intelligence (UAI), 2005.
[32] G. Shani, R. I. Brafman, and S. E. Shimony, "Forward search value iteration for POMDPs." in International Joint Conference on Artificial Intelligence (IJCAI), 2007.
[33] H. Kurniawati, D. Hsu, and W. S. Lee, "SARSOP: Efficient boint-based pomdp planning by approximating optimally reachable belief spaces," Robotics: Science and Systems, 2008.
[34] R. N. Shepard, "Stimulus and response generalization: A stochastic model relating general- ization distance in psychological space." Psychometrika, vol. 22, no. pp. 325-345, 1957.
[35] M. Kwon, E. Biyik, A. Talati, K. Bhasin, D. P. Losey, and D. Sadigh, "When humans aren't t optimal: Robots that collaborate with risk- humans," in ACM/IEEE International Con- ference on Human-Robot Interaction (HRI), 2020.
[36] C. Basu, E. Bıyik, Z. He, M. Singhal, and D. Sadigh, "Active learning of reward dynamics from hicrarchical queries," in IEEE/RSJ International Conference on Intelligen Robots and Systems (IROS), 2019.
[37] R. Holladay, S. Javdani, A. Dragan, and S. Srinivasa, "Active comparison based learning incorporating user uncertainty and noise," in RSS Workshop Model Learning for Human- Robot Communication, 2016.
[38] T. Smith and R. Simmons, "Heuristic search value iteration for POMDPs," in Conference on Uncertainty Artificial Intelligence (UAI), 2004.
[39] M. Egorov, Z. N. Sunberg, E. Balaban, T. A. Wheeler, J. K. Gupta, and M. J. Kochenderfer, "POMDPs.jI: A framework for sequential decision making under uncertainty." Journal of Machine Learning Research, vol. 18, no. 26, pp. 1-5, 2017.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Section 3.3 and section 3.3 provide the key contribu- tions and section 4 documents the experimentation results.
(b) Did you describe the limitations of your work? [Yes] Key assumptions were doeu- mented when applying them and a key limitation of being robust and not resilient was revisited section 5.
(c) Did you discuss any potential negative societal impacts of your work? [No] This paper provides general approach enable collaboration. Based on the guidelines and it being more of a foundational research paper. we did not discuss societal impacts as the list would include most applications of collaboration using a POMDP framework.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The step to in- tegrate the action suggestion as an independent observation was shown section 3.3. The assumption of dependence only on the state was mentioned in that section as well.
(b) Did you include complete proof's of all theoretical results? [N/A] The derivation of an independent update a Bayes' filter is well established, but the key steps were shown in section 3.3.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Ys]] The link the code is provided in the footnote on page 1. All policies for the environments were not provided based on file size, but the policies can generated with provided script.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Experiment details are documented in section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Our approach does not require lot of computation. Computation time and amount not concern and not provided.
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Experiments were built on the POMDPs.jl framework and documented in section 4.2.
(b) Did you mention the license of the assets? [No] POMDPs.l is available free to use, copy, modify, merge, publish, distribute, and/or sublicense and is distributed with an MIT "Expat" License.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "ndbURZ3kU3", "reviewer": "Reviewer_Fv1x", "summary": "The paper looks at the problem of designing a framework to support collaboration between an autonomous system and a human supervisor. In particular, the human supervisor provides action suggestions to the autonomous agent to help overcome any limitations the human could have in its sensor capabilities. The human is assumed to be a potentially suboptimal agent with full observability, while the agent is expected to operate under partial observability. The method proposes to use the action suggestions, by using the human policy as part of the observation model of the underlying POMDP, where the actions suggested by the human would correspond to the full observation. This means that the human suggestions could be used to refine the belief space the agent may hold regarding the current state. However, when implementing the approach, rather than directly using this new observation model during the POMDP planning, they use an approximation where in the human suggestion is used to further refine the belief space only if the suggestion differs from the current policy action. This approximate method is then tested on two benchmark domains (under various conditions), where the paper demonstrates that the human suggestions does make a positive impact on the system performance.", "questions": "I would appreciate it if the authors could respond to the weaknesses and the clarification I had mentioned in the previous section. To repeat these are,\n1. The merit in the assumption that the human would be a supervisor who constantly observes the system as opposed to other interaction paradigms\n2. How the choice to only use the human suggestion to update belief states when they differ from the robot’s policy may have affected the results in section 4.4?\n3. Please clarify, what you mean by the statement that the method doesn’t build a model of the human  \n", "limitations": "I don’t see any specific ethical concerns with the proposed methods. I have listed some of the limitations I have noticed of the proposed method under the weakness and other suggestions section. Most of these relate to practical challenges of interacting with humans. I would strongly urge the authors to consider checking human-factors literature on creating systems that are meant to effectively interact with people.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Developing methods that are able to effectively collaborate with people and incorporate their suggestions into their decision-making process is an important problem.\n- The paper presents a relatively clean and clear formulation. Additionally, the fact that the formulation allows the problem to be directly compiled into a standard POMDP means that one could always leverage the latest developments in POMDP planning/RL for the problem.\n- While the evaluations are limited they do present some intriguing results. Specifically, the results presented in section 4.4 in regards to robustness of the methods to suboptimal human behavior is interesting.\n\nWeaknesses:\nI have two big concerns of the paper:\n\n- Type of Interaction: To me the assumption that a human would be an ever present supervisor ready to provide suggestions to the system seems like the worst way to make use of the human. There are many results from psychology and human-factors (not to mention many high profile accidents) that show that humans are pretty bad in roles where they are expected to constantly pay attention and intervene as required. Humans constantly fall prey to issues like automation bias, where they may overtrust the system when they see that the system is performing as they expected in the beginning. I would argue that it would be better to treat the human as an expensive but highly information source of knowledge about the task. Rather than asking the human to come up with an action, the system could have equally asked the human about the underlying state. To avoid the system bothering the human too much, one could also attach some high cost to such information gathering actions.\n\n- Use of the approximation: It's quite unfortunate that the authors chose not to use the exact formulation for the system evaluation. As the paper itself points out, the human agreeing with the system could also present a lot of useful information. While it is a positive sign that the system is still performing well even while only using a subset of the information, it is not quite obvious to me how the inclusion of this additional information would have impacted the results in section 4.4.\n\n\nClarification:\n- Claims about human model: In multiple places in the paper, the authors claimed that unlike earlier works the proposed method builds no explicit human model (repeated in both section 2 and 3.3). As far as I can tell this is not true. It is true that the proposed method doesn’t make strong assumptions about the policy and in some sense the rationality of the human, however, you do make strong assumptions about the sensor model and the knowledge the human supervisor possesses. The authors assume that the human’s knowledge about the task dynamics is the same as what the system’s dynamics is, or at least that the system knows this model. So in some sense you don’t have to build a human model, because it's known beforehand. Additionally, to generate the human policy from this model you make additional assumptions about the rationality of the human planner. It’s also worth noting that there are many previous works (like [a] and [b]) that don't assume that the human’s knowledge about the task dynamics is the same as what the system knows.\n\n\n\nOther Suggestions\n- User Study: While the current experiments are a good starting point, I would strongly urge the authors to perform user studies to see how effectively their system can actually work with real people.\n- Other forms of information: There is no reason to believe that the information about the underlying state is the only kind of information the human can provide. In future versions, I would suggest the authors consider cases where the human feedback may also reveal additional information about the task, including information about the task reward or potential information about effective strategies the system could use to solve the task.\n\n[a] Reddy, Sid, Anca Dragan, and Sergey Levine. \"Where do you think you're going?: Inferring beliefs about dynamics from behavior.\" Advances in Neural Information Processing Systems 31 (2018).\n\n[b] Gong, Ze, and Yu Zhang. \"What is it you really want of me? generalized reward learning with biased beliefs about domain dynamics.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 03. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Qwf6WegmPF", "reviewer": "Reviewer_pPmT", "summary": "This paper presents a model and experiments to demonstrate the usefulness of (possibly intermittent and/or incorrect) human action suggestions on a POMDP-based system, treating the suggestion not as a direct input but rather as an indicator of the partially-observed implicit state. Simulated experiments on a discrete state problem with an offline generated policy show some improvement using (possibly intermittent and/or partially incorrect) action suggestions from a human with an omniscient view (when sending correct information).\n", "questions": "How do these experiments shed light on your original research impetus of reducing brittle behavior via human suggestions ?\n\nIs the assumption that the “suggested action is not influenced by the previous action” really reasonable, given that one might think the human would be especially wanting to suggest an action if the agent does something bad/wrong/ill-advised?\n", "limitations": "This work seems theoretical enough to not have significant societal impacts (beyond, perhaps, the use of computational agents, but here they have human supervisors!)", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality\nThe authors point out much related work; the specific contribution here is that the action is not taken itself as a suggestion to be used or ignored, but rather as another kind of observation that can update the agent’s estimate of the true state, thus potentially wringing more information from the suggestion than merely following it would (and being more helpful when the suggestions are intermittent).\n\nClarity\nThe paper is generally clear and easy to read. The details of computation of the scaled rational and noisy rational approaches are a bit unclear, but the authors claim to provide code in supplemental material.\n\nI strongly suggest that the authors mention early-on (e.g. in the abstract!) that the human is the one giving the suggestions (one could also view that multiple agents who have distinct partial views of the global state could also provide such suggestions that then fold into the local estimate of the full state, which is what I originally thought when I read the abstract. The paper *does* make this clear in the last line of paragraph 2 in the introduction.\n\nLast paragraph section 2 “we do not build an explicit model” —> “we do not build an explicit model of the suggester”\n\nI assume the notation “o_{o:t}” means all observations from time 0 through the current time?\n\nQuality\nThe abstract and introduction talk a lot about failure and brittle or inconsistent behavior of current MDP-variant-based systems, and that human suggestions could help with that. However, the experimental environments (Tag and RockSample) don’t clearly show these issues. Instead, the versions of the system that take into account the *omniscient* collaborator, who has information not available to the agent, do better (even the naive agent that simply uses suggestions directly). The authors do go to pains to demonstrate that the more complex indirect use of the intermittent (but often omnisciently correct) suggestions as observation about true state require fewer communications and more robust in the face of unreliability of the suggester, but how that might help reduce brittle and inconsistent behavior as discussed in the abstract and introduction is somewhat unclear.\n\n[After rebuttal: just downplay this, especially in abstract/intro, so the reader is not looking at this as the primary contribution. Could mention it later in discussion as a possible area to explore]\n\nAside from that: how many runs of each scenario? The text just says that they “varied” (why not some single, consistent numbers runs?)\n\nSignificance\nThe core idea seems to be a useful one, although the demonstration here is on small, discrete, 2-agent environments without catastrophic failures, brittle behavior, or clear inconsistencies. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K-mn_uJhoN-", "reviewer": "Reviewer_pmf6", "summary": "This paper describes an approach for collaborative decision-making where another agent can make suggestions (in the form of suggested actions) to the subject agent in order to influence their chosen actions, but the subject agent retains autonomy in its own decision-making.  In particular, the subject agents decision-making is modeled as a POMDP and the suggestions from other agents are taken as observations by the subject agent and incorporated in the agent's belief state revision.  Two approaches are provided for representing the stochastic observation function for suggested actions, and empirical evaluation on the Tag and variants of the RockSample domains are employed, demonstrating how incorporating suggested actions is beneficial to the subject agent's utility returns.", "questions": "1) How does the agent calculate a Q value for a state instead of a belief (to be used in Noisy Rational)?  Could this approach generalize to algorithms other than point based value iteration derived solutions (e.g., SARSOP)?  Especially since the vast majority of POMDP research is using online planning using Monte Carlo methods, which is vastly different (and maintains no representation for Q values over states)?\n\n2) From what I remember, a key feature of the SARSOP algorithm compared to previous PBVI-based algorithms is its focus on planning only for beliefs that are reachable from the initial belief.  This gave it a computational speedup in that it could prune out unreachable beliefs and not bother planning for them (in its point-based backups).  However, by adding new observations during runtime (i.e., suggestions from the other agent), you've changed the set of reachable beliefs -- a suggestion might cause the agent's new belief to not be near the ones considered during SARSOP.  In that case, your alpha vectors from SARSOP might not be a good approximation of the value function at your new, previously unreachable belief, and thus we have a new source of approximation error.  What impact might this have on the subject agent's performance in your approach, and how might it be prevented or mitigated?\n\n3) Do you have an idea of why your SARSOP (\"Normal\") performance on Tag is so much lower than originally reported?\n\n4) I appreciate that you included the code of your experiments, but some details of how the domains were changed from their typical implementations would be helpful in the paper.  For example, you state that you changed the opponent behavior in Tag to allow \"more movement away from the agent\", but what does that mean?", "limitations": "The limitations of the approach are adequately described.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of collaborative decision-making is relevant to the multiagent planning community at NeurIPS.  The paper is relatively easy to follow.  I appreciated that it included more than one experimental domain, although the problems are relatively and old (see the POMCP, DESPOT, and other papers for larger, more recent benchmark domains).  The paper was also thorough in testing different parameters to the settings in the experiments (e.g., the \\nu, \\tau, and \\lambda parameters).  I would have appreciated also seeing some theoretical analysis of the approach, such as regret bounds when the two different approaches for estimating the observation function are used.  \n\nI also had some concerns with the approach and its performance.  The Noisy Rational estimate of p(o^s_t | s_t) requires the subject agent to have access to Q(s_t, o^s_t).  However, in POMDPs, the agent typically constructs Q(b, a) not Q(s, a).  So I'm not sure when or how the agent will have access to Q(s, a) to use the approach.  Also, in the original SARSOP paper, the average performance on Tag was -6.13, whereas you've reported -10.7, which calls into question whether (1) your implementation is correct, or (2) you ran it with a reasonable set of hyperparameters (e.g., number of backups).\n\n### Post Rebuttal Period ###\n\nI thank the authors for all of their responses to the reviewers questions and comments, including my own.   I think the idea of treating recommended actions as observations is a clever way to adapt single-agent POMDPs to incorporate advice from another agent (human or artificial), and I think the approaches you proposed for defining an observation function over those new observations are reasonable first approaches (and could inspire additional mechanisms with beneficial properties). The experiments and empirical results were also extensive and used reasonable benchmark problems and baselines. At the same time, without any theoretical analysis (which is increasingly common in papers like this), it is difficult to know how the results might generalize to other domains and situations.  I strongly encourage you to continue this work, and if you were to add some theoretical analysis, I think it would be a valuable contribution to the literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept