Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems
Authors: Rohan R Paleja
Michael Joseph Munje
Kimberlee Chestnut Chang
Reed Jensen
Matthew Gombolay
Venue: NeurIPS 2024 poster (2024)
Keywords: Human-Machine Teaming
Adaptive AI
Primary area: human-AI_interaction
TL;DR: We develop approaches that enable iterative, mixed-initiative team development allowing end- users to interactively reprogram interpretable AI teammates and summarize our user study findings into guidelines for future research.

Abstract: Collaborative robots and machine learning-based virtual agents are increasingly entering the human workspace with the aim of increasing productivity and enhancing safety. Despite this, we show in a ubiquitous experimental domain, Overcooked-AI, that state-of-the-art techniques for human-machine teaming (HMT), which rely on imitation or reinforcement learning, are brittle and result in a machine agent that aims to decouple the machine and human’s actions to act independently rather than in a synergistic fashion. To remedy this deficiency, we develop HMT approaches that enable iterative, mixed-initiative team development allowing end-users to interactively reprogram interpretable AI teammates. Our 50-subject study provides several findings that we summarize into guidelines. While all approaches underperform a simple collaborative heuristic (a critical, negative result for learning-based methods), we find that white-box approaches supported by interactive modification can lead to significant team development, outperforming white-box approaches alone, and that black-box approaches are easier to train and result in better HMT performance highlighting a tradeoff between explainability and interactivity versus ease-of-training. Together, these findings present three important future research directions: 1) Improving the ability to generate collaborative agents with white-box models, 2) Better learning methods to facilitate collaboration rather than individualized coordination, and 3) Mixed-initiative interfaces that enable users, who may vary in ability, to improve collaboration.

Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems
Rohan Paleja
Michael Munje
MIT Lincoln Laboratory Lexington, MA 02142 rohan. paaaaaaalejj@111 mit. edu
The University of Texas at Austin
Austin, TX 78712
mi michaeelmunje@utexas edu
Kimberlee Chestnut Chang, Reed Jensen
Mathew Gombolay
MIT Lincoln Laboratory Lexington, MA 02142
Georgia Institute of Technology
Atlanta, GA 30332
{chestnut, rjensen}@11 wwt. edu
natthew.ggmmollyy@c... wwwps//wwwc.gttccc.ec. gatech.edu
Abstract
Collaborative robots and machine learning- caaning-basedd virtual agents increasingly tering the human workspace with the aim increasing productivity and enhancing safety. Despite this, show in ubiquitous experimental domain, Overcooked-AA that state-of-the-an techniques for human-machine teaming (HMT), which rely on imitation reinforcement learning, brittle and result in machine agent that aims to decouple the machine and human's actions to act independently rather than in synergistic fashion. To remedy this deficiency, develop HMT approaches that enable iterative, mixed-initiative team development allowing end-users to interactively reprogram interpretable Al teammates. Our 50-subject study provides several findings that we summarize into guidelines. While all approaches underper- form a simple collaborative heuristic (a critical. negative result for learning baased based methods), we find that white-box approaches supported by interactive modification can lead to significant team development, outperforming white-box approaches alone, and that black-box approaches are easier to train and result in better HMT performance, highlighting tradeoff between explainability and interactivity versus ease-of of-training. Together, these findings present three important future research directions: 1) Improving the ability generate collaborative agents with white-box models, 2) Better learning methods to facilitate collaboration rather than individu- alized coordination, and 3) Mixed-initiative interfaces that enable users, who may vary in ability, to improve collaboration.
Introduction
Successfu[ hamaaouummnnmnnna machine teaming (HMT) has long been sought after for its wide utility across potential applications, ranging from virtual agents such as "clippy" that provide on-demand support
DISTRIBUTION STATEMENT A. Approved for public release. Distribution unlimited. This material is based upon work supported the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702- 15-D-0001 Any opinions, lindings. author(s)
not necessarily reflect Secrelary of Defense for Research and Engineering. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 DEARS 252.227-7014 detailed above. Use of this work other than specifically authorized by the U.S. Govemment violate any copyrights that exist work.
conclusions or recommenndations expressed in material are
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

for improving documents to embodied robotic healthcare aides that can provide doctors with helping hand (2). While promising, achieving fluent HMT is challenging because interactions with humans can be incredibly complex due to the diversity across users [26]. human teammates benefit from explainable systems to support the development of mental models [1.. and the lack of bidirectional communication (i.e., unclear how humans can "tell" a machine online to perform a desired behavior) [46]. In this paper, we transition from the conventional approach of crafting an HMT solution that aims for flawless out-of-the-box performance to a paradigm where end-users can actively interact with and program AI teammates, fostering a more dynamic and developmental interaction between humans and AI. Specifically, we explore enabling humans to perform user-specific modifications to a collaborative Al's interpretable policy representation across repeated iterations of teaming episodes and provide set of design guidelines to support team development in HMT drawn from large-scale user study.
Recently, data-driven techniques (e.g., imitation and reinforcement learning) have become popular in HMT, allowing for the generation of collaborative agent behavior without cumbersome manual programming [40 []. However, these prior works utilize opaque, black-box models, limiting human's ' ability to develop a shared mental model and maintain situational awareness [27], crucial for high- performance teaming 377 We posit that successful, real-world HMT is not feasible without the use of white-box methods, especially in safety-critice domains such as healthcare and manufacturing. Furthermore, collaborative interactions with machines have often lacked the ability effectively learn with and adapt to human teammates in real-time [9]. In ad hoc numan-humaa teams, effective teaming often developed through iterative process 4 Bi-directional communication is often key component of this process, enabling the development of successful coordination strategies [36). In our work, build towards such team development paradigm in HMT by 1) creating a pathway of bi-directional communication, utilizing interpretable policy representations as a mechanism to allow users to understand their machine teammates and allowing for explicit teammate policy modification through an interface (users can modify the machine's 's tree-based policy via a GUT), and 2) allowing for the process of iterative mixed-initiative team development through repeated teaming episodes. We believe this paradigm is necessary because human-partneree partnered systems need explainable components and adaptable systems. We provide the following contributions:
We provide case study regarding prior work in HMT 40) finding that the generated machine behavior is unable to adapt to human-preferred strategies, and that high performance is typically driven by independent machine actions rather than collaboration, which can ultimately result in higher team score.
We create novel InterpretableMLL architecture to support the creation of tree-based cooperative agent policies via reinforcement learning and a GUI to allow users to modify the Al's behavior to their specifications. This capability promising, enabling end-users to "go under-the-hooke of machine learning models and tune affordances or interactively and iteratively reprogram behavior. crtween-subjeee user study assessing the effects of interpretability
We conduct a 50 50-participant and interactive policy modification across repeated interactions with an AL. We summarize our study findings into a set of design guidelines to support future HMT research.
2 Preliminaries
Here, we introduce prior work in HMT and Explainable AI, our experimental domain, Overcooked- AI, model of team development used to understand our findings. Tuckman's 's Model, and the mathematical framework under which we generate agents, Markov Games.
Human-Machine Teaming - The field of HMT is concerned with understanding, designing, and evaluating machines for use by or with humans 6 44 30 A popular technique that has been used to produce collaborative AI agents is Reinforcement Learning (RL) (28). where researchers have concentrated efforts on reducing the dissimilarity between synthetic human training partners and testing with human end-users. Approaches that have achieved some success include utilizing human gameplay data to finetune simulated training partners to behave more human like [5), which can be expensive, and training with diverse-skille population of synthetic partners to create an agent that can better generalize to non-expert end-users (40), which may bias the Al teammate to exhibit individualized strategies, we display in Section 3. We note our work focuses on interaction different from Al-assisted decision-makin or decision support. Here, a human and an agent must collaborate across a series of timesteps, aiming to maximize multifaceted joint objective function.
2

---

Explainable AI xAI is concerned with understanding and interpreting the behavior of Al systems 23 In our work, we follow recent trends that show black-box methods paired with local explanations can be harmful 34 and utilize interpretable, white-box tree-based models in a multi-agent sequential decision-making problem. These models have been shown to beneficial improving the user's ability to simulate decision-making model [42 and providing users with increased situational awareness over a teammate's behavior in an HMT setting [31]. While tree-based models can provide users insight into the model, the complexity of the tree-based model limits its utility (24). While we note this as a potential weakness of utilizing tree-based models, effective state representations can provide tradeoff between granular control and tree depth. Accordingly, we design our trees to reason over a state-space with high-level binary features and multi-step macro-actions, expanded on below. Furthermore, in our work, we explore a paradigm where user directly modify and visualize a tree-hased Al teammate the user is interacting with after a teaming episode. Prior work in explainable debugging [18] and robotics [R] 2 has explored similar paradigms, creating interactive systems that allow end-users to modify agent behavior to increase performance, but has not explored deploying tree-based models trained via RL in a collaborative HMT setting. We provide working definition of what we mean by "interpretable" within the Appendix Section G
Overcooked-Al Overcooked-AI [5] is testbed to evaluate human-Al interaction and has been used across HMT research concerned with collaboration [40], teammate identification [4], intention prediction 45 and behavior influence [16]. Here, two agents tasked with creating and delivering as many soups as possible within given time. Achieving a high score requires agents to navigate a kitchen and repeatedly complete set of sequential high-level actions, including collecting ingredients, placing ingredients in pots, cooking ingredients into a soup, collecting a dish, getting the soup, and delivering it. Both players receive the same score increase upon delivering the soup. We modify the original Overcooked-Al game to be a simultaneous-moo move game as opposed to the original formulation of allowing agents to perform actions asynchronously, This modification prevents the collaborative score metric from being dominated by super-human AI speed, causing the overall score to be more reliant upon effective collaboration and strategy. We provide details about the state and action space below and complete details in the appendix.
State-Space:
Policies reason over semantically meaningful feature space as opposed to pixel space, detailing the objects each agent is holding, pot statuses, and counter objects. This state space allows for learning an interpretable rrsed policy that can be understood and manipulated by end-users.
Sction-Space: Instead of using cardinal actions, we allow the Al to utilize macro-actions that can accomplish high-level objectives such as ingredient collection, ingredient placement, and soup serving. Macro-actions are planned using an A * planner, and we perform dynamic replanning at each timestep. Constructing trees higher level of abstraction results in smaller trees that are easier to interpret.
Tuckman's 's Model Tuckman 43 reaching high performance, including "Forming", "Storrming,. "Norming" and "Performing," often seeing a drop in performance as team members acclimate, followed by rise as team members understand how to collaborate. Assuming that human- machhne teams will follow similar stages to numan-human teams, this paper looks into how we can support suman-maching teams in reaching the Performing stage, where the team is achieving its full potential and exhibiting the highest level of cooperation. We provide a depiction of these stages as part of Figure
describes the different stages that team goes through before
Markoy Game - We formulate our setting as a Markov Game 25 defined by a set of global states, S1, S2 E S, set of actions, AI, A2 A, transition function, T: S x Ai X A + S. and reward function r; S x A; + RR. Agent i aims to maximize its discounted reward R. = Er= 7/r,, where E [0, 1] is a discount factor. For training, we utilize agent-agent collaborative training, which trains two separate agents jointly via single-agent PPO. We utilize PantheonRL (88 for training our agents, incorporating our novel tree-based architecture (Section 4.1 into the codebase.
3 A Gap in Teaming Performance
In this section, we present two examples to display gap in the quality of AIs in HMT. Specifically, we look at two recent approaches to produce collaborative AI agents (40 5. We argue and display that the AIs trained via these approaches are rigid and exhibit individualized behaviors, missing out on collaborative teaming strategies that can ultimately result in higher team scores. We require Al agents that can effectively reach consensus with humans on a leaming strategy that ultimately results in high performance. In cases where the human has a preferred strategy, the Al teammate should be able to support said strategy.
3

---

(a) We display the numan-preferre collaboration behavior that focuses on minimizing agent movement and efficient handoff's using the middle counter. This unsuccessful HMT receives a score of O.
(b) We display human adapting to Al-preferred suboptimal teaming strategy, where agents act indi- vidually. This individualized coordination results in minor success, achieving a low score of 40.
Figure 1:
Case Study in Machnne Teaming with Different Teaming Strategies. It is clear that the models are not robust to multiple strategies of play and can result in agents performing nonsensical behavior (e.g.. stuck in place).
In Figure D we display the Coordination Ring scenario. A simple collaboration strategy (which term "uuaan-preferee") in this domain is utilize the counter to continuously pass objects, minimizing agent movement through efficient handoffs. To test a set of collaboration strategies, we utilize agents publicly available from Carroll al. [5]. In Figure [I] we display a frame-by-frame of the puman-preferred coordination strategy (Figure Lab and Al-preferred coordination strategy (Figure [ which was strategy where agents act individually to collect ingredients and place them in pots. The latter behavior was inferred through repeated play with the publicly-availann With the human- meena--reeerred strategy, the AI agent freezes for the majority of the game, creating an extremely frustrating and low wow-perfrrming AI teammate, In this scenario, the human (green) picks up an ingredient and places it on the counter the start of the game. The AI agent (blue), unfamiliar with this teaming strategy, freezes for approximately 80% of the remaining episode before finally placing an onion in the pot. With the Al Al-preferred strategy, the human is able to successfully team with the AI, with each agent retrieving and placing ingredients while moving in clockwise motion, but the strategy is not optimal or what the human prefers. As the Al produced by Carroll et al. (5) is created via RL teaming human-like AI teammates, the generated behavior may not be ideal for the current teammate, especially the current teammate's s preferred strategy was not present the original training dataset used create human-liks like AI training partners. This highlights a need for systems that can explain strategies exhibited by trained agent policies and allow humans to adapt these pre- pre-trained policies toward human human-prefene preferred behavior.
AL.
In second example, we utilize the Optional Collaboration domain, displayed on the right- side of Figure 4b which is also utilized in our human- munan-s experiment. This domain was designed to incentivize collaboration, where creating mixed- iinrediigeedient dishes facilitated by agents passing ingredients across the central counter will result in a higher score per dish. Here, we program two intelligent deterministic heuristics: In the first, each agent acts completely individually, cooking single-ingredient dishes and serving. In the second, agents share ingredients, which costs additional timesteps, but are able to successfully cook mixed ingredient dishes. We find that the collaboration strategy achieves 408 cumulative team score, approximately 30% more score compared to the individualized strategy of 306. However, we find that trained policies under Ficticious Co-Play [40] exhibit similar team score that of the individual coordination strategy and further, find that real human end-users collaborating with these agents are unable to far surpass the individual strategy score. As Strouse et al. (40) trains an agent work well with a population of agents, where approximately a third of the diverse-skilled population of agents used in training are completely random agents, we posit that the teammate agent must compensate and exhibit individualized behavior, limiting the algorithm's 's ability to effectively learn effective team coordination strategies. In line with the first case study, the trained collaborative agent policies miss out -eghorerformanc teaming behaviors, and thus, we need systems where humans can iteratively improve agent behavior online.
Thus, in the rest of the paper, we look to explore xAl techniques a mechanism for closing this gap and allowing agents within a human-man maman-- team to facilitate collaborative strategies that outperform the individualized and rigid behaviors trained agents assume.
4 Methodology
In this section, we first present our architecture for training interpretable Al teammates. We then present contextual pruning algorithm, allowing for ease-of-training and enhanced interpretability for neural bree-based models. We display overview of our training procedure as part of Figure

---

4.1 Interpretable Discrete Control Trees
We create an interpretable machine learning architecture, Interpretable Discrete Control Trees (ID- CTs). that can used directly with RL to produce interpretable teammate policies. Below, we briefly detail our architecture, as well as advancements to enhance ease-of -fainnnnnt and interpretability.
Architecture Our IDCTss are based differentiable decision trees (DDTs) 41 neural network architecture that takes the topology of decision tree (DT). DDTs contain decision nodes and leaf nodes; however, each decision node within the DDT utilizes sigmoid activation function (i.e., a "soft" decision) instead of a Boolean decision (i.e., a "hard" decision). Each decision node, i, represented by a sigmoid function, displayed representation is difficult to interpret, Paleja et al. (32) presented differentiable crispification, which recasts each decision node split upon single dimension of the imput feature and translates the outcome of a decision node so that the outcome is Boolean decision rather than a set of probabilities. This, in turn, allows for an interpretable forward propagation through the model that traces down single branch of a tree as well as gradient flow afforded by the straight-through trick to update parameters of the neural tree model. We utilize this approach to learn interpretable tree-based teammate policies via reinforcement learning.
yt (1 + exp(-a(wj)
b4)))= As this
We initialize our IDCTs be symmetric DTs with N₁ decision leaves and N₂ -1-11 1 decision nodes. Each decision leaf is represented by a sparse categorical probabil- ity distribution over actions. At each timestep, a state variable is propagated through each de- cision node, split on a single de- cision rule, with the output be- ing a Boolean causing the deci- sion to proceed via the left or right branch until arrival at a leaf node. At each leaf node, sam- ple from the respective distribu- tion to produce a macro-action (e.g., in Overcooked-Al, "get an onion" or "place ingredient counter"). Further, we improve model predictability by applying an L1 norm loss over leaf node distributions to ensure sparsity, penalizing high entropy action distributions at lea Importantly, the resultant representation after training is that of simple decision tree with categorical probability distributions each leaf node.
Training in Simulation
Contextual Pruning
Interpretable Al Policy
Simulated Human Policy
Reduced-Size Tree Pollicy
Interactive HMT
Team Development
Team Effectiveness
Performing
Policy Modification and Visualization
Forming
Norming
Storming
Teamiing Episode
Time
Figure 2: Here, we provide overview of the steps to produce a collaborative Al teammate with an interpretable policy and the proposed policy modification scheme evaluated in our user study.
Contextual Pruning As we focus on creating agents that must cooperate with and be interpreted by humans, we must limit the size of our tree-based models to a certain depth promote user understanding. Analogous to the "lottery ticket hypothesis"
network training that supports the practicality of employing large models (11), small tree with a limited number of sub-trees (lottery tickets) may not have the representational power to learn a nigh-performing policy. Thus, the ability effectively train IDCT's is odds with maintaining user readability and simulatability. Following work in neural network pruning [22), we design a post-hoc contextual pruning algorithm that allows us simplify large IDCT models while precisely adhering model behavior by accounting for:
Boundaries of variable's state distribution: We utilize the minimum and maximum of each variable's 's range to parse impossible subspaces of a tree.
Node hierarchy: Ancestor nodes for specific decision node may have already captured specific splitting criterion and, thus, may lead to redundancy. By detecting redundancies, we can prune subspaces the tree.
'Wiie utilizing deterministic Al policies may be easier to understand for users, we found these models could not converge to similar performance the stochastic-leat IDCT policies during training.
5

---

We provide further details and an algorithm for contextual pruning in the supplementary material. This, in nurn, allows us the benefit of training large tree-based models, greatly improving ease-of-training while still being able to simplify the resultant model to a smaller, equivalent representation.
4.2 Modifying an Interpretable Policy
While the above architecture can be used alongside RI. to produce a collaborative Al policy, the result may not actually be helpful what the human wants. Humans, when leaming with machines, should be able to intuitively update what the robot has learned or change it based upon preferences that may evolve over time. Such is critical in the positive development of coordination strategies and is associated with the calibration of trust, assignment of roles, and development of a shared mental model. As such, we propose a policy modification scheme that allows the user repeatedly team with an Al maintaining an IDCT policy, visualize the current behavior in tree form, and modify its Al's behavior. The iterative process generated through this scheme can facilitate a feedback loop, allowing for the possibility of team development and improved HMT performance over teaming episodes.
Dectsion Node
Decision Node
Action Node 3
Action Node
Action Node 2
Action
Action Node
Tree Deepening (Enhance Complexity)
We term our modification scheme human-led policy modification. We provide humans with an explicit path- way to "communicate" with an Al af- ter each teaming interaction through
Onion Pat?
you Holding Onion?
Action Node
Action
Action Node
GUI, with capabilities displayed in Figure 3 Within this interface, users start with the pre-trained collabora- tive AI IDCT policy and can modify the Al's behavior by creating new tree structure that may vary in what state features appear in the decision nodes, actions taken in leaf nodes, and the respective probabilities of actions within the leaf node. It is important to note that users are limited expand- ing the tree to a depth of four (i.e., a max of 16 leaves), and the modifica- tion is not timed.
Decision Variable Modification (Enhance Complexity)
Decision Node 1
Decision Node 1
Get Tomato
Action Node 2
Get Onlon
Action Node
Leaf Node Modification (Modify Actions and/or Associated Probabllities)
Figure Users have several capabilities creating an ef- fective teammate, including modifying the tree structure by adding or removing decision nodes, changing state features the tree is conditioned on, and modifying actions and/or their respective probabilities at leaf nodes.
4.3 Trained Collaborative Teammate Policies
Across experiment, study collaboration in two domains, Forced Coordination and Optional Collaboration, displayed on the left-hand side of Figure 4 In each domain, we train an IDCT policy via agent-agen collaborative training and neural network (NN) policy following the population- based training scheme in Strouse et al. 400. In the first domain of Forced Coordination, the IDCT policy converged to policy with average reward of 315.22 1 14.59, and the neural network policy converged to an average reward of 403.16 + 16.08 evaluated over 50 teaming simulations with the synthetic human teammate the policy was trained with. In the second domain, Optional Collaboration, the IDCT policy converged to a policy with an average reward of 171.46 18.89, and the neural network policy converged an average reward of 295.02 = 1.86. Thus, a consequent confound due to the current difference in performance capabilities between interpretable vs. black-box models is that the NN policy outperforms the IDCT policy in both domains. This displays need for improving optimization algorithms for interpretable models representing collaborative agent policies. However importantly, while the initial simulated performance of interpretable models may underperform black-box models, the ability for humans to understand machine behavior and improve upon behavior may allow these appreaches to compete or even outperform black-box NN models. We can also compare to the heuristic policies presented in Section 3, observing that the training performance of the IDCT and NN policies in the Optional Collaboration domain underperform the collaborative heuristic (408 vs. 295.02 and 171.46). We provide visualizations of the trained IDCT policies for

---

each domain in the appendix, finding that after contextual pruning, the Al IDCT policy has two and three leaves, respectively.
5 Human-Subjects Study
Here, we discuss our between-subjects user study that seeks understand how users interact with an Al across repeated play under different factors. Below, we introduce our research questions, provide a description of the independent variables and procedure, and discuss our findings.
Research Questions The presented research questions below seek to understand changes in overall numan-machine teaming performance and performance changes across repeated gameplay. The latter question pivots from an episodic attitude of teaming to longer-term gauge, allowing us to study the process of adaptation in HMT.
1. RQI: How does human-machine teaming performance vary across factors?
2. RQ2: How does team development vary across factors?
Independent Variables We have two independent variables, IV1: the teaming method, and IV2: the domain. For IV1, we consider the following conditions (abbreviated by IVI-C):
1. IV1-C1: Human-Led Policy Modification: After interacting with the agent (one teaming episode), the user can modify the policy via the GUI, allowing the user to update decision nodes and action nodes in the tree as well as tune affordances. Upon completion, the user can visualize the updated policy in its tree form prior to the next interaction.
2. IV1-C2: AI-Led Policy Modification: After interacting with the agent, the Al utilizes recent gameplay to fine-tune a human gameplay model via Behavioral Cloning and performs reinforce- ment learning for five minute to optimize its own policy to better support the human teammate. Upon completion of policy optimization, the user can visualize the updated Al policy in its interpretable tree form prior to the next interaction. This is similar to HA-PPO [5, adapted to an online setting.
3. IV1-C3: Static Policy Interpretability: After interacting with the agent, the
visualize the Al's policy in its interpretable tree form prior to the next interaction. Throughout this condition, the Al's policy is static.
IV1-C4: Static Policy Black-Box: After interacting with the agent, the user does not see the Al's policy. Here, the Al policy is the same IVI-C3, but the human has lost access to direct insight into the model.
5. IV1-C5: Static Policy Fictitious Co-Play: 40)
User teams with an AI maintaining static black-box, neural network (NN) policy trained across a diverse partner set. As this is a baseline, utilize an NN rather than the legible IDCT policy used in other conditions (IVI:C1-4).
For IV2, we consider the following domains displayed on the left-hand side of Figure
1. IV2-DI: Forced Coordination:
Users team with an Al that is separated by barrier and must pass over items in a timely manner. Here, agents are forced to collaborate.
IV2-D2: Optional Collaboration: In this domain, the team can operate individually or collabo- ratively. This domain has increased complexity, both with respect to the size of the domain and the types of soups that can be cooked. Collaboration is incentivized through a higher reward for mixed-ingredient dishes (combining onions and tomatoes) over single-ingredier dishes.
Procedure: A participant is first randomly placed into one of the five conditions in IV1. The participant starts with a vreeexperiment sur- vey collecting demographic information, expe- rience with video games and decision trees, and the Big Five Personality Questionnaire [7]. Af- terward, participant conducts brief tutorial in Overcooked with random Al agent, improving
Table 1: A comparison across different IV1 factors.
Explicit Interaction
Changes Berations
Approches IVF-CI
White-Box
Policy
TDCT
IVL-C2
IDCT
IVL-C3
IVL-C4
x
x
IDCT
IV1-C5
x
x
NN
teammate to five minutes create a feasible user-study. This RL optimization is challenging as only a limited number of samples can be obtained in this time, and thus, the policy is not guaranteed to improve. In cases where the policy degrades, we use the original policy prior to optimization.
We limit the online optimization time for
7

---

Led Modification
Static (elack Box)
Mudiltcation
(Interpnatable)
Co-Play
Reward
900
200
100
Iteration
(a) Performance Data in IV2-D1: Forced Coordination.
Human-ted Policy Modification
Al-eed Policy Modification
Static [Ineeppetabble]
Static jBlack- Box)
Fictitious Co-Play
300
230
2000
1.0
50
Iteration
(b) Performance Data in IV2-D2: Optional Collaboration
Figure 4: User gameplay scores across teaming iterations with per-iteration means connected by the red dotted line and the per-iteration standard deviation shaded in red.
the user's understanding of game controls and the assigned task. Once completed, the primary experi- mentation begins. Users will team with an Al four times in each domain (randomly ordered), starting with the unique domain- -spccfn pre-trained agent, and are told that their goal maximize their score in the last teaming interaction, the "performance round." After each teaming interaction, in the first three factors, the user will modify and visualize Al's policy (IVI-CI), the Al will optimize its own policy proceeded by user visualization (IV1-C2), or the user will solely view the policy (IV1-C3). IV1CC4 and IV1-C5, as the AT black-box (perceived to be black-box in IV1-C4 and truly black-box in IV1-C5), transitionary pages shown to the participant, providing them pause before they team with the agent again. Upon completion of the condition-sppi cspconn--pcfficcec (or lack of) actions, users complete NASA-TLX A-LX Workload Survey. After users have completed domain, providing us with four episodes of teaming data and workload assessments, we administer several post-study scales, including the Human-Ronn -Rboot Collaborative Fluency Assessment [15]. Inclusion of Other in the Self scale [1, and Godspeed Questionnaire (2]. Upon completion of the two domains, the experiment concludes.
5.1 Results
Our experiment is 5 (teaming method; between-subjects) x 2 (no. of domains; within-suujects: x (no. of repeated evaluations; within-subjects) mixed-factorial experiment. We recruited 50 participants under IRB-approved protocol, whose ages range from 18 to 32 (Mean age: 24. 14; Std. Dev.: 40; 46% Female, 52% Male, 2% Non-Binary), with participants randomly assigned to each of the factor levels, with ten total subjects per level. The duration of the experiment was 70.98 + 19.71 minutes 5 Our data was modeled as full-factorial, between-subjects ANOVA. We test for normality and homoschedascity (see appendix) and employed corresponding non- parametreeei test if the data failed to meet these assumptions. display our objective findings the right-hame side Figure 4
RQ1: Team Performance: In analyzing reward, we find trends with respect to the maximum reward participants obtained within a domain across iterations (Figure 55. Using Friedman's test, find significant difference across domains (x2 (1)(4006,, 0.001) and analyze the domains separately.
'The significant variance experiment duration arises from granularity across our conditions. The increase in human effort to understand and interact with the policy results an increase duration. We note that as our experiment is relatively short, it is unlikely that experiment fatigue played role in our results as would be common in experiments with large task variances.

---

In IV2-D1, a Kruskal-Wallis Test was conducted to analyze differences in maximum performance
obtained across teaming paradigms, finding significant effect (x) = 20.146, p 0.001). We conduct post-hoc pairwise comparisons, utilizing Dunn's s test, and find that IV1-C5 (Fictitious Co- Play) is significantly better than IV1-C1 (p < 0.001). IV1-C2 (p < 0.01), IV1-C3 (p < 0.01), and IV1-C4 (p < 0.05). Even though Fictitious Co-Play (IV1-C5) outperformed the tree-based models, likely due to its ability to converge to a higher- Sether-performncee teaming policy, interesting that Human-Led Policy Modification (IV1-C1) has several participants that outperform the maximum performance of IV1-C5 in teaming iterations three and four (Figure 4a)
In IV2-D2, Kruskal-Wan Test was conducted to analyze differences in participant teaming performance across conditions. finding a significant effect (x (x) = 29.992,, p < 0.001). We conduct post-hoe pairwise comparisons, utilizing Dunn's ' test, and find that IV1-C5 (Ficticious Co-Play) significantly better than IV1-C2 (p < 0.001), IV1-C3 (p < 0.001), and IV1-C4 (p < 0.001), and IV1-C1 (Human-Led Policy Modication) is significantly better than IV1-C2 (p < 0.05). IV1- C3 (p < 0.05), and IV1-C4 (p < 0.05). For white-box Al teammates (IVI:C1-3), the latter finding displays the benefit of Human-Led Policy Modification in improving HMT performance for interpretable models. These findings display that 1) white-box approaches supported with policy modification can outperform white-box approaches alone, 2) black-box models can outperform white-box approaches in HMT, and 3) by comparing IV1-C3 to IV1-C4, interpretability alone afforded via tree visualizations did not provide any direct objective benefits. Finally, in Optional Collaboration, across all conditions we see that HMT scores are not near that of the collaborative heuristic, displaying a gap that must be addressed to achieve effective HMT.
RQ2: Team Development: In analyzing RQ2, we look at the change in reward across iterations one to four and relate our findings to Tuckman's 's model. Utilizing Friedman's 's test, we find a difference across domains (x2(1)==0..4)) 48, p<0.001) and analyze the domains separately. IV2-D1, we see that none of the conditions results in a significant improvement in teaming performance over repeated iterations. IV2-D2, we see IVI-C1 (p < 0.01) and IVI-C2 (p < 0.01) significantly improve over repeated teaming interactions. The improving interactions can be connected to the Norming stage in team development, where teams begin to develop a strategy and team mental models. We see conditions that facilitate Norming have the attribute of policy adaptation and are white-box.
Next, we analyze whether different pprson-specific factors allow HMT to improve more quickly than others. In IV2-D1, we find that conscientiousness is trend- ing in its correlation with im- provement (0.05 < P < 0.1).
Stale Talcy
Scatic
Interp-stable
In IV2-D2, we find that partic- ipants with high familiarity with
Reward
300
Trees improve more across it-
200
erations (F(1) = 7.448, <
0.01). These findings signify that positive interaction with in- terpretable models may be more
100
0
Forced Coordination
Optional Collaboration
beneficial to those with an engi- neering background and specific personality traits.
17.5
15
Aggregate Likert Score
Finally, we detect an interesting trend in IV2-D1 under the IV1- C1 condition. We see drop in performance between the first teaming iteration and later iter- ations, followed by a rise. We believe this relates to the Form- ing and Storming stages, where team members are still develop- ing effective strategies to coordinate. As the last iteration shows an improvement in performance, we hypothesize that the team was shifting into the Norming stage. In future, it would be interesting to evaluate larger number of iterations to see if the behavior would continue to uptrend. This requires further research due to the additional resources and time needed for more teaming iterations.
10
7.5
5
2.5
Positive Teaming Traits
Teaming Fluency
Figure 5: Maximum Reward and Subjective Ratings Across IV1 Factors.

---

Subjective Findings: In IV2-D1, we find that users did not find any subjective differences toward the teaming interaction across conditions. In IV2-D2 (Figure we find that users find collaboration with Als under condition IV1-C2 and IV1-C4, on average as less fluent than IVI-C1 (p<00.1, p<00.01,, and IV1-C4 as less fluent than IV1-C5 (p < 0.05). Users also trusted the AI and perceived the Al contributed more in IV1-C5 than in IV1-C2 (p<00.55, p<0.05) and VVICC4 (p < 0.05, < 0.05). Furthermore, the users viewed the Al more positively in IV1-C1 and IV1-C5 than in both IV1- C2 (p<0005,, p=0.05) and IV1-C4 (p<0.05,, p<0..1). Overall, participants generally assessed higher-performing agents more positively in their subjective ratings. In considering conditions that utilized tree-based model (IV1-C1, IV1-C2, IV1-C3, and IV1-C4), interaction with the tree policy provides significant subjective benefits in positive teaming traits and collaborative fluency (defined within the Human- Roboot Collaborative Fluency Assessment (2)). In including the remaining condition, which utilizes a black-box model, IV1-C5: Fictitious Co- Play, and comparing it IV1-C1: Human-Led Policy Modification, we that even though Fictitious Co-Play outperformed Human-Led Policy Modification in terms of team reward (though not significantly in the domain of Optional Collaboration), no significant subjective differences were observed between these two conditions. This presents an interesting relationship between transparency, interaction, and performance in relation subjective perception that warrants future research.
the addition of
Design Guidelines: To achieve fluent HMT, specify the following forward- facingg guidelines.
1.
The creation of white-box learning approaches that can produce interpretable collaborative agents that achieve competitive initial performance to that of black-box agents. This guideline is critical to providing humans with the subjective benefits obtained from interactivity with white-box models, objective benefits of black-box models, and the ability to interact with policies to facilitate team development.
2. The design of learning schemes to support the generation of collaborative Al behaviors rather than individual coordination. We need techniques that avoid converging to the local maxima of individual coordination and scenarios that allow for properly evaluating cooperation.
3. The creation of mixed-initiatio ence, to improve team collaboration across and within interactions. As we found a large diversity in perceived usability of our interface (finding average score of 58.25 + 27.61, with some users finding the interface good (>75) and others poor (<35)), effective interfaces are vital in shifting from only a subset of users benefiting to all users being able to create effective teammates.
interfaces that enable users, who may vary in ability and experi-
4. The evaluation of leaming in a larger number of interactions. As agents are deployed, team perfor- mance will change over time, going through transient period before reaching peak performance. Understanding this process of team development is essential in creating high- high-performance HMT.
Conclusion
This work investigates repeated interactions with machine learning models within a sequential decision-making HMT paradigm. We present a key gap in HMT, displaying that current methods do not facilitate human-machine collaboration to the fullest. We find that human-led policy modification allows for a team to achieve higher performance than white-box models without this capability. However, as interpretable models are more difficult to generate, Fictitious Co-Play is able to better support high performance. Given these mixed findings, future work must focus on developing better white-box teammates, study the modality of communication in HMT, and explore mechanisms to allow HMT to scale beyond individual coordination and toward effective collaboration.
References
[1] A. Aron, E. Aron, Michael Tudor, and Greg Nelson. Close relationships as including other in the self. Journal of Personality and Social Psychology, 60:241-253, 1991.
[2] Christoph Bartneck, Dana Kulic, E. Croft, and Susana Zoghbi. Godspeed questionnaire series. 2019.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville, Estimating or propagating gradients through stochastic neurons for conditional computation. ArXiv, 1bs.1333,3333, 2013.
10

---

[4] Benjamin Beyret, Ali Shafti, and A. Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5014-5019, 2019.
[5] Micah Carroll, Rohin Shah, Mark K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. In NeurIPS, 2019.
[6] Jessie Y.C. Chen and M. Barnes. Human-agent teaming for multirobot control: A review of human factors issues. IEEE Transactions on Human-Machine Systems, 44:13-29, 2014.
[7] Michael S. Chmielewski and Theresa A. Morgan. Five-Factor Model of Personality, pages 803-804. Springer New York, New York, NY, 2013.
[8] David B. D' Ambrosio, Saminda Abeyruwan, Laura Graesser, Atil Iscen, Heni Ben Amor, Alex Bewley, Barney Reed, Krista Reymann, Leila Takayama, Yuval Tassa, Krzysztof Choromanski, Erwin Coumans, Deepali Jain, Navdeep Jaitly, Natasha Jaques, Satoshi Kataoka, Yuheng Kuang, Nevena Lazic, Reza Mahjourian, Sherry Moore, Kenneth Oslund, Anish Shankar, Vikas Sindhwani, Vincent Vanhoucke, Grace Vesom, Peng Xu, and Pannag R. Sanketi. Achieving human level competitive robot table tennis, 2024.
[9] Daniela Fogli, L. Gargioni, Giovanni Guida, and Fabio Tampalini. A hybrid approach user- oriented programming of collaborative robots. Robotics Comput. Integr. Manuf., 73:102234, 2022.
[10] Matthew C. Fontaine, Ya-Chuan Hsu, Yulun Zhang, Bryon Tjanaka, and Stefanos Nikolaidis. On the importance of environments in human-robot coordination. In Dylan A. Shell, Marc Toussaint, and M. Ani Hsieh, editors, Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021.
[11] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv: Learning, 2018.
[12] Ali Ghadirzadeh, Xi Chen, Wenjie Yin, Zhengrong Yi, Märten Björkman, and Danica Kragic. Human-centered collaborative robots with deep reinforcement learning. IEEE Robotics and Automation Letters, 6:566-571, 2020.
[13] Abhishek Ghose and Balaraman Ravindran. Interpretability with accurate small models. Fron- tiers in Artificial Intelligence, 2020.
[14] Cong Guan, Feng Chen, Ke Xue, Chunpeng Fan, Lichao Zhang, Ziqian Zhang, Pengyao Zhao, Zongzhang Zhang, Chao Qian, Lei Yuan, and Yang Yu. One by one, continual coordinating with humans via hyper-teammate identification, 2024.
[15] Guy Hoffman. Evaluating fluency in human-robot collaboration. IEEE Transactions Human-Maching Systems, 49:209-2111 01099221,770 2019.
[16] Joey Hong, Sergey Levine, and Anca Dragan. Learning to influence human behavior with offline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[17] Matteo Iovino, Jonathan Styrud, Pietro Falco, and Christian Smith. A framework for learning behavior trees in collaborative robotic applications. 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pages 1-8, 2023.
[18] Todd Kulesza, Margaret M. Burnett, Weng- Wong, and Simone Stumpf. Principles of explanatory debugging to personalize interactive machine learning. Proceedings of the 20th International Conference on Intelligent User Interfaces, 2015.
[19] Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. CoRR, abs/1604.00289, 2016.
[20] Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, KDD '16, page 1675-1684, New York. NY, USA, 2016. Association for Computing Machinery.

---

[21] Kin Man Lee. Arjun Krishna, Zulfiqar Haider Zaidi, Rohan R. Paleja, Letian Chen, Erin Hedlund- Medlund-Bottt, Mariah L. Schrum, and Matthew Craig Gombolay. The effect of robot skill level and communication in rapid, proximate human-robot collaboration. Proceedings of the 2023 ACM/IEEF, International Conference on Human-Robot Interaction, 2023.
[22] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing. 461:370-403,
2021.
[23] Pantelis Linardatos, Vasilis Papastefanopoulos, and Kotsiantis. Explainable review of machine learning interpretability methods. Entropy, 23, 2021.
[24] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57 2018.
[25] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[26] Maja J. Mataric. Robots for the people, by the people: Personalizing machine interaction. Sci. Robotics, 3(21), 2018.
[27] John E Mathicu, Tonia Heffner, Gerald F Goodwin, Eduardo Salas, and Janis Cannon- Bowers. The influence of shared mental models team process and performance. Journal of applied psychology, 85(2):273, 2000.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[29] Dave Muoio. Diligent robotics collects S3m seed funding, launches autonomous robot assistants for hospitals, Oct 2019.
[30] Manisha Natarajan, Esmaeil Seraj, Batuhan Altundas, Rohan Paleja, Sean Ye, Letian Chen, Reed Jensen, Kimberlee Chestnut Chang, and Matthew Gombolay. Human-robot teaming: grand challenges. Current Robotics Reports, 4(3):81-100, 2023.
[31] Rohan Paleja, Muyleng Ghuy, Nadun Ranawaka Arachchige, Reed Jensen, and Matthew Gombolay. The utility of explainable ai in ad hoc human-machine teaming. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 610-623. Curran Associates, Inc., 2021.
[32] Rohan R. Paleja, Yaru Niu, Andrew Silva, Chace Ritchie, Sugju Choi, and Matthew Craig Gombolay. Learning interpretable, high-performing policies for autonomous driving. Robotics: Science and Systems XVIII, 2022.
[33] Chris Paxton, Andrew T Hundt, Felix Jonathan, Kelleher Guerin, and Gregory Hager. Costar: Instructing collaborative robots with behavior trees and vision. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 5,,4577,,001
[34] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1 1:206-215, 2018.
[35] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. ArXiv, abs/2103.1 1251, 2021.
[36] E. Salas, N. Cooke, and M. Rosen. On teams, teamwork, and team performance: Discoveries and developments. Human Factors: The Journal of Human Factors and Ergonomic Society, 50:540 547, 2008
[37] E. Salas, Dickinson, Sharolyn A. Converse, and S. Tannenbaum, Toward an understanding of team performance and training. 1992.
12

---

[38] Bidipta Sarkar, Aditi Talati, Andy Shih, and Sadigh Dorsa. Pantheonrl: A marl library for dynamic training interactions. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (Demo Track), 2022.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. ArXiv, abs/1 0es/1707.0634,, 2017.
[40] DJ Strouse, Kevin R. McKee, Matt M. Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. pages 14502-14515, 2021.
[41] Alberto Suárez and James F Lutsko. Globally optimal fuzzy decision trees for classification and regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21112)1199911,1 1999.
[42] Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, and Matthew Craig Gombolay. Speci- fying and interpreting reinforcement learning policies through simulatable machine learning. 2021.
[43] Bruce W. Tuckman. Developmental sequence in small groups. Psychological bulletin, 63:384- 99, 1965.
[44] Silvia Tulli, Stylianos Loukas Vasileiou, and Sarath Sreedharan. Human-modeling in sequential decision-making: An analysis through the lens of human-aware ai. ArXiv, abs/2405.07773, 2024.
[45] Chenxu Wang, Zilong Chen, and Huaping Liu. On the utility of external agent intention predictor for human-ai coordination. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS '24, page 2546-2548, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems.
[46] Julia L. Wright, Shan G. Lakhmani, and Jessie Y. C. Chen. Bidirectional communications in human-agent teaming: The effects of communication style and feedback. International Journal of Climan-Compute Interaction, 38:1972 1985, 2022.
A Appendix
In the Appendix, we provide further information regarding our testbed for ration, Overcooked-A1 (Section B). additional model and training details for our interpretableMI. architecture, the Interpretable Discrete Control Tree (Section [.. complete information regarding our statistical analysis (Section EE, further discussion regarding our paper's results, limitations, and future work (Section D. and finally, a working definition of what we mean by "interpretable".
Collabo-
B Overcooked-AI
Overcooked-A1 S testbed to evaluate human-Al interaction and has been used across numerous prior work studying human-Al collaboration 40 [0]. Here, two agents are tasked with creating and delivering as many soups as possible within given time. Achieving a high score requires agents to navigate kitchen and repeatedly complete set of sequential high-level actions, including collecting ingredients, placing ingredients in pots, cooking ingredients into soup, collecting dish, getting the soup, and delivering it. Both players receive the same score increase upon delivering the soup. We modify the original Overcooked-Al game to be a simultaneous move game as opposed to the original formulation of allowing agents to perform actions asynchronously. This modification prevents the collaborative score metric from being dominated by super-human AI speed, causing the overall score to be more reliant upon effective collaboration and strategy.
We utilize two map configurations we term, Forced Coordination and Optional Collaboration, displayed in Figure 4 of the main paper. Each domain was chosen so that collaborating with the teammate would result higher score than working individually. In our newly-created domain, Optional Collaboration, creating mixed-ingredient dishes (combining onions and tomatoes) will receive higher score than single-ingredient dishes. Teammates have 200 timesteps to collaborate and cook as many dishes possible.
13

---

State-Space, Action-Space, and Reward Scheme Policies reason over semantically meaningful 13-dimensional feature space as opposed pixel space, detailing the objects each agent is holding, pot statuses, and counter objects. Each of these features is binary. For the action space, instead of using cardinal actions, we allow the Al to utilize macro-actions that can accomplish high-level objectives such as ingredient collection, ingredient placement, and soup serving. Macro-actions are planned using an A planner, and we perform dynamic replanning at each timestep. Prior work has shown macro-actions can enhance interpretability [4]. This state and action space allow forlearning an interpretable tree-based policy that can be understood and manipulated by end-users.
In Forced Coordination, for the reward scheme, we follow a similar distribution as prior work and give reward score of 60 per dish served, for an item placed into a pot. 3 for a useful dish pickup. and 5 for soup pickup. In Optional Collaboration, for the reward scheme, we give reward score of 50 for a mixed-ingredient dish, 30 for a single ingredient dish, 3 for an item placed into a pot, 3 for a useful dish pickup, and 5 for a soup pickup.
C Additional IDCT Model Details
Here, we provide additional model details for the proposed Interpretable Diserete Control Tree (IDCT).
C.1 Architecture
Our IDCT's are based on differentiable decision trees (DDTs) 41 a neural network architecture that takes the topology of a decision tree (DT). DDTs contain decision nodes and leaf nodes; however, each decision node within the DDT utilizes sigmoid activation function (i.e., "soft" decision) instead of a Boolean decision (i.e., a "hard" decision). Each decision node, i, is represented by sigmoid function, displayed yı I+exp(-o(d?) X-b,])' where and b; represents the weight and
bias terms of the decision node, respectively. As this representation is difficult to interpret, [32] presented differentiable crispification, consisting of two components: 1) Decision node crispification, which recasts each decision node to split upon single dimension of our input feature, and 2) Decision outcome crispification, which translates the outcome of decision node that the outcome is trick
Boolean decision rather than a set of probabilities. Both operations utilize the straight-throup 3 to maintain gradients, allowing for both interpretable forward propagation through the model that traces down a single branch of a tree as well as gradient flow to update parameters of the neural tree model. We utilize this approach in our IDCT's to maintain interpretability.
We initialize our IDCT's to be symmetric complete decision trees with Nj decision leaves and N; decision nodes. Each decision leaf is represented by a sparse categorical probability distribution over actions. At each timestep, a state variable is propagated through each decision node, split on a single decision rule, with the output being Boolean causing the decision to proceed via the left or right branch until arrival at a leaf node. At each leaf node, we sample from the respective probability distribution to produce macro- acctinn (e.g., "get onion" or "place held ingredient counter").
C.2 Training
For training this model, we utilize agent-agent collaborative training where interpretable tree-based agent (maintaining an IDCT) is paired with a second policy (representing the human player), and both models are trained via decentralized PPO [39]. is important to note that each agent maintains its own buffer and optimizers. Further, we improve model predictability by applying an LI norm loss over leaf node distributions for the IDCT agent to ensure sparsity, penalizing high entropy action distributions at a leaf. Our training procedure mimics that of PPO, utilizing modified loss function displayed in Equation ! and policy update in Equation 2 where 9 represents the aggregate set of weights for the IDCT, A, represents the advantage estimate at time I, and aj represents the distribution maintained at each leaf, l.
14

---

I(0) = E, min (r:(0)A,, (r.(0), A.))
(I)
Alad
oc+1 arg max T((0)
C.3
Contextual Pruning
As focus on creating agents that cooperate with humans, we must limit the size of our interpretable tree-based models to certain depth to promote user understanding. This follows prior work, finding
trees of arbitrarily large depths can be difficult understand [13] and simulate [24), and that
sufficiently sparse DT is desirable and considered interpretable [20]. However, this can make training
difficult, as small tree may not have the representational power learn high-performing policy.
Algorithm 1 Contextual Pruning Algorithm
Input: IDCT I()
Output: Pruned IDCT
1: SET NODE DOMAINS(IDC=-L, min Value=0 max Value=1)
2: queue = [[.root]
3: while queue is not empty do
4: currentNode quuuuepppp)
cuurentNode.compaare Value currennNode.loweeBBuud then currentNodee prunable True 7: end if
8: if currentNode. wwwpoode...eeeeaaaeeeeeeeeee curemnNadeeuppeerBuudd then currentNode, prunable True
9:
end if
11: UPDATE DOMAINS FOR CHILDRENNCurenNNood:
lowerBound,
upperBound,
currentN.
Val.
12: ADD_ OOD_CHILDREN_TT LEEIRRENTTT_UuUES Tow.TTTTTQUUEEeeeenntieeeeee queuc)
13: end while
< PRUNE_NODES NeDDE_ODDSs_eS [RRM__RREOOO
15:
return
In Algorithm D. we present details how contextual pruning is accomplished. In Step 1, we initialize domain vector representing the current minimum and maximum values for each feature. Since our Overcooked domain utilizes binary features, all bounds are initialized to 0 and 1. Formally, this can be written by the Cartesian product dimensionality of the state space). In Step 2. we initialize a queue that will be used to perform a breadth-first search visit each node in hierarchical order. In Step 4, receive node from the queue. In Step 5, we check the threshold value of the current node and compare it to the current node's 's vector of minimum values. This operation looks to see if the node results in tree sub-space that is out of bounds (i.c., impossible to reach). We perform a similar computation in step 8, checking the maximum values. both cases, we look find child nodes that do yield reduction in the hyperspace as candidates for pruning. In Step 11, we update the children based on the threshold value of our current node and sign (as we can have < or within node), creating new bounding box. In step 12, we add the children of the current node to the queue, and loop back to Step 4, repeating steps 5-12 until the queue is empty. Step 14, we prune tree sub-spaces that are impossible to reach.
[0. x ... [0, 1]. of cardinality d (where is the
C.3.1 Computational Analysis
The computational complexity of our contextual pruning algorithm can be analyzed in terms of both time and space complexity. terms of time complexity, is equivalent to that of Breadth-Finn Search (BFS), specifically, O(V + E),, where V denotes the number of vertices and E represents the number of edges the tree. Regarding space complexity, our algorithm exhibits similar characteristies to BFS for trees with only two leaves. In such cases, the space complexity of BFS is OVV), as it stores
15

---

all the vertices at the maximum breadth level in the queue during the traversal. Consequently, the space complexity of our contextual pruning algorithm is also O(V), making it efficient and scalable for trees with a limited number of leaves.
Utilizing contextual pruning alongside training framework allows the benefit of training large tree-based models, greatly improving case-of-traiming while still being able to simplify the resultant model a smaller, equivalent representation.
C.3.2 Results of Pruning
To evaluate the utility of pruning, we train models of various sizes (8-leaf, 16-leaf, 32-leaf, 64- leaf, 128-leaf, 256-leaf) Forced Coordination and perform pruning on the resultant model. We find that models of larger size converge to higher performance (i.e., easier-to-train) follow- ing prior work displaying the utility of larger models. Further, empirically, we find we can re- duce model sizes by 64-128x in tree depth. We provide pipeline to allow for model training and contextual pruning in our GitHub repository httpt::/ //ta:////thb...../// com/CORE com/CORR-Roooticcs-sseoooo Robotics-Latt wwth- Transparent-Policies
C.4 Hyperparameters
In IV2-D1: Forced Coordination and IV2-D2, we train an IDCT with 256 leaves, a learning rate of le 1e-3, and regularization parameter of 1e-4. This hyperparameters were chosen through trial and error, where we find larger models with small learning rate and regularization exhibited greater learning early on. The rest of the parameters follow default parameters from the PantheonRL, codebase 38 for training Overcooked agents. After contextual pruning, both domains, we end up with an Al policy with two and three leaves in Forced Coordination and Optional Collaboration, respectively.
For training fictitious co-play agents, we train 32 models of teammates each domain, saving policies at every 100 epochs. At the end of training, we sort the performance of saved policies and utilize the initial, mid- peifoperfor,ing. and highest to create our population of diverse agents, totaling 96 agents, A neural network model is then paired in multi-task training framework to team with this agent.
Our models are all trained on a local desktop computer containing a Nvidia RTX 2080 GPU and 16 GB of CPU memory. Training time for each agent took approximately 12 hours across single core. We provide further instructions to replicate our models within the above codebase.
We include high-level diagram of how IDCT agents are generated in Figure
Creating Collaborative Agent Models
Model Training in Each Domain
IDCT Agent
IDCT Agent
Contextual Pruning
3-leaf IDCT (Figure 7|
2-leaf (Figure 8)
Pre- Ex-errmerie
Figure 6: Tree Policy Generation for Conditions IV1-C1-C4
16

---

Decision Node
DBB Sonee Countr
Action Node
Dispenser
Tree
False
Fille
Figure 7: Trained Interpretable Discrete Control Tree in the Forced Coordination Domain.
Decision Node
Action Node
Tru
Talse
Key
Dounter Dispenser
Figure 8: Trained Interpretable Discrete Control Tree in the Optional Collaboration Domain.
C.5
Visualization of IDCT Policies in Each Domain
Here, we present visualizations of trained IDCT models in each domain. As seen Figures and 8 the resultant policies have and three leaves for the Forced Coordination and Optional Collaboration domains, respectively. Note that these images are pulled from our interface and thus have extra annotations improve readability.
D Additional User Study Information
Our experiment was reviewed and approved by the Institutional Review Board at the Georgia Institute of Technology under Protocol Number H23043, All participants in our experiment signed a consent form, received a description of the risks involved in our study, and received compensation for participating. Below, describe specifics regarding the consent procedure, additional details that describe the experiment procedure, and the compensation scheme.
D.1 Consent Procedure
At the start of the experiment, the participant is provided consent document. This document de- scribes the purpose the experiment, exclusion/inclusion criteria, the experiment procedure, the risks of the experiment, the compensation scheme, and details regarding data storage and confidentiality.
D.2 Additional Information Regarding Specific Conditions
IV1-CI: Human-Led Policy Modification is enabled through the contribution of the interpretable machine learning architecture to train collaborative AI teammates, training advancement to enhance
17

---

interpretability, and a mechanism to allow humans to modify the tree in simple ways, including tree deepening. decision variable modification, and leaf node modification. The following conditions: IV1-C2: AI-Led Policy Modification, IV1-C3: Static Policy Interpretability. and IV1-C4: Static Policy- Black-Box all utilize the same architecture and starting policy but ablate different components of the interaction and interpretability.
After teaming episode in the IV1-C2: AI-Led Policy Modification condition, the Al utilizes recent gameplay to fine-tune a human gameplay model via Behavioral Cloning and performs reinforcement learning for five minutes to optimize its own policy to better support the human teammate. In this collaborative agent policy optimization stage, we utilize the parameters described in Section C.4 and add timer to stop the optimization. Upon completion of policy optimization, we check if the policy has improved through simulated interactions with the behavior cloning agent, and if so, update the policy. In the case that the policy degrades, we use the original policy prior to optimization. The user can visualize the updated Al policy in its interpretable tree form prior to the next teaming interaction.
IV1-C3: Static Policy- Interpretability and IVI-C4: Static Policy- Black-Box are static poli-
cies that do not change across repeated gameplay. Thus, we do not have any specific additional hyperparameters to discuss within the appendix.
To improve the transparency of the conditions in our experiment, we provide a flow diagram that displays the interaction being assumed within each condition in Figure
D.3 Compensation Scheme
Participants were compensated at a rate of 20 US dollars per hour of the experiment.
E
Complete Statistical Analysis
Here, we present complete details regarding our analysis, including all test statistics well as nonsignificant and trending comparisons.
E.1 RQ1: Team Coordination Performance
As mentioned in the main paper, we allow humans team with the Al across four episodes, providing us with four teaming scores. Within the main paper, we reported differences with respect to the maximum score participants were able to obtain across iterations. Here, we analyze data in the performance round (the last iteration), where participants were told to maximize performance. We note that participants self-reported their gaming familiarity (100-point scale) and weekly hours playing video games. Across all participants, self-reported gaming familiarity and weekly gaming hours was 4.44 = 5.32. This information was used in our statistical analysis, not found in performance variation a function of gaming expertise. Utilizing
rated 73.19 + 23.80
and significance a Friedman's 's test, we find that there is a significant difference across domains (x²(1) = 38.7. < 0.001). Accordingly, we analyze the two domains separately.
In IV2-D1, we find our data does not meet the necessary assumptions and utilize non-parametric tests. A Kruskal-Wallis Test was conducted to analyze differences in performance round reward across conditions, and we find a significant effect (x²(4) 20.85, p < 0.001) across conditions. We conduct post-hoc pairwise comparisons, utilizing Dunn's 's test, and find that IV1-CS is significantly better than IV1-C1 (p < 0.01), IV1-C3 (p < 0.01), and IV1-C4 (p < 0.01). IV1-C5 is trending as significantly better than IV1-C2 with a p-value of 0.0275 (significance is < 0.025 or (a/2) due to the Dejamini-Hochbeee adjustment).
In IV2-D2, we test for normality and homoschedascity and do not reject the null hypothesis in either case, using Shapiro-Wii Wilk (p > .50) and Levene's Test (p > 0.05). An ANOVA was conducted to analyze differences in performance round reward across conditions, taking several observed variables into account. We find a significant effect (4,338) = 18.93; p < 0.001) across conditions and decision tree familiarity (F(1,38) utilizing Tukey HSD, and find that 1) IV1-C5 is significantly better than IV1-C2 (p < 0.01), IV1-C3
16.12;p < 0.05). We conduct post-hoc pairwise comparisons,
(p < 0.01), and IV1-C4 (p < 0.01), and 2) IV1-C1 is significantly better than IV1-C3.
18

---

Introduction
Domain (Randomly Ordered)
Second Domain (Randomily Ordered)
End-of-Experiing Survan Surveys
Demographicc Survey
Human plays with
Humen plays with tearmmatt
Colleboratwv Apsessment
Expeilence Video
Modify Al'S bohovlor
Modily Al's bichevior Via GUT
Questionnaire
Tutorial on Came
Contrals Task
Visuolizo the Al's Polley Tree
Vauualic the Al's Polley Tree
Inclusion of other the Sist!
NASA TLX Worklord Survey
NASAA-LLX Workload
Survey
System Liaabilitt Survey
Repreto Urmes and concledis final teaming spicade
(a) Experiment Flow for IVI-CI: Human- Led Policy Modification
Introduction
First Domain
Second Domain
End-of-Exxerrimen Surveys
(Randomly Ordered)
(Randamly Orderod)
Humaan with Al
Humon
Human -lobot Collbbeuiin Faxeayy
Experience with Vidco
Assescment
Gomes and Dediiion
gamoplay modil with opicede doto
Fino. tunc human gomeplay
Trees Survey
model with recent epicode data
Lostspeer
Cptmmua Alts poticy with synthetic human model
Optimize Al's policy with synthetic human modkl
Tatoriat Game Controls
Inclusion of other Seltt
Visuolize the Al'S Polcc Trec
Visuslize the ATS Policy Thee
NASA- TLX Worklood Survey
NASA TLX Workooo
System Usabiiity Survey
Repeats times and can cludes with a final twannin eqiende
Repeate times conetudis witir final bearning sp/sodie
(b) Experiment Flow for IV1-C2: AI-Led Policy Modification
Introduction
First Domein (Randomly Ordered)
Second Domain (Randomly Ordered:
End-of- End-of-Exxprineee of-Experiment Surveys
Domographice
Human Ro
playss with teammate
Human plays
Collaboratwe Fluency Assesement
Experlence th Viceo
Gamae and Decition
Trees Survey
Godspeed Quectionnaiz
Visualito the Al's
Visuelize Tree
Tutoriat on Garne Contrate and Tack
Instusson of olther the Self
NASA, TLX Workload Survey
NASA-TLX Worklood
Survey
System Utabiiitty Survey
Reprots timnee and cancludee with a traal beaming spacdd
Repeats and concludee boaming uposde
(c) Experiment Flow for IV1-C3: Static Policy Interpretahility
Introdluction
Domain (Randomly Orderedi
End-of-Ex Todd-ofEExeermeent Surveys
(Randomly Ordered)
Demooraphics Survey
Hulmann-obbot Collaborative
Human plays with
Hummn plays teammate
Experience with Viéco
teammmate
Deditian
Trees Survey
Godllpend Questionnairc
Tutorial on Come Controtx Tank
Inclusion other in the Set?
NASAA-TXX Workdadd Survey
Survey
System Usaaiiitty Survey
Rapests and concludas
with findl aprinde
(d) Experiment Flow for IV1-C4: Static Policy Black-Box and IV1-C5: Fictitious Co-Play
Figure 9; This figure displays experiment flow diagram for each condition.
19

---

These results are similar to those in the paper when analyzing the maximum reward and result in a similar set of conclusions: 1) black-box models can outperform white-box approaches, and 2) white-box approaches with policy modification have some benefit over white-box approaches alone. Further, as we see that tree familiarity positively correlates with performance round rewards, exploring alternative paradigms, such as natural language for describing and programming trees may benefit users unfamiliar with decision trees.
E.2
Team Development
Here, we analyze the trends across iterations (did agents improve from iteration one to four) and identify characteristics of users that performed well team development. Utilizing a Friedman's 's test, we find that there is significant difference across domains (x²(1)==0448, p < 0.001).
We conduct separate Wilcoxin signed-rank tests for each condition, and utilize the Bonferroni correction in determining significance (a/5). In IV2-D1, we no condition significantly improves significantly over repeated iterations. In IV2-D2, we find that IV1-C1 (p < 0.01) and IV1-C2 (p < 0.01) significantly improve over repeated teaming interactions.
F
Discussion, Limitations, Future Work, and Societal Impacts
Discussion: In this paper, we provide several contributions towards interactive HMT. We first present weaknesses in prior work, displaying that learned collaborative agents can be individualistic and rigid. To address these weaknesses, we propose an interactive scheme termed human-led led policy modification to bridge the gap between individualized coordination and adaptive, effective collaboration. We do so by creating feedback loop that facilitates team policy changes during HMT. This is accomplished by 1) utilizing an interpretable policy representation to provide users with insight into the teammate's ppcision-making specifically the IDCT, interpretable tree-based model that can be trained via reinforcement learning and pruned to a smaller, equivalent representation, and 2) creating a user interface to support the end-user modifying the policy their evolving specifications. We deploy and compare our interactive policy modification scheme to several other techniques, including two popular prior works and variations of our proposed condition. While we do not direct objective benefit of human-led policy modification compared to utilizing a black-box model supported with a population based training scheme [40), we find important takeaways that motivate the importance of conducting longer-term, facilitate interpretation can be used within feedback loop to lead policy improvement, users may require larger number of interactions to reach a team consensus and maximal performance, and there are person-specific characteristics that may lead to some users being able to take advantage of interpretable models and interaction more than others.
studies. Specifically, white-box approaches that
This study was conducted university. While the population was diverse in age. gender, and university major, all students had some college education and most students were based in engineering, presenting a population bias. Furthermore, the population represented by the age group of 18 to 32 years old (mean of 24.14, std of 4. within our experiment may not directly generalize to an older population with extensive training. Furthermore, the experiment findings may not generalize to all contexts and scenarios within HMT. We reiterate that our findings are within a two-agent human- mamanne team within a relatively di-eeniiiionl and short-horizon game, Overcooked-Ar AI. In scaling to more complex and dynamic environments, the tree size needed to represent high-performing agent will likely increase. these cases, users may require more time to interact with and understand an agent's policy. There may be several capabilities that can be added to the Human Policy Modification interaction paradigm, which may make the process quicker and casier. For example, model verification or forward simulation can be used to provide the human with other types of feedback prior to the next teaming iteration. Furthermore, for increasingly complex games, agent policies can also operate over different levels of abstraction, providing the human with a tradeoff with Ene-grained control of the agent policy and tree size. Finally, different policy visualizations may better support certain populations of users, emphasizing the need for collecting user background information and future research in interpretability for embodied agents.
Limitations:
Future Work: In the future, it would be interesting to conduct similar experiment to a higher number of iterations, or until the team converges to a set coordination strategies (the "performing" stage in Tuckman's 's model). Further, the possibility of adding in feedback from the AI regarding human-led
20

---

policy modification (checking for logic inconsistencies, etc.) may be used to facilitate speedier team development. It would also be interesting to utilize different paradigms in communicating with the human as language may be an easier medium than a decision tree interface. Future work should also be done to optimize the accessibility of GUTs for policy modification via xAI techniques. Finally, expanding this research to real-world collaborative robot settings in healthcare of manufacturing that utilize tree-based policies, such as collaborative packaging [12 [7 or agile robotics [8] 21], would lead to additional insight into human-maching team development with robot teammates.
This work investigates repeated interactions with inter- pretable machine- muchine-learning- boaning-basedd agents in collaborative game. As autonomous agents (e.g.. robots) are deployed in the real world, insights from this work may be applied to assist in creating fruitful working relationship between human and an agent. We do not believe this work has any negative societal impacts.
Positive and Negative Societal Impact:
G Working Definition of Interpretability
As mentioned in the main paper, agent representation is that of an Interpretable Discrete Control Tree, which reasons over a state space with high-level binary features and multi-step macro-actions This model (which, in layman's 's terms, is decision tree with action probabilities at each node) is the true learned model produced via reinforcement learning, not an abstraction created post hoc. This model is interpretable its representation is "constrained in model form that it is either useful to someone, or obeys structural knowledge of the domain, such as monotonicity, causality, structural (generative) constraints, additivity, or physical constraints that come from domain knowledge" (35). In our case, the model constraints are inherent within the novel IDCT architecture, and the utility of this model to user is that this model is able provide users with some awareness over the agent's 's behavior (and possibly, simulate the agent's 's decision making) and 2) provides users with the ability explicitly modify agent behavior (a capability not possible with black-box models).
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", it perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" " or "we were unable to find the license for the dataset we used"). ) In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately represent the contributions of this work and supported directly by our results. The case study and its implications are described in Section 3 the interpretableML. architecture and modification scheme is described in Section and the user study and its findings are discussed in Section
5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
The limitations are clearly described within the Section within the Appendix.
Guidelines:
22

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texs system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The necessary details to reproduce teaming agents is described within the text (general model details in Section and specific hyperparameters Section C.4 and the experimental procedure to evaluate these agents with real humans is in Section 5 Furthermore, our GitHub repository contains specific scripts and instructions to reproduce models.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide our code in the following anonymous https: / thub. com/ CORE--oooRRtoecccccooooooooooocc cs C Team Denen-Dvellogg wwweent-pptttt..mmo wi with- Transparent- formation on how to set up the environment, train agents, and evaluate agents online.
Justification:
GitHub repository:
This repository contains in-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/tips..cc/ublic/guicee geiiiisggguie//eeeeeeeeo.eeeesee (oodeSumiiss ssioniyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our high-level training details are provided within the text and we report agent training accuracy (performance of the model with a synthetic human teammate) and agent testing accuracy with real humans via our human-subjects study. For lower-level training details, please look the Appendix in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we provide exact information with regard to the statistical tests used to analyze our experiment data in Section 5 Further context, which ensures that the assumptions for these tests are met, is provided in the Appendix Section E
Error bars are
displayed in Figures 4 and 5 and represent the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
25

---

Answer: [Yes]
Justification: We provide compute information within Section C.4 including the type of computer and its resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neurri: httpppss//norrsp..ccppbbcllllee http:/pllic/chccccuu.eeeee
Answer: [Yes]
Justification: We confirm that our paper conforms to the NeurIPS Code of Ethics. The conducted subjects study was reviewed by a University Internal Review Board to comply with ethical practices.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, our paper discusses potential positive and negative societal impact within
the Appendix Section F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
26

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited the authors of PantheonRL, which was the backbone code- based for collaborative agent training
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, snneeswwthcode com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets.
27

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, sereenshots of our experiment are provided in the attached codebase as well complete code to run our experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We are aware of our use of human-subjects and conducted experiment with caution. Our experiment was reviewed and approved by the Institutional Review Board at the Georgia Institute of Technology under Protocol Number H23043. Furthermore, all participants signed a consent form, received a description the risks involved in our study, and received compensation for participating.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "a9akWzo6NY", "reviewer": "Reviewer_yVVF", "summary": "This paper focuses on developing strategies to enhance transparency and interpretability in human-AI teaming settings. Based on my understanding, two collaboration contexts have been considered: human-preferred collaboration and AI-preferred suboptimal teaming strategy. The authors have implemented specific strategies to tackle these contexts and address interpretability. The implemented strategies have been evaluated on the Overcooked game with real human users.", "questions": "Three main questions are:\n\n- How do you envision improving the transparency of the proposed methodologies? How can Section 4 be better linked with Section 5?\n- What are the evaluation metrics for the chosen collaboration settings? How is interpretability evaluated?\n- Are there any other collaboration environments that can be considered for this line of research? How do you envision these findings being generalized to other collaboration tasks?", "limitations": "The authors discussed the limitations and the broader impact of their work adequately in the appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well-motivated, particularly in its approach to learning human preferences and using reinforcement learning (RL) to train the agent.\n\nThe paper provides valuable insights into designing human-AI collaboration interfaces to enhance user trust and acceptance, addressing a timely and relatively unexplored domain.", "weaknesses": "The main weakness of the paper is that it is very hard to follow, and the methodology lacks transparency. For instance, the paper does not present a single method to investigate explainability. Overall, the chosen modifications and the designed collaboration setups are not clearly presented. It is not straightforward to link Section 4 (Methodology) with Section 5 (Studies).\n\nSimilarly, the experimental results are very hard to follow. The numbers are presented but are not straightforward to interpret or use for cross-checking the claims. In particular, the evaluation metrics are not clear.\n\nAdditionally, only one collaboration environment has been considered. The findings might be biased toward this specific collaboration task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting that our paper is well-motivated, provides valuable insight into designing human-AI collaboration interfaces, and studies a relatively underexplored domain. We have responded to the weaknesses and questions noted by the reviewer.\n\n**Comparison to Explainability Approaches** --  The reviewer is correct in that we do not compare against explainability approaches that utilize local explanations to explain the decision-making of autonomous agents. While these approaches should be tested to explain behavior in the tight human-AI collaboration settings we consider, local explanations can be misleading and may not accurately represent the agent's decision-making behavior. This would lead to another dimension that needs validation. As this area of research (transparency and adaptability of collaborative agents in repeated human-AI collaboration) is relatively new, we hope our research can spawn further studies of this kind.\n\n**How can Section 4 be better linked with Section 5?** -- Section 4 presents an interpretable machine learning architecture to train collaborative AI teammates, Interpretable Discrete Control Trees (4.1), a training advancement to enhance interpretability, and a mechanism to allow humans to modify the tree in simple ways, including tree deepening, decision variable modification, and leaf node modification.  The creation of this capability (interpretable tree architecture + human modification) is our main proposed condition, IV1-C1: Human-Led Policy Modification in Section 5.\n\nThe following conditions IV1-C2: AI-Led Policy Modification, IV1-C3: Static Policy - Interpretability, IV1-C4: Static Policy - Black-Box all utilize the architecture introduced in Section 4 but ablate the interaction and interpretability. We tried to depict this gradual feature reduction across our proposed conditions in Table 1. To improve the linkage between Section 4 and Section 5, we propose to create a new section between 4 and 5 that presents information regarding the training and results of the IDCT model. In creating this section to improve clarity, we would augment details from the author response alongside the paper lines 301-321.\n\n**How do you envision improving the transparency of the proposed methodologies?  --** To improve the transparency of the proposed methodologies, we have created a flow diagram (uploaded as Figure 1 in the attached rebuttal pdf) of each condition that helps display visually the experiment flow and interaction being assumed within the proposed approach. We have also included a diagram of how IDCT agents are generated as part of Figure 2 (left). Both of these figures will be added into the main paper as part of the appendix and alongside the new aforementioned section will improve the paper's clarity.\n\n**What are the evaluation metrics for the chosen collaboration settings?** -- The main objective evaluation metric used to evaluate the performance of the collaboration is the game score. Section B of the appendix describes the exact scoring function that makes up the game score. In short, high score bonuses are obtained for full dishes served, and minor score bonuses are given for smaller objectives like filling a pot or picking up a dish. In the first domain, Forced Coordination, successful dish serving is not possible without resources being handed between the AI and human. In the second domain: Optional Collaboration, agents can serve dishes without explicitly collaborating with the human, but this domain was intentionally designed such that collaboration with the human via timely resource handoffs would result in a higher score than without collaboration.\n\n**How is interpretability evaluated?** -- The interpretability of the model isn't evaluated explicitly. In our hyperparameter search for training the tree-based agent policy, we choose the highest-performing model with under 32 leaves. Prior work [1] notes that there is a cognitive limit on how complex a model can be while still being human-understandable and thus, we prioritized selecting a high-performing IDCT model that was still relatively small. We note that as conditions IV:C1-C4 utilize the same model, utilizing interpretability metrics such as tree size do not provide additional information in understanding the tradeoff between these conditions.\n\n[1] Lakkaraju et al. Interpretable decision sets: A joint framework for description and prediction. \n\n**Are there any other collaboration environments that can be considered for this line of research? How do you envision these findings being generalized to other collaboration tasks?** -- We believe research in repeated teaming with collaborative, transparent agents should be studied in more complex domains such as Minecraft, Dota 2, and Starcraft.  Our work utilizes the relatively low-dimensional Overcooked setting where techniques like online optimization and tree manipulation by a human end-user can be done within a short time period. This is vital in allowing for a feasible single-session user study to be conducted. \n\nHowever, many of our takeaways generalize beyond our single setting of Overcooked to the field of human-AI collaboration and also have applicability to real-world applications in collaborative robotics. In many settings, teams of agents and humans will need to go through stages of team development. Our work shows that model transparency and the ability to interact with policies help in this regard. While there may be domains where white-box models are already competitive with the initial performance of black-box models, there is still research needed to improve interfaces so that users can interact successfully with agent models. Within our domain, we found through a usability survey that there was a large disparity between users finding the interface good (>75) and bad (<35). This finding also generalizes to many domains that have human collaborators coming from different backgrounds and expertise levels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsGB4nlkR1", "reviewer": "Reviewer_HJn7", "summary": "In this manuscript, it focuses on the collaboration in human-machine teaming (HMT) based on interactive and explainable systems. In order to address the existing issues, such as decoupling, the author(s) explored an interesting paradigm in HMT and proposed some guidelines based on the study. Also, the author(s) pointed out some potential research directions, although some of the conclusions are intuitive, but interesting.", "questions": "I read the manuscript, and I have the following questions/comments. Thanks.\n\n1. We need to balance sparsity and stability in designing a learning algorithm. In this manuscript, the author(s) adopted L1 regularization; I am not so sure if it will cause some issues due to the instability of l1.\n\n2. Regarding how users interact with AI across repeated play under different factors, the author(s) performed some human-subject-involved studies. About the population used in the study, can you discuss the implications of the population bias in more detail and how it might affect the generalizability of your findings?\nAlso, maybe sex/age-based difference analysis is also worth exploring.\n\n3. If possible, could you explain more about the potential challenges in scaling the human-led policy modification approach to more complex and dynamic environments beyond the Overcooked-AI domain?\n\n4. Regarding the citations, I do not think the current references are sufficient, at least, for a manuscript preparing to submit to computer conferences such as NeurIPS, since the newest citation was two years ago, and there are only 4 such 2022 citations.\n\n5. In Line 58, maybe \"reinforcement learning\" should be with its abbreviation, not until Line 70.\n6. It would be great to be with the full name for PPO, such as Proximal ... (PPO)\n\nSome other format issues in references:\n\n(1) The author's name, sometimes using the abbreviation, sometimes not, such as Ref.[1], Ref.[33].\n\n(2) Sometimes, the conference name has both the full name and its abbreviation, and sometimes not, such as, Ref.[4] vs Ref.[5].\n\n(3) In Ref.[17], \"ai\"=>\"AI\".\nPlease check carefully; it would be great if the author(s) could correct these issues.\n\n\nI would like to consider adjusting my current score based on the responses from the author(s). Thanks.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my opinion, the strengths of this manuscript are as follows:\n\n1. Demonstrating that current HMT approaches struggle to adapt to human-preferred strategies, often resulting in suboptimal team performance.\n\n2. The proposed architecture enables end-users to modify AI behavior, facilitating a feedback loop for team development.", "weaknesses": "In my opinion, the weaknesses of this manuscript are as follows:\n\n1. As the author(s) mentioned, due to the population setting, the findings have a population bias and may not generalize to wider situations.\n\n2. The cited work is a little out of date. This makes this manuscript seem not ready for publishing at conferences such as NeurIPS.\n\n3. Some descriptions in the main text are not so clear.\n\nFor more details, please see the Section \"Questions\" below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We have responded to the weaknesses and questions noted by the reviewer.\n\n**Instability of L1** -- We thank the reviewer for this comment. The L1 regularization is only applied to the action leaf nodes of the tree policy. This regularization serves to make the categorical distributions within each leaf node more sparse so that the agent acts more deterministically. The mechanism used in our architecture to train an interpretable tree model via reinforcement learning is called differentiable crispification and was formulated by prior work (see [1]). From our observation, the L1 regularization successfully improved how deterministic the resultant tree-based policies were.\n\n[1] Paleja, R., Niu, Y., Silva, A., Ritchie, C., Choi, S., & Gombolay, M. (2022). Learning interpretable, high-performing policies for continuous control problems. arXiv preprint arXiv:2202.02352.\n\n**Population bias and how it might affect the generalizability of your findings? Also, maybe sex/age-based difference analysis is also worth exploring.** -- Our study was conducted at a university with a diverse population majoring in a variety of different engineering disciplines (Electrical and Computer, Biomedical, Aerospace, Chemical and Biomolecular), economics, and variety of sciences (computer science, neuroscience, atmospheric). All users within our experiment had some college education and were enrolled at an engineering-focused university. As such, these findings may not generalize beyond this population. In our data collection, we collected demographic information including age and sex. In our analysis, these variables were used and we did not find any significant trends that displayed that age or sex led to differences in Human-AI collaboration performance.\n\nIn generalizing our results to a broader population, we believe are findings can be augmented by other literature in human-AI interaction to design better interfaces for specific populations or domain-specific collaborative agents. \n\n**If possible, could you explain more about the potential challenges in scaling the human-led policy modification approach to more complex and dynamic environments beyond the Overcooked-AI domain?** -- Thank you for this excellent question. In scaling to more complex and dynamic environments, the tree size needed to represent a high-performing agent will likely increase. In these cases, users may require more time to interact with and understand an agent's policy. There are also several capabilities that can be added to human-led policy modification which may make the process quicker and easier. For example, model verification can be added into the tree modification interface to detect problems with logic (detect regions of the tree that cannot be reached). This would allow the human to receive other types of feedback prior to teaming with the AI. For complex games, agent policies can also operate over different levels of abstraction providing the human with a tradeoff with fine-grained control of the agent policy and tree size. There may also be other more accessible mediums, such as language, that the human can use to program a large tree policy.\n\n**Missing citations** -- We thank the reviewer for noting this deficiency. Within our literature review, we will add the following recent papers that are closely related to our work. \n\n[2] Hong, Joey, Sergey Levine, and Anca Dragan. \"Learning to influence human behavior with offline reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024). \n\n[3] Wang, Chenxu, et al. \"On the Utility of External Agent Intention Predictor for Human-AI Coordination.\" arXiv preprint arXiv:2405.02229 (2024). \n\n[4] Guan, Cong, et al. \"One by One, Continual Coordinating with Humans via Hyper-Teammate Identification.\" \n\n[5] Tulli, Silvia, Stylianos Loukas Vasileiou, and Sarath Sreedharan. \"Human-Modeling in Sequential Decision-Making: An Analysis through the Lens of Human-Aware AI.\" arXiv preprint arXiv:2405.07773 (2024).\n\n**Abbreviations and Reference Format ** -- We have updated our paper language per your comments and updated our references to be consistent.\n\n\n**Ethics Review Flag** -- As mentioned in Line 342 of our paper, our experiment was approved by a university institutional review board (IRB). All participants in our experiment signed a consent form, received description of the risks involved in our study, and received compensation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAC9jBUDMR", "reviewer": "Reviewer_cdZ4", "summary": "In this paper the authors present an approach to human-AI teaming in the common Overcooked domain via Interpretable Discrete Control Trees (IDCTs), which are differentiable decision trees which the authors visualize and make controllable. The authors present two examples of where existing blackbox models may demonstrate a gap in teaming performance. They then present results of a human subject study that demonstrates that authors' approach is outperformed by a fictitious co-play baseline.", "questions": "1. What was the size of the ICDTs used in the study?\n2. Did participants experience the two domains in a fixed or random order?", "limitations": "The major limitations come from the poor performance of the ICDTs and the decision to limit the size of the human subject study. The authors do not sufficiently address these limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Human subject studies studying human-AI teaming are still relatively rare due to their difficulty. As such, a new study is always beneficial and of interest to the community. The authors also include many variations of their approach, which is beneficial in terms of more deeply understanding the mechanisms of human-AI teaming.", "weaknesses": "I like this work, but at present this paper has a number of key weaknesses holding it back. \n\nFirst, the current text of the paper contains a large number of unsupported claims. Almost all of the text in italics represents claims that are not substantiated by an argument, a citation, or the results of the paper. There are similarly design decisions that are not motivated or explained, such as why the authors chose to use IDCTs. \n\nSecond, Section 3 introduces the possibility of a gap in teaming performance with current models. However, the evidence for this is presented as two examples. This is not sufficient to demonstrate that this specific approach has failure cases but not to demonstrate a need for the approach the authors' propose. I'd recommend working on a more full survey of existing Overcooked approaches to quantitatively evaluate the rate at which collaboration gaps may be occurring. This section represents the primary motivation for the authors' work, and so it being a weak point makes the whole paper weaker. \n\nThird, the authors do not give full technical details for any of the approaches, not the approach of Carroll et al., nor their own ICDT approach, nor their implementation of fictitious co-play. This is an issue as it makes it very difficult as a reader to understand what has been done at a technical level. For example, it's unclear to me what size the ICDTs used in the study were, I only know that the users were limited to expanding the tree to a depth of 4. But this doesn't tell me if the initial tree had a depth of 3. Similarly, the pruning approach only seems to remove redundant nodes so it's unclear how much this would actually prune (the authors state 8-16x smaller but it's unclear to which trees they refer). Given that a major problem with the ICDT's performance in the user study was their performance, these details are crucial. While there's an anonymized GitHub in the appendices, it should not be necessary to go through it to understand the work. \n\nFourth, while the human subject study seems well-designed, certain details are unclear. It's unclear what population is being drawn on for the participants, though I would guess university undergraduates given the demographics. It's unclear how knowledgeable these participants were about AI except for one mentioned figure around familiarity with trees. It's unclear why the authors stopped recruiting at 50, since 10 per condition is considered an absolute minimum. Finally, the methodology is somewhat unclear, its unclear what order the participants experienced the domains. Was it consistent, if so, why? If not, there were 10 conditions, not 5.\n\nFinally, the results do not seem to support the authors' claims. From the result, my takeaway is that participants preferred performance over interpretability. This goes against several of the authors' claims in the paper, most notably \"As seen through these findings, the ability to interact with an interpretable model is perceived significantly better across several measures\". The authors also make a claim about trending towards significance, which is not statistically sound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting that our study in Human-AI teaming would be beneficial to the community. \n\n**Unsupported Claims and Design Decisions** -- As the field of Human-AI Teaming is still relatively new, some motivations are positional. For example, in considering lines 129-132, some may argue that AI agents and humans do not need consensus or that the human should always defer to the AI's strategy. In some domains, this may be effective. However, in domains where agents and humans need to collaborate closely, we believe the process of consensus will lead to better collaboration performance and understanding between teammates. This is validated in our human-subject study findings (see Figure 4b of the manuscript), where users that can interact with the AI's policy significantly improve over repeated gameplay. There are several works that display that social robots that can adapt online and develop social relationships achieve more successful and sustained interactions with users [1]. \n\n[1] Leite et al. Empathic robots for long-term interaction: evaluating social presence, engagement and perceived support in children.\n\nThe design guidelines are directly related to the results of our paper. \n\n- Creation of white-box agents that achieve competitive initial performance to black-box agents: If model performance is equal, interpretable models provide increased accountability compared to black-box models. Importantly, these models with user interaction led to positive team development. \n- Design of learning schemes to support the generation of collaborative behaviors: This is derived from the case study and analysis of Fictitious Co-Play. Designing objective functions so that agents collaborate well with humans and/or online adaptation schemes to lead to personalized, effective teammates is an important research direction.\n- Creation of interfaces that enable mixed-ability users to improve team collaboration: This is based on system usability scores collected from users, emphasizing the need for creating modification interfaces that support a wider variety of users. \n- Evaluation of teaming in a larger number of interactions: This was derived from our user study findings, where a higher # of interactions may have provided a better understanding of the team development process. \n\nThe design decision behind the IDCT is that this tree-based model affords transparency and supports training via RL. This allows us to directly compare with prior frameworks in Overcooked that leverage RL for training collaborative agents and provides us with an interpretable model that users can modify and visualize. \n\n**Case Study regarding the Teaming Gap with Current Models** -- We agree that this case study could be expanded. This first study (Figure 1) is conducted with an actual human player following a scripted strategy while collaborating with an agent publicly available from Carroll et al. While a human could have conducted a larger number of trials with the agent, it was clear from the set of trials that the agent could not adapt to the human-preferred strategy. \n\nThe second study focuses on Ficticious Co-Play. A heuristic collaborative strategy and heuristic individual strategy are programmed, receiving scores of 408 and 306. Then, an FCP agent is trained, converging to a score of 295.06 \\pm 1.86 over 50 teaming simulations. This FCP agent is also evaluated with humans in the study, achieving scores ranging from approximately 120 to 315 in the last teaming round (Figure 4b-right). These gameplay scores of FCP, both in teaming with synthetic agents and real humans, are far below a heuristic collaborative strategy, signifying the gap. A full assessment including other benchmarks and analyzing agent collaborativeness, both quantitatively and qualitatively, while interesting, would likely be a full paper in itself.\n\n**Technical Details regarding the Approaches** -- The details regarding the IDCT approach are found in Appendix C alongside pictures of the agent models (Figure 6 and 7). In line 321, we note that the trained policies had two leaves in Forced Coordination and three leaves in Optional Collaboration. As mentioned in Appendix C.4, we train our IDCT models with 256 leaves. A reduction to a depth of one (2 leaves) or two (3 leaves) is a pruning reduction of 128x and 64x. We will update these numbers and shift these details into the main paper. \n\nDetails of the FCP baseline are also found in Appendix Section C.4. The AI-Led Policy Modification is an adaptation of Carroll et al.'s approach to an online setting with an IDCT. We provided high-level details about this approach in Line 279-284 and within the footnote on page 6. We will include further details about the online optimization procedure in the Appendix.\n\n**User Study Detail Clarifications --** Our study was conducted at a university with a diverse population majoring in different engineering disciplines, economics, and sciences. All users had some college education and were enrolled at an engineering-focused university as an undergraduate or graduate student. Users were asked demographics information, experience with games and decision trees, and conducted a personality survey. This information was included when determining significant trends.\n\nIn future studies, it would be beneficial to increase the # of participants and gameplay trials. As noted in the Procedure, the domains are randomly ordered. Our experiment is a 5 (teaming method; between) x 2 (no. of domains; within) x 4 (no. of repeated evaluations; within) mixed-factorial experiment.\n\n**Results Supporting Author Claims** - The statement noted was comparing conditions IV1-C1 to IV1-C2, IV1-C3, and IV1-C4. This statement meant that given the same tree-based model, the ability to interact adds subjective benefits. The reviewer is correct in noting that participants did assess higher-performing agents more positively in their subjective ratings. We will clarify this statement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8u9wP1Go2O", "reviewer": "Reviewer_MFRs", "summary": "The paper proposes a white-box approach to human-machine teaming, in which human teammates can see their virtual counterparts’ policies and adjust them accordingly. The framework is built on top of differentiable decision trees, and the authors propose contextual pruning as a means of simplification for training purposes and readability by the human team members. They use this framework to quantify the benefits of interpretability in HMT and the tradeoff between high interpretability and high accuracy, the latter typically better achieved by traditional black box solutions. The authors use their framework to conduct a thorough statistical analysis comparing different white and black box conditions and use their findings to list a series of guidelines for HMT.", "questions": "- In line 230, the authors state that contextual pruning significantly improves the ease of training. Are the two pruning strategies enough? I am unfamiliar with DDT, so I wonder if this is novel. How is this different than traditional pruning methods in DDTs?\n- As noted by the authors in line 357, some participants outperformed the maximum performance of IVI-C5 in teaming iterations 3 and 4. I wonder if those participants show an increasing trend in performance from iteration 1 to 4? I cannot tell that from the scatter plot. I am asking this because I am curious to know if the improvements are due to randomness or if humans are applying a conscious strategy when modifying the agents’ policies, leading to incremental improvements at every iteration.", "limitations": "The authors discuss the limitations of their work in the Appendix. I would argue there's one more limitation:\n\nIt seems performance in the mixed-initiative scenario is highly dependent on presentation, as validated by better performance improvements in participants more familiar with Trees. This work shows that interpretability is also relative. What is interpretable to me may be convoluted to another, which may also affect the team's performance. It would be interesting to check for differences in IVI-C1 with different visualization methods in future work.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- I enjoyed reading this paper, and I think the idea is novel, and the contribution of this paper to the field is significant. Interpretability is crucial in HMT if virtual agents are to be seen as real teammates instead of mere tools. Being able to understand these agents' policies and change them to agree with the human's beliefs is one step in that direction. The key point here is that the changes made by humans may not be the best ones (typically achieved by blackbox solutions), but they may be the ones that foster more collaboration between humans and machines.\n- The paper is very well-written, and the ideas are very clear. The steps for reproducibility are detailed, and the statistical analysis is comprehensive.", "weaknesses": "The authors propose a pruning strategy to simplify training and interpretability but provide no evidence supporting either. It would be helpful to validate such claims with time and memory usage assessment and IV1-C1 analysis with and without pruning. The authors briefly discuss the former in C.3.2, but it could be improved by displaying the numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting that they enjoyed reading our paper, that the paper's contribution to the field is significant, and that our paper is well-written and the ideas are clear. We have responded below to the weaknesses and questions noted by the reviewer.\n\n**Contextual Pruning Results and Novelty** -- The two post-hoc pruning strategies we utilize are to prune a tree given the boundaries of each state variable and removing redundancies within the tree. This need for pruning arises naturally during the training of the IDCT model via reinforcement learning as different subspaces of the tree model may become more important (higher probability of reaching certain sub-trees) or completely inactivated (impossible to reach a certain sub-space due to changes in weights caused by gradient descent). It is important to note that these pruning techniques follow a similar ideology behind neural network pruning but do not modify the model's behavior in any way. Neural network pruning approaches, on the other hand, often remove weights with smaller magnitudes or activations, which result in model changes that can harm performance.\n\nThese specific pruning strategies have not been applied to differentiable decision tree models before to improve interpretability. In training the IDCT model, we conducted a hyperparameter search over different tree sizes and chose the best-performing model post-pruning with under 32 leaves. In the end, we found that a 256-leaf IDCT model was most amenable to training in each domain, and contextual pruning was able to reduce this large tree into a simple tree of three leaves and two leaves, respectively (a 64-128x reduction in model size). This pruning technique is essential as trees of arbitrarily large depths can be difficult to understand [1] and simulate [2], and that a sufficiently sparse DT is desirable and considered interpretable [3].\n\n[1] Abhishek Ghose and Balaraman Ravindran. Interpretability with accurate small models. Frontiers452\nin Artificial Intelligence, 3, 2020.\n\n[2] Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. Interpretable decision sets: A454\njoint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD455\nInternational Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1675–1684,456\nNew York, NY, USA, 2016. Association for Computing Machinery.\n\n[3] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of458\ninterpretability is both important and slippery. Queue, 16(3):31–57, 2018.\n\n**Human Improvement from Iteration 1 to 4** -- In the rebuttal document, we have added in an additional display of the reward trajectories for each participant, with a single color assigned to each participant's behavior for each domain. These figures, while showcasing the trajectories of each participant well, can get \ncluttered and thus we chose to showcase our findings via a scatter plot. In IV2-D1: Forced Coordination, you can see that many participants do not linearly increase from Iterations 1 to 4. Often, mistakes in tree logic can lead to an agent that does not collaborate well and as this domain requires collaboration, leads to a sub-100 score. \n\nThe improvement in IV2-D2: Optional Collaboration between iterations one and four was found to be significant (p<0.01). This implies that that in this domain, users were applying a conscious strategy when modifying the agents policies. However, even in this domain, the improvement was not necessarily monotonically increasing for all participants.\n\n**Additional Limitation** -- We thank the reviewer for noting this additional limitation and will add this into our section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their insightful reviews and valuable feedback on our paper. We have included a rebuttal document with additional figures as well as provided rebuttals to each reviewer below.

Meta-review: All the reviews for this paper are positive, even though 2 of them are borderline. These reviewers raised issues that have been somewhat addressed in the author's rebuttal. However, there are still issues with the clarity and generality of the results.

Nonetheless, all reviewers agree that the paper is quite interesting and covers a very relevant topic. I recommend acceptance as a poster.

Finally, I recommend that the authors thread lightly regarding unsupported claims, as discussed with Reviewer cdZ4.
Decision: Accept (poster)