Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

Authors: rohan  paleja
michael  munje
kimberlee  chestnut
reed  jensen
mathew  gombolay
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:16:41

Abstract: Collaborative robots and machine learning-based virtual agents are increasingly entering the human workspace with the aim of increasing productivity and enhancing safety. Despite this, we show in a ubiquitous experimental domain, Overcooked-AI, that state-of-the-art techniques for human-machine teaming (HMT), which rely on imitation or reinforcement learning, are brittle and result in a machine agent that aims to decouple the machine and human’s actions to act independently rather than in a synergistic fashion. To remedy this deficiency, we develop HMT approaches that enable iterative, mixed-initiative team development allowing end-users to interactively reprogram interpretable AI teammates. Our 50-subject study provides several findings that we summarize into guidelines. While all approaches underperform a simple collaborative heuristic (a critical, negative result for learning-based methods), we find that white-box approaches supported by interactive modification can lead to significant team development, outperforming white-box approaches alone, and that black-box approaches are easier to train and result in better HMT performance highlighting a tradeoff between explainability and interactivity versus ease-of-training. Together, these findings present three important future research directions: 1) Improving the ability to generate collaborative agents with white-box models, 2) Better learning methods to facilitate collaboration rather than individualized coordination, and 3) Mixed-initiative interfaces that enable users, who may vary in ability, to improve collaboration.

Main paper: 
 introduction: 
 Successful human-machine teaming (HMT) has long been sought after for its wide utility across potential applications, ranging from virtual agents such as "clippy" that provide on-demand support DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014) . Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.for improving documents to embodied robotic healthcare aides that can provide doctors with a helping hand [29] . While promising, achieving fluent HMT is challenging because interactions with humans can be incredibly complex due to the diversity across users [26] , human teammates benefit from explainable systems to support the development of mental models [31] , and the lack of bidirectional communication (i.e., unclear how humans can "tell" a machine online to perform a desired behavior) [46] . In this paper, we transition from the conventional approach of crafting an HMT solution that aims for flawless out-of-the-box performance to a paradigm where end-users can actively interact with and program AI teammates, fostering a more dynamic and developmental interaction between humans and AI. Specifically, we explore enabling humans to perform user-specific modifications to a collaborative AI's interpretable policy representation across repeated iterations of teaming episodes and provide a set of design guidelines to support team development in HMT drawn from a large-scale user study.Recently, data-driven techniques (e.g., imitation and reinforcement learning) have become popular in HMT, allowing for the generation of collaborative agent behavior without cumbersome manual programming [40, 5] . However, these prior works utilize opaque, black-box models, limiting human's ability to develop a shared mental model and maintain situational awareness [27] , crucial for highperformance teaming [37] . We posit that successful, real-world HMT is not feasible without the use of white-box methods, especially in safety-critical domains such as healthcare and manufacturing. Furthermore, collaborative interactions with machines have often lacked the ability to effectively learn with and adapt to human teammates in real-time [19] . In ad hoc human-human teams, effective teaming is often developed through an iterative process [43] . Bi-directional communication is often a key component of this process, enabling the development of successful coordination strategies [36] . In our work, we build towards such a team development paradigm in HMT by 1) creating a pathway of bi-directional communication, utilizing interpretable policy representations as a mechanism to allow users to understand their machine teammates and allowing for explicit teammate policy modification through an interface (users can modify the machine's tree-based policy via a GUI), and 2) allowing for the process of iterative mixed-initiative team development through repeated teaming episodes. We believe this paradigm is necessary because human-partnered systems need explainable components and adaptable systems. We provide the following contributions:• We provide a case study regarding prior work in HMT [5, 40] , finding that the generated machine behavior is unable to adapt to human-preferred strategies, and that high performance is typically driven by independent machine actions rather than collaboration, which can ultimately result in a higher team score. • We create a novel InterpretableML architecture to support the creation of tree-based cooperative agent policies via reinforcement learning and a GUI to allow users to modify the AI's behavior to their specifications. This capability is promising, enabling end-users to "go under-the-hood" of machine learning models and tune affordances or interactively and iteratively reprogram behavior. • We conduct a 50-participant between-subjects user study assessing the effects of interpretability and interactive policy modification across repeated interactions with an AI. We summarize our study findings into a set of design guidelines to support future HMT research. 
preliminaries: 
 Here, we introduce prior work in HMT and Explainable AI, our experimental domain, Overcooked-AI, a model of team development used to understand our findings, Tuckman's Model, and the mathematical framework under which we generate agents, Markov Games.Human-Machine Teaming -The field of HMT is concerned with understanding, designing, and evaluating machines for use by or with humans [6, 44, 30] . A popular technique that has been used to produce collaborative AI agents is Reinforcement Learning (RL) [28] , where researchers have concentrated efforts on reducing the dissimilarity between synthetic human training partners and testing with human end-users. Approaches that have achieved some success include utilizing human gameplay data to finetune simulated training partners to behave more human-like [5] , which can be expensive, and training with a diverse-skilled population of synthetic partners to create an agent that can better generalize to non-expert end-users [40] , which may bias the AI teammate to exhibit individualized strategies, as we display in Section 3. We note our work focuses on an interaction different from AI-assisted decision-making or decision support. Here, a human and an agent must collaborate across a series of timesteps, aiming to maximize a multifaceted joint objective function.Explainable AI -xAI is concerned with understanding and interpreting the behavior of AI systems [23] . In our work, we follow recent trends that show black-box methods paired with local explanations can be harmful [34] and utilize interpretable, white-box tree-based models in a multi-agent sequential decision-making problem. These models have been shown to be beneficial in improving the user's ability to simulate a decision-making model [42] and providing users with increased situational awareness over a teammate's behavior in an HMT setting [31] . While tree-based models can provide users insight into the model, the complexity of the tree-based model limits its utility [24] . While we note this as a potential weakness of utilizing tree-based models, effective state representations can provide a tradeoff between granular control and tree depth. Accordingly, we design our trees to reason over a state-space with high-level binary features and multi-step macro-actions, expanded on below. Furthermore, in our work, we explore a paradigm where a user can directly modify and visualize a tree-based AI teammate the user is interacting with after a teaming episode. Prior work in explainable debugging [18] and robotics [33, 9] has explored similar paradigms, creating interactive systems that allow end-users to modify agent behavior to increase performance, but has not explored deploying tree-based models trained via RL in a collaborative HMT setting. We provide a working definition of what we mean by "interpretable" within the Appendix Section G.Overcooked-AI -Overcooked-AI [5] is a testbed to evaluate human-AI interaction and has been used across HMT research concerned with collaboration [40] , teammate identification [14] , intention prediction [45] , and behavior influence [16] . Here, two agents are tasked with creating and delivering as many soups as possible within a given time. Achieving a high score requires agents to navigate a kitchen and repeatedly complete a set of sequential high-level actions, including collecting ingredients, placing ingredients in pots, cooking ingredients into a soup, collecting a dish, getting the soup, and delivering it. Both players receive the same score increase upon delivering the soup. We modify the original Overcooked-AI game to be a simultaneous-move game as opposed to the original formulation of allowing agents to perform actions asynchronously. This modification prevents the collaborative score metric from being dominated by super-human AI speed, causing the overall score to be more reliant upon effective collaboration and strategy. We provide details about the state and action space below and complete details in the appendix.State-Space: Policies reason over a semantically meaningful feature space as opposed to pixel space, detailing the objects each agent is holding, pot statuses, and counter objects. This state space allows for learning an interpretable tree-based policy that can be understood and manipulated by end-users.Action-Space: Instead of using cardinal actions, we allow the AI to utilize macro-actions that can accomplish high-level objectives such as ingredient collection, ingredient placement, and soup serving. Macro-actions are planned using an A* planner, and we perform dynamic replanning at each timestep. Constructing trees on a higher level of abstraction results in smaller trees that are easier to interpret.Tuckman's Model -Tuckman [43] describes the different stages that a team goes through before reaching high performance, including "Forming", "Storming", "Norming" and "Performing," often seeing a drop in performance as team members acclimate, followed by a rise as team members understand how to collaborate. Assuming that human-machine teams will follow similar stages to human-human teams, this paper looks into how we can support human-machine teams in reaching the Performing stage, where the team is achieving its full potential and exhibiting the highest level of cooperation. We provide a depiction of these stages as part of Figure 2 .Markov Game -We formulate our setting as a Markov Game [25] , defined by a set of global states,S 1 , S 2 ∈ S, a set of actions, A 1 , A 2 ∈ A, transition function, T : S × A 1 × A 2 → S. and reward function r i : S × A i → R. Agent i aims to maximize its discounted reward R i = T t=0 γ t r t i , where γ ∈ [0, 1] is a discount factor.For training, we utilize agent-agent collaborative training, which trains two separate agents jointly via single-agent PPO. We utilize PantheonRL [38] for training our agents, incorporating our novel tree-based architecture (Section 4.1) into the codebase.In this domain, the team can operate individually or collaboratively. This domain has increased complexity, both with respect to the size of the domain and the types of soups that can be cooked. Collaboration is incentivized through a higher reward for mixed-ingredient dishes (combining onions and tomatoes) over single-ingredient dishes. IV1-C1 ✓ ✓ ✓ IDCT IV1-C2 ✗ ✓ ✓ IDCT IV1-C3 ✗ ✗ ✓ IDCT IV1-C4 ✗ ✗ ✗ IDCT IV1-C5 ✗ ✗ ✗ NNProcedure: A participant is first randomly placed into one of the five conditions in IV1.The participant starts with a pre-experiment survey collecting demographic information, experience with video games and decision trees, and the Big Five Personality Questionnaire [7] . Afterward, a participant conducts a brief tutorial in Overcooked with a random AI agent, improving the user's understanding of game controls and the assigned task. Once completed, the primary experimentation begins. Users will team with an AI four times in each domain (randomly ordered), starting with the unique domain-specific pre-trained agent, and are told that their goal is to maximize their score in the last teaming interaction, the "performance round." After each teaming interaction, in the first three factors, the user will modify and visualize the AI's policy (IV1-C1), the AI will optimize its own policy proceeded by user visualization (IV1-C2), or the user will solely view the policy (IV1-C3). In IV1-C4 and IV1-C5, as the AI is black-box (perceived to be black-box in IV1-C4 and truly black-box in IV1-C5), transitionary pages are shown to the participant, providing them a pause before they team with the agent again. Upon completion of the condition-specific (or lack of) actions, users complete a NASA-TLX Workload Survey. After users have completed a domain, providing us with four episodes of teaming data and workload assessments, we administer several post-study scales, including the Human-Robot Collaborative Fluency Assessment [15] , Inclusion of Other in the Self scale [1] , and Godspeed Questionnaire [2] . Upon completion of the two domains, the experiment concludes. 
a gap in teaming performance: 
 In this section, we present two examples to display a gap in the quality of AIs in HMT. Specifically, we look at two recent approaches to produce collaborative AI agents [40, 5] . We argue and display that the AIs trained via these approaches are rigid and exhibit individualized behaviors, missing out on collaborative teaming strategies that can ultimately result in higher team scores. We require AI agents that can effectively reach a consensus with humans on a teaming strategy that ultimately results in high performance. In cases where the human has a preferred strategy, the AI teammate should be able to support said strategy. In Figure 1 , we display the Coordination Ring scenario. A simple collaboration strategy (which we term "human-preferred") in this domain is to utilize the counter to continuously pass objects, minimizing agent movement through efficient handoffs. To test a set of collaboration strategies, we utilize agents publicly available from Carroll et al. [5] . In Figure 1 , we display a frame-by-frame of the human-preferred coordination strategy (Figure 1a ) and AI-preferred coordination strategy (Figure 1b ), which was a strategy where agents act individually to collect ingredients and place them in pots. The latter behavior was inferred through repeated play with the publicly-available AI. With the human-preferred strategy, the AI agent freezes for the majority of the game, creating an extremely frustrating and low-performing AI teammate. In this scenario, the human (green) picks up an ingredient and places it on the counter at the start of the game. The AI agent (blue), unfamiliar with this teaming strategy, freezes for approximately 80% of the remaining episode before finally placing an onion in the pot. With the AI-preferred strategy, the human is able to successfully team with the AI, with each agent retrieving and placing ingredients while moving in a clockwise motion, but the strategy is not optimal or what the human prefers. As the AI produced by Carroll et al. [5] is created via RL teaming human-like AI teammates, the generated behavior may not be ideal for the current teammate, especially if the current teammate's preferred strategy was not present in the original training dataset used to create human-like AI training partners. This highlights a need for systems that can explain strategies exhibited by trained agent policies and allow humans to adapt these pre-trained policies toward human-preferred behavior.In a second example, we utilize the Optional Collaboration domain, displayed on the right-hand side of Figure 4b , which is also utilized in our human-subjects experiment. This domain was designed to incentivize collaboration, where creating mixed-ingredient dishes facilitated by agents passing ingredients across the central counter will result in a higher score per dish. Here, we program two intelligent deterministic heuristics: In the first, each agent acts completely individually, cooking single-ingredient dishes and serving. In the second, agents share ingredients, which costs additional timesteps, but are able to successfully cook mixed ingredient dishes. We find that the collaboration strategy achieves a 408 cumulative team score, approximately 30% more score compared to the individualized strategy of 306. However, we find that trained policies under Ficticious Co-Play [40] exhibit similar team score to that of the individual coordination strategy and further, find that real human end-users collaborating with these agents are unable to far surpass the individual strategy score. As Strouse et al. [40] trains an agent to work well with a population of agents, where approximately a third of the diverse-skilled population of agents used in training are completely random agents, we posit that the teammate agent must compensate and exhibit individualized behavior, limiting the algorithm's ability to effectively learn effective team coordination strategies. In line with the first case study, the trained collaborative agent policies miss out on high-performance teaming behaviors, and thus, we need systems where humans can iteratively improve agent behavior online.Thus, in the rest of the paper, we look to explore xAI techniques as a mechanism for closing this gap and allowing agents within a human-machine team to facilitate collaborative strategies that outperform the individualized and rigid behaviors trained agents assume.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
methodology: 
 In this section, we first present our architecture for training interpretable AI teammates. We then present a contextual pruning algorithm, allowing for ease-of-training and enhanced interpretability for neural tree-based models. We display an overview of our training procedure as part of Figure 2 .We create an interpretable machine learning architecture, Interpretable Discrete Control Trees (ID-CTs), that can be used directly with RL to produce interpretable teammate policies. Below, we briefly detail our architecture, as well as advancements to enhance ease-of-training and interpretability.Architecture Our IDCTs are based on differentiable decision trees (DDTs) [41] -a neural network architecture that takes the topology of a decision tree (DT). DDTs contain decision nodes and leaf nodes; however, each decision node within the DDT utilizes a sigmoid activation function (i.e., a "soft" decision) instead of a Boolean decision (i.e., a "hard" decision). Each decision node, i, is represented by a sigmoid function, displayed asy i = (1 + exp(-α( ⃗ w T i ⃗ x -b i ))) -1 .As this representation is difficult to interpret, Paleja et al. [32] presented differentiable crispification, which recasts each decision node to split upon a single dimension of the input feature and translates the outcome of a decision node so that the outcome is a Boolean decision rather than a set of probabilities. This, in turn, allows for an interpretable forward propagation through the model that traces down a single branch of a tree as well as gradient flow afforded by the straight-through trick to update parameters of the neural tree model. We utilize this approach to learn interpretable tree-based teammate policies via reinforcement learning.Figure 2 : Here, we provide an overview of the steps to produce a collaborative AI teammate with an interpretable policy and the proposed policy modification scheme evaluated in our user study.We initialize our IDCTs to be symmetric DTs with N l decision leaves and N l -1 decision nodes. Each decision leaf is represented by a sparse categorical probability distribution over actions. At each timestep, a state variable is propagated through each decision node, split on a single decision rule, with the output being a Boolean causing the decision to proceed via the left or right branch until arrival at a leaf node. At each leaf node, we sample from the respective distribution to produce a macro-action (e.g., in Overcooked-AI, "get an onion" or "place ingredient on counter"). Further, we improve model predictability by applying an L1 norm loss over leaf node distributions to ensure sparsity, penalizing high entropy action distributions at a leaf 1 . Importantly, the resultant representation after training is that of a simple decision tree with categorical probability distributions at each leaf node.Contextual Pruning As we focus on creating agents that must cooperate with and be interpreted by humans, we must limit the size of our tree-based models to a certain depth to promote user understanding. Analogous to the "lottery ticket hypothesis" in network training that supports the practicality of employing large models [11] , a small tree with a limited number of sub-trees (lottery tickets) may not have the representational power to learn a high-performing policy. Thus, the ability to effectively train IDCTs is at odds with maintaining user readability and simulatability. Following work in neural network pruning [22] , we design a post-hoc contextual pruning algorithm that allows us to simplify large IDCT models while precisely adhering to model behavior by accounting for:We provide further details and an algorithm for contextual pruning in the supplementary material. This, in turn, allows us the benefit of training large tree-based models, greatly improving ease-of-training, while still being able to simplify the resultant model to a smaller, equivalent representation.While the above architecture can be used alongside RL to produce a collaborative AI policy, the result may not actually be helpful or what the human wants. Humans, when teaming with machines, should be able to intuitively update what the robot has learned or change it based upon preferences that may evolve over time. Such is critical in the positive development of coordination strategies and is associated with the calibration of trust, assignment of roles, and development of a shared mental model. As such, we propose a policy modification scheme that allows the user to repeatedly team with an AI maintaining an IDCT policy, visualize the current behavior in tree form, and modify its AI's behavior.Figure 3 : Users have several capabilities in creating an effective teammate, including modifying the tree structure by adding or removing decision nodes, changing state features the tree is conditioned on, and modifying actions and/or their respective probabilities at leaf nodes.The iterative process generated through this scheme can facilitate a feedback loop, allowing for the possibility of team development and improved HMT performance over teaming episodes.We term our modification scheme human-led policy modification. We provide humans with an explicit pathway to "communicate" with an AI after each teaming interaction through a GUI, with capabilities displayed in Figure 3 . Within this interface, users start with the pre-trained collaborative AI IDCT policy and can modify the AI's behavior by creating a new tree structure that may vary in what state features appear in the decision nodes, actions taken in leaf nodes, and the respective probabilities of actions within the leaf node. It is important to note that users are limited to expanding the tree to a depth of four (i.e., a max of 16 leaves), and the modification is not timed.Across our experiment, we study collaboration in two domains, Forced Coordination and Optional Collaboration, displayed on the left-hand side of Figure 4 . In each domain, we train an IDCT policy via agent-agent collaborative training and a neural network (NN) policy following the populationbased training scheme in Strouse et al. [40] . In the first domain of Forced Coordination, the IDCT policy converged to a policy with an average reward of 315.22 ± 14.59, and the neural network policy converged to an average reward of 403.16 ± 16.08 evaluated over 50 teaming simulations with the synthetic human teammate the policy was trained with. In the second domain, Optional Collaboration, the IDCT policy converged to a policy with an average reward of 171.46 ± 18.89, and the neural network policy converged to an average reward of 295.02 ± 1.86. Thus, a consequent confound due to the current difference in performance capabilities between interpretable vs. black-box models is that the NN policy outperforms the IDCT policy in both domains. This displays a need for improving optimization algorithms for interpretable models representing collaborative agent policies. However importantly, while the initial simulated performance of interpretable models may underperform black-box models, the ability for humans to understand machine behavior and improve upon behavior may allow these approaches to compete or even outperform black-box NN models. We can also compare to the heuristic policies presented in Section 3, observing that the training performance of the IDCT and NN policies in the Optional Collaboration domain underperform the collaborative heuristic (408 vs. 295.02 and 171.46). We provide visualizations of the trained IDCT policies for each domain in the appendix, finding that after contextual pruning, the AI IDCT policy has two and three leaves, respectively.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The necessary details to reproduce teaming agents is described within the text (general model details in Section 4 and specific hyperparameters in Section C.4) and the experimental procedure to evaluate these agents with real humans is in Section 5. Furthermore, our GitHub repository contains specific scripts and instructions to reproduce our models. 
human-subjects study: 
 Here, we discuss our between-subjects user study that seeks to understand how users interact with an AI across repeated play under different factors. Below, we introduce our research questions, provide a description of the independent variables and procedure, and discuss our findings.Research Questions The presented research questions below seek to understand changes in overall human-machine teaming performance and performance changes across repeated gameplay. The latter question pivots from an episodic attitude of teaming to a longer-term gauge, allowing us to study the process of adaptation in HMT.1. RQ1: How does human-machine teaming performance vary across factors? 2. RQ2: How does team development vary across factors? Independent Variables We have two independent variables, IV1: the teaming method, and IV2: the domain. For IV1, we consider the following conditions (abbreviated by IV1-C):1. IV1-C1: Human-Led Policy Modification: After interacting with the agent (one teaming episode), the user can modify the policy via the GUI, allowing the user to update decision nodes and action nodes in the tree as well as tune affordances. Upon completion, the user can visualize the updated policy in its tree form prior to the next interaction. 2. IV1-C2: AI-Led Policy Modification: After interacting with the agent, the AI utilizes recent gameplay to fine-tune a human gameplay model via Behavioral Cloning and performs reinforcement learning for five minutes2 to optimize its own policy to better support the human teammate. Upon completion of policy optimization, the user can visualize the updated AI policy in its interpretable tree form prior to the next interaction. This is similar to HA-PPO [5] , adapted to an online setting. 3. IV1-C3: Static Policy -Interpretability: After interacting with the agent, the user can visualize the AI's policy in its interpretable tree form prior to the next interaction. Throughout this condition, the AI's policy is static. 4. IV1-C4: Static Policy -Black-Box: After interacting with the agent, the user does not see the AI's policy. Here, the AI policy is the same as IV1-C3, but the human has lost access to direct insight into the model. 5. IV1-C5: Static Policy -Fictitious Co-Play: [40] : User teams with an AI maintaining a static black-box, neural network (NN) policy trained across a diverse partner set. As this is a baseline, we utilize an NN rather than the legible IDCT policy used in other conditions (IV1:C1-4).For IV2, we consider the following domains displayed on the left-hand side of Figure 4 :1. IV2-D1: Forced Coordination: Users team with an AI that is separated by a barrier and must pass over items in a timely manner. Here, agents are forced to collaborate.Our experiment is a 5 (teaming method; between-subjects) × 2 (no. of domains; within-subjects) × 4 (no. of repeated evaluations; within-subjects) mixed-factorial experiment. We recruited 50 participants under an IRB-approved protocol, whose ages range from 18 to 32 (Mean age: 24.14; Std. Dev.: 4.10; 46% Female, 52% Male, 2% Non-Binary), with participants randomly assigned to each of the factor levels, with ten total subjects per level. The duration of the experiment was 70.98 ± 19.71 minutes3 . Our data was modeled as a full-factorial, between-subjects ANOVA. We test for normality and homoschedascity (see appendix) and employed a corresponding non-parametric test if the data failed to meet these assumptions. We display our objective findings in the right-hand side of Figure 4 .RQ1: Team Performance: In analyzing reward, we find trends with respect to the maximum reward participants obtained within a domain across iterations (Figure 5 ). Using Friedman's test, we find a significant difference across domains (χ 2 (1)=46.08, p < 0.001) and analyze the domains separately.In IV2-D1, a Kruskal-Wallis Test was conducted to analyze differences in maximum performance obtained across teaming paradigms, finding a significant effect (χ 2 (4) = 20.146, p < 0.001). We conduct post-hoc pairwise comparisons, utilizing Dunn's test, and find that IV1-C5 (Fictitious Co-Play) is significantly better than IV1-C1 (p < 0.001), IV1-C2 (p < 0.01), IV1-C3 (p < 0.01), and IV1-C4 (p < 0.05). Even though Fictitious Co-Play (IV1-C5) outperformed the tree-based models, likely due to its ability to converge to a higher-performance teaming policy, it is interesting that Human-Led Policy Modification (IV1-C1) has several participants that outperform the maximum performance of IV1-C5 in teaming iterations three and four (Figure 4a ).In IV2-D2, a Kruskal-Wallis Test was conducted to analyze differences in participant teaming performance across conditions, finding a significant effect (χ 2 (4) = 29.922, p < 0.001). We conduct post-hoc pairwise comparisons, utilizing Dunn's test, and find that IV1-C5 (Ficticious Co-Play) is significantly better than IV1-C2 (p < 0.001), IV1-C3 (p < 0.001), and IV1-C4 (p < 0.001), and IV1-C1 (Human-Led Policy Modication) is significantly better than IV1-C2 (p < 0.05), IV1-C3 (p < 0.05), and IV1-C4 (p < 0.05). For white-box AI teammates (IV1:C1-3), the latter finding displays the benefit of Human-Led Policy Modification in improving HMT performance for interpretable models. These findings display that 1) white-box approaches supported with policy modification can outperform white-box approaches alone, 2) black-box models can outperform white-box approaches in HMT, and 3) by comparing IV1-C3 to IV1-C4, interpretability alone afforded via tree visualizations did not provide any direct objective benefits. Finally, in Optional Collaboration, across all conditions we see that HMT scores are not near that of the collaborative heuristic, displaying a gap that must be addressed to achieve effective HMT. others. In IV2-D1, we find that conscientiousness is trending in its correlation with improvement (0.05 < p < 0.1).In IV2-D2, we find that participants with high familiarity with Trees improve more across iterations (F (1) = 7.448, p < 0.01). These findings signify that positive interaction with interpretable models may be more beneficial to those with an engineering background and specific personality traits.Finally, we detect an interesting trend in IV2-D1 under the IV1-C1 condition. We see a drop in performance between the first teaming iteration and later iterations, followed by a rise. We believe this relates to the Forming and Storming stages, where team members are still developing effective strategies to coordinate. As the last iteration shows an improvement in performance, we hypothesize that the team was shifting into the Norming stage. In future, it would be interesting to evaluate a larger number of iterations to see if the behavior would continue to uptrend. This requires further research due to the additional resources and time needed for more teaming iterations.Subjective Findings: In IV2-D1, we find that users did not find any subjective differences toward the teaming interaction across conditions. In IV2-D2 (Figure 5 ), we find that users find collaboration with AIs under condition IV1-C2 and IV1-C4, on average as less fluent than IV1-C1 (p<0.01, p<0.01), and IV1-C4 as less fluent than IV1-C5 (p < 0.05). Users also trusted the AI and perceived the AI contributed more in IV1-C5 than in IV1-C2 (p<0.05, p<0.05) and IV1-C4 (p < 0.05, p < 0.05). Furthermore, the users viewed the AI more positively in IV1-C1 and IV1-C5 than in both IV1-C2 (p<0.05, p<0.05) and IV1-C4 (p<0.05, p<0.01). Overall, participants generally assessed higher-performing agents more positively in their subjective ratings. In considering conditions that utilized a tree-based model (IV1-C1, IV1-C2, IV1-C3, and IV1-C4), we see the addition of interaction with the tree policy provides significant subjective benefits in positive teaming traits and collaborative fluency (defined within the Human-Robot Collaborative Fluency Assessment [2] ).In including the remaining condition, which utilizes a black-box model, IV1-C5: Fictitious Co-Play, and comparing it to IV1-C1: Human-Led Policy Modification, we see that even though Fictitious Co-Play outperformed Human-Led Policy Modification in terms of team reward (though not significantly in the domain of Optional Collaboration), no significant subjective differences were observed between these two conditions. This presents an interesting relationship between transparency, interaction, and performance in relation to subjective perception that warrants future research.Design Guidelines: To achieve fluent HMT, we specify the following forward-facing guidelines.1. The creation of white-box learning approaches that can produce interpretable collaborative agents that achieve competitive initial performance to that of black-box agents. This guideline is critical to providing humans with the subjective benefits obtained from interactivity with white-box models, objective benefits of black-box models, and the ability to interact with policies to facilitate team development. 2. The design of learning schemes to support the generation of collaborative AI behaviors rather than individual coordination. We need techniques that avoid converging to the local maxima of individual coordination and scenarios that allow for properly evaluating cooperation. 3. The creation of mixed-initiative interfaces that enable users, who may vary in ability and experience, to improve team collaboration across and within interactions. As we found a large diversity in perceived usability of our interface (finding an average score of 58.25 ± 27.61, with some users finding the interface good (>75) and others poor (<35)), effective interfaces are vital in shifting from only a subset of users benefiting to all users being able to create effective teammates. 4. The evaluation of teaming in a larger number of interactions. As agents are deployed, team performance will change over time, going through a transient period before reaching peak performance. Understanding this process of team development is essential in creating high-performance HMT.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification:We provide our code in the following anonymous GitHub repository: https://github.com/CORE-Robotics-Lab/ Team-Development-with-Transparent-Policies. This repository contains information on how to set up the environment, train agents, and evaluate agents online. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 This work investigates repeated interactions with machine learning models within a sequential decision-making HMT paradigm. We present a key gap in HMT, displaying that current methods do not facilitate human-machine collaboration to the fullest. We find that human-led policy modification allows for a team to achieve higher performance than white-box models without this capability. However, as interpretable models are more difficult to generate, Fictitious Co-Play is able to better support high performance. Given these mixed findings, future work must focus on developing better white-box teammates, study the modality of communication in HMT, and explore mechanisms to allow HMT to scale beyond individual coordination and toward effective collaboration. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Yes, we provide exact information with regard to the statistical tests used to analyze our experiment data in Section 5. Further context, which ensures that the assumptions for these tests are met, is provided in the Appendix Section E. Error bars are displayed in Figures 4 and 5 , and represent the standard deviation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide compute information within Section C.4, including the type of computer and its resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We confirm that our paper conforms to the NeurIPS Code of Ethics. The conducted human-subjects study was reviewed by a University Internal Review Board to comply with ethical practices.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Yes, our paper discusses potential positive and negative societal impact within the Appendix Section F.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: Yes, screenshots of our experiment are provided in the attached codebase as well as complete code to run our experiment. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: We are aware of our use of human-subjects and conducted our experiment with caution. Our experiment was reviewed and approved by the Institutional Review Board at the Georgia Institute of Technology under Protocol Number H23043. Furthermore, all participants signed a consent form, received a description of the risks involved in our study, and received compensation for participating. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the Appendix, we provide further information regarding our testbed for Human-Machine Collaboration, Overcooked-AI (Section B), additional model and training details for our interpretableML architecture, the Interpretable Discrete Control Tree (Section C), complete information regarding our statistical analysis (Section E), further discussion regarding our paper's results, limitations, and future work (Section F), and finally, a working definition of what we mean by "interpretable".Overcooked-AI [5] is a testbed to evaluate human-AI interaction and has been used across numerous prior work studying human-AI collaboration [40, 10] . Here, two agents are tasked with creating and delivering as many soups as possible within a given time. Achieving a high score requires agents to navigate a kitchen and repeatedly complete a set of sequential high-level actions, including collecting ingredients, placing ingredients in pots, cooking ingredients into a soup, collecting a dish, getting the soup, and delivering it. Both players receive the same score increase upon delivering the soup. We modify the original Overcooked-AI game to be a simultaneous-move game as opposed to the original formulation of allowing agents to perform actions asynchronously. This modification prevents the collaborative score metric from being dominated by super-human AI speed, causing the overall score to be more reliant upon effective collaboration and strategy.We utilize two map configurations we term, Forced Coordination and Optional Collaboration, displayed in Figure 4 of the main paper. Each domain was chosen so that collaborating with the teammate would result in a higher score than working individually. In our newly-created domain, Optional Collaboration, creating mixed-ingredient dishes (combining onions and tomatoes) will receive a higher score than single-ingredient dishes. Teammates have 200 timesteps to collaborate and cook as many dishes as possible.State-Space, Action-Space, and Reward Scheme Policies reason over a semantically meaningful 13-dimensional feature space as opposed to pixel space, detailing the objects each agent is holding, pot statuses, and counter objects. Each of these features is binary. For the action space, instead of using cardinal actions, we allow the AI to utilize macro-actions that can accomplish high-level objectives such as ingredient collection, ingredient placement, and soup serving. Macro-actions are planned using an A* planner, and we perform dynamic replanning at each timestep. Prior work has shown macro-actions can enhance interpretability [4] . This state and action space allow forlearning an interpretable tree-based policy that can be understood and manipulated by end-users.In Forced Coordination, for the reward scheme, we follow a similar distribution as prior work and give a reward score of 60 per dish served, 3 for an item placed into a pot, 3 for a useful dish pickup, and 5 for a soup pickup. In Optional Collaboration, for the reward scheme, we give a reward score of 50 for a mixed-ingredient dish, 30 for a single ingredient dish, 3 for an item placed into a pot, 3 for a useful dish pickup, and 5 for a soup pickup.Here, we provide additional model details for the proposed Interpretable Discrete Control Tree (IDCT).Our IDCTs are based on differentiable decision trees (DDTs) [41] -a neural network architecture that takes the topology of a decision tree (DT). DDTs contain decision nodes and leaf nodes; however, each decision node within the DDT utilizes a sigmoid activation function (i.e., a "soft" decision) instead of a Boolean decision (i.e., a "hard" decision). Each decision node, i, is represented by a sigmoid function, displayed asy i = 1 1+exp(-α( ⃗ w T i ⃗ x-bi)), where ⃗ w i and b i represents the weight and bias terms of the decision node, respectively. As this representation is difficult to interpret, [32] presented differentiable crispification, consisting of two components: 1) Decision node crispification, which recasts each decision node to split upon a single dimension of our input feature, and 2) Decision outcome crispification, which translates the outcome of a decision node so that the outcome is a Boolean decision rather than a set of probabilities. Both operations utilize the straight-through trick [3] to maintain gradients, allowing for both an interpretable forward propagation through the model that traces down a single branch of a tree as well as gradient flow to update parameters of the neural tree model. We utilize this approach in our IDCTs to maintain interpretability. We initialize our IDCTs to be symmetric complete decision trees with N l decision leaves and N l -1 decision nodes. Each decision leaf is represented by a sparse categorical probability distribution over actions. At each timestep, a state variable is propagated through each decision node, split on a single decision rule, with the output being a Boolean causing the decision to proceed via the left or right branch until arrival at a leaf node. At each leaf node, we sample from the respective probability distribution to produce a macro-action (e.g., "get an onion" or "place held ingredient on counter").For training this model, we utilize agent-agent collaborative training where an interpretable tree-based agent (maintaining an IDCT) is paired with a second policy (representing the human player), and both models are trained via decentralized PPO [39] . It is important to note that each agent maintains its own buffer and optimizers. Further, we improve model predictability by applying an L1 norm loss over leaf node distributions for the IDCT agent to ensure sparsity, penalizing high entropy action distributions at a leaf. Our training procedure mimics that of PPO, utilizing a modified loss function displayed in Equation 1, and policy update in Equation 2, where θ represents the aggregate set of weights for the IDCT, Ât represents the advantage estimate at time t, and a l represents the distribution maintained at each leaf, l.L(θ) = E τ min r t (θ) Ât , clip (r t (θ), 1 -ϵ, 1 + ϵ) Ât + L 1 λ|a l | (1)EQUATIONAs we focus on creating agents that cooperate with humans, we must limit the size of our interpretable tree-based models to a certain depth to promote user understanding. This follows prior work, finding trees of arbitrarily large depths can be difficult to understand [13] and simulate [24] , and that a sufficiently sparse DT is desirable and considered interpretable [20] . However, this can make training difficult, as a small tree may not have the representational power to learn a high-performing policy. In Algorithm 1, we present details of how contextual pruning is accomplished. In Step 1, we initialize a domain vector representing the current minimum and maximum values for each feature. Since our Overcooked domain utilizes binary features, all bounds are initialized to 0 and 1. Formally, this can be written as by the Cartesian product B = [0, 1] × • • • [0, 1], of cardinality d (where d is the dimensionality of the state space). In Step 2, we initialize a queue that will be used to perform a breadth-first search to visit each node in a hierarchical order. In Step 4, we receive a node from the queue. In Step 5, we check the threshold value of the current node and compare it to the current node's vector of minimum values. This operation looks to see if the node results in a tree sub-space that is out of bounds (i.e., impossible to reach). We perform a similar computation in step 8, checking the maximum values. In both cases, we look to find child nodes that do not yield a reduction in the hyperspace as candidates for pruning. In Step 11, we update the children based on the threshold value of our current node and its sign (as we can have < or > within a node), creating a new bounding box. In step 12, we add the children of the current node to the queue, and loop back to Step 4, repeating steps 5-12 until the queue is empty. In Step 14, we prune tree sub-spaces that are impossible to reach.The computational complexity of our contextual pruning algorithm can be analyzed in terms of both time and space complexity. In terms of time complexity, it is equivalent to that of Breadth-First Search (BFS), specifically, O(V + E), where V denotes the number of vertices and E represents the number of edges in the tree. Regarding space complexity, our algorithm exhibits similar characteristics to BFS for trees with only two leaves. In such cases, the space complexity of BFS is O(V ), as it stores all the vertices at the maximum breadth level in the queue during the traversal. Consequently, the space complexity of our contextual pruning algorithm is also O(V ), making it efficient and scalable for trees with a limited number of leaves.Utilizing contextual pruning alongside our training framework allows us the benefit of training large tree-based models, greatly improving ease-of-training, while still being able to simplify the resultant model to a smaller, equivalent representation.To evaluate the utility of pruning, we train models of various sizes (8-leaf, 16-leaf, 32-leaf, 64leaf, 128-leaf, 256-leaf) in Forced Coordination and perform pruning on the resultant model. We find that models of larger size converge to higher performance (i.e., easier-to-train), following prior work displaying the utility of larger models. Further, empirically, we find we can reduce model sizes by 64-128x in tree depth. We provide a pipeline to allow for model training and contextual pruning in our GitHub repository https://github.com/CORE-Robotics-Lab/ Team-Development-with-Transparent-Policies.In IV2-D1: Forced Coordination and IV2-D2, we train an IDCT with 256 leaves, a learning rate of 1e -3 , and regularization parameter of 1e -4 . This hyperparameters were chosen through trial and error, where we find larger models with a small learning rate and regularization exhibited greater learning early on. The rest of the parameters follow default parameters from the PantheonRL codebase [38] for training Overcooked agents. After contextual pruning, in both domains, we end up with an AI policy with two and three leaves in Forced Coordination and Optional Collaboration, respectively.For training fictitious co-play agents, we train 32 models of teammates in each domain, saving policies at every 100 epochs. At the end of training, we sort the performance of saved policies and utilize the initial, mid-performing, and highest to create our population of diverse agents, totaling 96 agents. A neural network model is then paired in a multi-task training framework to team with this agent.Our models are all trained on a local desktop computer containing a Nvidia RTX 2080 GPU and 16 GB of CPU memory. Training time for each agent took approximately 12 hours across a single core. We provide further instructions to replicate our models within the above codebase.We include a high-level diagram of how IDCT agents are generated in Figure 6 . Here, we present visualizations of trained IDCT models in each domain. As seen in Figures 7 and 8 , the resultant policies have two and three leaves for the Forced Coordination and Optional Collaboration domains, respectively. Note that these images are pulled from our interface and thus have extra annotations to improve readability.Our experiment was reviewed and approved by the Institutional Review Board at the Georgia Institute of Technology under Protocol Number H23043. All participants in our experiment signed a consent form, received a description of the risks involved in our study, and received compensation for participating. Below, we describe specifics regarding the consent procedure, additional details that describe the experiment procedure, and the compensation scheme.At the start of the experiment, the participant is provided a consent document. This document describes the purpose of the experiment, exclusion/inclusion criteria, the experiment procedure, the risks of the experiment, the compensation scheme, and details regarding data storage and confidentiality.IV1-C1: Human-Led Policy Modification is enabled through the contribution of the interpretable machine learning architecture to train collaborative AI teammates, a training advancement to enhance interpretability, and a mechanism to allow humans to modify the tree in simple ways, including tree deepening, decision variable modification, and leaf node modification. The following conditions: IV1-C2: AI-Led Policy Modification, IV1-C3: Static Policy -Interpretability, and IV1-C4: Static Policy -Black-Box all utilize the same architecture and starting policy but ablate different components of the interaction and interpretability.After a teaming episode in the IV1-C2: AI-Led Policy Modification condition, the AI utilizes recent gameplay to fine-tune a human gameplay model via Behavioral Cloning and performs reinforcement learning for five minutes to optimize its own policy to better support the human teammate. In this collaborative agent policy optimization stage, we utilize the parameters described in Section C.4 and add a timer to stop the optimization. Upon completion of policy optimization, we check if the policy has improved through simulated interactions with the behavior cloning agent, and if so, update the policy. In the case that the policy degrades, we use the original policy prior to optimization. The user can visualize the updated AI policy in its interpretable tree form prior to the next teaming interaction.IV1-C3: Static Policy -Interpretability and IV1-C4: Static Policy -Black-Box are static policies that do not change across repeated gameplay. Thus, we do not have any specific additional hyperparameters to discuss within the appendix.To improve the transparency of the conditions in our experiment, we provide a flow diagram that displays the interaction being assumed within each condition in Figure 9 .Participants were compensated at a rate of 20 US dollars per hour of the experiment.Here, we present complete details regarding our analysis, including all test statistics as well as nonsignificant and trending comparisons.As mentioned in the main paper, we allow humans to team with the AI across four episodes, providing us with four teaming scores. Within the main paper, we reported differences with respect to the maximum score participants were able to obtain across iterations. Here, we analyze data in the performance round (the last iteration), where participants were told to maximize performance. We note that participants self-reported their gaming familiarity (100-point scale) and weekly hours playing video games. Across all participants, self-reported gaming familiarity was rated as 73.19 ± 23.80 and weekly gaming hours was 4.44 ± 5.32. This information was used in our statistical analysis, and significance was not found in performance variation as a function of gaming expertise. Utilizing a Friedman's test, we find that there is a significant difference across domains (χ 2 (1) = 38.7, p < 0.001). Accordingly, we analyze the two domains separately.In IV2-D1, we find our data does not meet the necessary assumptions and utilize non-parametric tests. A Kruskal-Wallis Test was conducted to analyze differences in performance round reward across conditions, and we find a significant effect (χ 2 (4) = 20.85, p < 0.001) across conditions. We conduct post-hoc pairwise comparisons, utilizing Dunn's test, and find that IV1-C5 is significantly better than IV1-C1 (p < 0.01), IV1-C3 (p < 0.01), and IV1-C4 (p < 0.01). IV1-C5 is trending as significantly better than IV1-C2 with a p-value of 0.0275 (significance is < 0.025 or (α/2) due to the Bejamini-Hochberg adjustment).In IV2-D2, we test for normality and homoschedascity and do not reject the null hypothesis in either case, using Shapiro-Wilk (p > .50) and Levene's Test (p > 0.05). An ANOVA was conducted to analyze differences in performance round reward across conditions, taking several observed variables into account. We find a significant effect (F (4, 38) = 18.93; p < 0.001) across conditions and decision tree familiarity (F (1, 38) = 16.12; p < 0.05). We conduct post-hoc pairwise comparisons, utilizing Tukey HSD, and find that 1) IV1-C5 is significantly better than IV1-C2 (p < 0.01), IV1-C3 (p < 0.01), and IV1-C4 (p < 0.01), and 2) IV1-C1 is significantly better than IV1-C3. These results are similar to those in the paper when analyzing the maximum reward and result in a similar set of conclusions: 1) black-box models can outperform white-box approaches, and 2) white-box approaches with policy modification have some benefit over white-box approaches alone. Further, as we see that tree familiarity positively correlates with performance round rewards, exploring alternative paradigms, such as natural language for describing and programming trees may benefit users unfamiliar with decision trees.Here, we analyze the trends across iterations (did agents improve from iteration one to four) and identify characteristics of users that performed well in team development. Utilizing a Friedman's test, we find that there is a significant difference across domains (χ 2 (1)=20.48, p < 0.001).We conduct separate Wilcoxin signed-rank tests for each condition, and utilize the Bonferroni correction in determining significance (α/5). In IV2-D1, we see no condition significantly improves significantly over repeated iterations. In IV2-D2, we find that IV1-C1 (p < 0.01) and IV1-C2 (p < 0.01) significantly improve over repeated teaming interactions.F Discussion, Limitations, Future Work, and Societal Impacts Discussion: In this paper, we provide several contributions towards interactive HMT. We first present weaknesses in prior work, displaying that learned collaborative agents can be individualistic and rigid. To address these weaknesses, we propose an interactive scheme termed human-led policy modification to bridge the gap between individualized coordination and adaptive, effective collaboration. We do so by creating a feedback loop that facilitates team policy changes during HMT. This is accomplished by 1) utilizing an interpretable policy representation to provide users with insight into the teammate's decision-making, specifically the IDCT, an interpretable tree-based model that can be trained via reinforcement learning and pruned to a smaller, equivalent representation, and 2) creating a user interface to support the end-user modifying the policy to their evolving specifications. We deploy and compare our interactive policy modification scheme to several other techniques, including two popular prior works and variations of our proposed condition. While we do not a direct objective benefit of human-led policy modification compared to utilizing a black-box model supported with a population-based training scheme [40] , we find important takeaways that motivate the importance of conducting longer-term, repeated-interaction studies. Specifically, white-box approaches that facilitate interpretation can be used within a feedback loop to lead to policy improvement, users may require a larger number of interactions to reach a team consensus and maximal performance, and there are person-specific characteristics that may lead to some users being able to take advantage of interpretable models and interaction more than others.Limitations: This study was conducted at a university. While the population was diverse in age, gender, and university major, all students had some college education and most students were based in engineering, presenting a population bias. Furthermore, the population represented by the age group of 18 to 32 years old (mean of 24.14, std of 4.1) within our experiment may not directly generalize to an older population with extensive training. Furthermore, the experiment findings may not generalize to all contexts and scenarios within HMT. We reiterate that our findings are within a two-agent human-machine team within a relatively low-dimensional and short-horizon game, Overcooked-AI. In scaling to more complex and dynamic environments, the tree size needed to represent a high-performing agent will likely increase. In these cases, users may require more time to interact with and understand an agent's policy. There may be several capabilities that can be added to the Human-Led Policy Modification interaction paradigm, which may make the process quicker and easier. For example, model verification or forward simulation can be used to provide the human with other types of feedback prior to the next teaming iteration. Furthermore, for increasingly complex games, agent policies can also operate over different levels of abstraction, providing the human with a tradeoff with fine-grained control of the agent policy and tree size. Finally, different policy visualizations may better support certain populations of users, emphasizing the need for collecting user background information and future research in interpretability for embodied agents.Future Work: In the future, it would be interesting to conduct a similar experiment to a higher number of iterations, or until the team converges to a set of coordination strategies (the "performing" stage in Tuckman's model). Further, the possibility of adding in feedback from the AI regarding human-led policy modification (checking for logic inconsistencies, etc.) may be used to facilitate speedier team development. It would also be interesting to utilize different paradigms in communicating with the human as language may be an easier medium than a decision tree interface. Future work should also be done to optimize the accessibility of GUIs for policy modification via xAI techniques. Finally, expanding this research to real-world collaborative robot settings in healthcare of manufacturing that utilize tree-based policies, such as collaborative packaging [12, 17] or agile robotics [8, 21] , would lead to additional insight into human-machine team development with robot teammates.Positive and Negative Societal Impact: This work investigates repeated interactions with interpretable machine-learning-based agents in a collaborative game. As autonomous agents (e.g., robots) are deployed in the real world, insights from this work may be applied to assist in creating a fruitful working relationship between a human and an agent. We do not believe this has any negative societal impacts.As mentioned in the main paper, our agent representation is that of an Interpretable Discrete Control Tree, which reasons over a state space with high-level binary features and multi-step macro-actions. This model (which, in layman's terms, is a decision tree with action probabilities at each node) is the true learned model produced via reinforcement learning, not an abstraction created post hoc. This model is interpretable as its representation is "constrained in model form so that it is either useful to someone, or obeys structural knowledge of the domain, such as monotonicity, causality, structural (generative) constraints, additivity, or physical constraints that come from domain knowledge" [35] . In our case, the model constraints are inherent within the novel IDCT architecture, and the utility of this model to a user is that this model 1) is able to provide users with some awareness over the agent's behavior (and possibly, simulate the agent's decision making) and 2) provides users with the ability to explicitly modify agent behavior (a capability not possible with black-box models).• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Boundaries of a variable's state distribution: We utilize the minimum and maximum of each variable's range to parse impossible subspaces of a tree.Node hierarchy: Ancestor nodes for a specific decision node may have already captured a specific splitting criterion and, thus, may lead to redundancy. By detecting redundancies, we can prune subspaces of the tree.1 While utilizing deterministic AI policies may be easier to understand for users, we found these models could not converge to similar performance as the stochastic-leaf IDCT policies during training.We limit the online optimization time for the AI teammate to five minutes to create a feasible user-study. This RL optimization is challenging as only a limited number of samples can be obtained in this time, and thus, the policy is not guaranteed to improve. In cases where the policy degrades, we use the original policy prior to optimization.The significant variance in experiment duration arises from the granularity across our conditions. The increase in human effort to understand and interact with the policy results in an increase in duration. We note that as our experiment is relatively short, it is unlikely that experiment fatigue played a role in our results as would be common in experiments with large task variances. 


Decision: All the reviews for this paper are positive, even though 2 of them are borderline. These reviewers raised issues that have been somewhat addressed in the author's rebuttal. However, there are still issues with the clarity and generality of the results.

Nonetheless, all reviewers agree that the paper is quite interesting and covers a very relevant topic. I recommend acceptance as a poster.

Finally, I recommend that the authors thread lightly regarding unsupported claims, as discussed with Reviewer cdZ4.

Reviews: review: paper_summary: paper_summary: This paper focuses on developing strategies to enhance transparency and interpretability in human-AI teaming settings. Based on my understanding, two collaboration contexts have been considered: human-preferred collaboration and AI-preferred suboptimal teaming strategy. The authors have implemented specific strategies to tackle these contexts and address interpretability. The implemented strategies have been evaluated on the Overcooked game with real human users.
strength_weakness: strength_weakness: The main weakness of the paper is that it is very hard to follow, and the methodology lacks transparency. For instance, the paper does not present a single method to investigate explainability. Overall, the chosen modifications and the designed collaboration setups are not clearly presented. It is not straightforward to link Section 4 (Methodology) with Section 5 (Studies).

Similarly, the experimental results are very hard to follow. The numbers are presented but are not straightforward to interpret or use for cross-checking the claims. In particular, the evaluation metrics are not clear.

Additionally, only one collaboration environment has been considered. The findings might be biased toward this specific collaboration task.
questions: questions: Three main questions are:

- How do you envision improving the transparency of the proposed methodologies? How can Section 4 be better linked with Section 5?
- What are the evaluation metrics for the chosen collaboration settings? How is interpretability evaluated?
- Are there any other collaboration environments that can be considered for this line of research? How do you envision these findings being generalized to other collaboration tasks?
limitations: limitations: The authors discussed the limitations and the broader impact of their work adequately in the appendix.
paper_summary: paper_summary: In this manuscript, it focuses on the collaboration in human-machine teaming (HMT) based on interactive and explainable systems. In order to address the existing issues, such as decoupling, the author(s) explored an interesting paradigm in HMT and proposed some guidelines based on the study. Also, the author(s) pointed out some potential research directions, although some of the conclusions are intuitive, but interesting.
strength_weakness: strength_weakness: In my opinion, the weaknesses of this manuscript are as follows:

1. As the author(s) mentioned, due to the population setting, the findings have a population bias and may not generalize to wider situations.

2. The cited work is a little out of date. This makes this manuscript seem not ready for publishing at conferences such as NeurIPS.

3. Some descriptions in the main text are not so clear.

For more details, please see the Section "Questions" below.
questions: questions: I read the manuscript, and I have the following questions/comments. Thanks.

1. We need to balance sparsity and stability in designing a learning algorithm. In this manuscript, the author(s) adopted L1 regularization; I am not so sure if it will cause some issues due to the instability of l1.

2. Regarding how users interact with AI across repeated play under different factors, the author(s) performed some human-subject-involved studies. About the population used in the study, can you discuss the implications of the population bias in more detail and how it might affect the generalizability of your findings?
Also, maybe sex/age-based difference analysis is also worth exploring.

3. If possible, could you explain more about the potential challenges in scaling the human-led policy modification approach to more complex and dynamic environments beyond the Overcooked-AI domain?

4. Regarding the citations, I do not think the current references are sufficient, at least, for a manuscript preparing to submit to computer conferences such as NeurIPS, since the newest citation was two years ago, and there are only 4 such 2022 citations.

5. In Line 58, maybe "reinforcement learning" should be with its abbreviation, not until Line 70.
6. It would be great to be with the full name for PPO, such as Proximal ... (PPO)

Some other format issues in references:

(1) The author's name, sometimes using the abbreviation, sometimes not, such as Ref.[1], Ref.[33].

(2) Sometimes, the conference name has both the full name and its abbreviation, and sometimes not, such as, Ref.[4] vs Ref.[5].

(3) In Ref.[17], "ai"=>"AI".
Please check carefully; it would be great if the author(s) could correct these issues.


I would like to consider adjusting my current score based on the responses from the author(s). Thanks.
limitations: limitations: Yes.
paper_summary: paper_summary: In this paper the authors present an approach to human-AI teaming in the common Overcooked domain via Interpretable Discrete Control Trees (IDCTs), which are differentiable decision trees which the authors visualize and make controllable. The authors present two examples of where existing blackbox models may demonstrate a gap in teaming performance. They then present results of a human subject study that demonstrates that authors' approach is outperformed by a fictitious co-play baseline.
strength_weakness: strength_weakness: I like this work, but at present this paper has a number of key weaknesses holding it back. 

First, the current text of the paper contains a large number of unsupported claims. Almost all of the text in italics represents claims that are not substantiated by an argument, a citation, or the results of the paper. There are similarly design decisions that are not motivated or explained, such as why the authors chose to use IDCTs. 

Second, Section 3 introduces the possibility of a gap in teaming performance with current models. However, the evidence for this is presented as two examples. This is not sufficient to demonstrate that this specific approach has failure cases but not to demonstrate a need for the approach the authors' propose. I'd recommend working on a more full survey of existing Overcooked approaches to quantitatively evaluate the rate at which collaboration gaps may be occurring. This section represents the primary motivation for the authors' work, and so it being a weak point makes the whole paper weaker. 

Third, the authors do not give full technical details for any of the approaches, not the approach of Carroll et al., nor their own ICDT approach, nor their implementation of fictitious co-play. This is an issue as it makes it very difficult as a reader to understand what has been done at a technical level. For example, it's unclear to me what size the ICDTs used in the study were, I only know that the users were limited to expanding the tree to a depth of 4. But this doesn't tell me if the initial tree had a depth of 3. Similarly, the pruning approach only seems to remove redundant nodes so it's unclear how much this would actually prune (the authors state 8-16x smaller but it's unclear to which trees they refer). Given that a major problem with the ICDT's performance in the user study was their performance, these details are crucial. While there's an anonymized GitHub in the appendices, it should not be necessary to go through it to understand the work. 

Fourth, while the human subject study seems well-designed, certain details are unclear. It's unclear what population is being drawn on for the participants, though I would guess university undergraduates given the demographics. It's unclear how knowledgeable these participants were about AI except for one mentioned figure around familiarity with trees. It's unclear why the authors stopped recruiting at 50, since 10 per condition is considered an absolute minimum. Finally, the methodology is somewhat unclear, its unclear what order the participants experienced the domains. Was it consistent, if so, why? If not, there were 10 conditions, not 5.

Finally, the results do not seem to support the authors' claims. From the result, my takeaway is that participants preferred performance over interpretability. This goes against several of the authors' claims in the paper, most notably "As seen through these findings, the ability to interact with an interpretable model is perceived significantly better across several measures". The authors also make a claim about trending towards significance, which is not statistically sound.
questions: questions: 1. What was the size of the ICDTs used in the study?
2. Did participants experience the two domains in a fixed or random order?
limitations: limitations: The major limitations come from the poor performance of the ICDTs and the decision to limit the size of the human subject study. The authors do not sufficiently address these limitations.
paper_summary: paper_summary: The paper proposes a white-box approach to human-machine teaming, in which human teammates can see their virtual counterparts’ policies and adjust them accordingly. The framework is built on top of differentiable decision trees, and the authors propose contextual pruning as a means of simplification for training purposes and readability by the human team members. They use this framework to quantify the benefits of interpretability in HMT and the tradeoff between high interpretability and high accuracy, the latter typically better achieved by traditional black box solutions. The authors use their framework to conduct a thorough statistical analysis comparing different white and black box conditions and use their findings to list a series of guidelines for HMT.
strength_weakness: strength_weakness: The authors propose a pruning strategy to simplify training and interpretability but provide no evidence supporting either. It would be helpful to validate such claims with time and memory usage assessment and IV1-C1 analysis with and without pruning. The authors briefly discuss the former in C.3.2, but it could be improved by displaying the numbers.
questions: questions: - In line 230, the authors state that contextual pruning significantly improves the ease of training. Are the two pruning strategies enough? I am unfamiliar with DDT, so I wonder if this is novel. How is this different than traditional pruning methods in DDTs?
- As noted by the authors in line 357, some participants outperformed the maximum performance of IVI-C5 in teaming iterations 3 and 4. I wonder if those participants show an increasing trend in performance from iteration 1 to 4? I cannot tell that from the scatter plot. I am asking this because I am curious to know if the improvements are due to randomness or if humans are applying a conscious strategy when modifying the agents’ policies, leading to incremental improvements at every iteration.
limitations: limitations: The authors discuss the limitations of their work in the Appendix. I would argue there's one more limitation:

It seems performance in the mixed-initiative scenario is highly dependent on presentation, as validated by better performance improvements in participants more familiar with Trees. This work shows that interpretability is also relative. What is interpretable to me may be convoluted to another, which may also affect the team's performance. It would be interesting to check for differences in IVI-C1 with different visualization methods in future work.

Comments: title: Re: improved claims
Re: Official Comment by Authors
reply: Rebuttal by Authors
Updating
Re: Rebuttal by Authors
I maintain my decision
comment: Thank you for providing this valuable ethics review.

We will follow the recommendation of the reviewer and add an acknowledgment to the main paper that we are aware of the use of human subjects and that human-subject experiments should be conducted with caution. Furthermore, we will note that all participants in our experiment signed a consent form, received a description of the risks involved in our study, and received compensation for participating. In our appendix, will include full details of our consent procedure and compensation scheme. We will also include the IRB approval number in our de-anonymized manuscript.
Thank you for providing this valuable ethics review. 

**IRB** -- As noted by the reviewer, our experiment was approved by a university institutional review board (IRB). We would like to note that all participants in our experiment signed a consent form, received a description of the risks involved in our study, and received compensation for participating. *We will include the IRB approval number in our de-anonymized manuscript.*

**Population Bias Due to Age Group** -- We thank the reviewer for highlighting this important concern. The population represented by the age group of 18 to 32 years old (mean of 24.14, std of 4.1)  within our experiment presents a certain population bias and as the reviewer mentioned, our experimental findings may not directly translate to an older population with extensive training. We will note this as a limitation and see validating our findings in complex domains such as healthcare and manufacturing as an important future direction.

**Experiment Duration Variance** -- The large variance in experiment duration arises from the granularity across our conditions. We build from conditions that have no interaction or transparency (IV1-C4: Static Policy - Black-Box and IV1-C5: Fictitious Co-Play) to conditions with transparency but still no interaction (IV1-C2: AI-Led Policy Modification and IV1-C3: Static Policy - Interpretability) to conditions with both transparency and interaction (IV1-C1: Human-Led Policy Modification). This increase in human effort to understand and interact with the policy results in an increase in the experiment's duration. While this allows us to detect the effects of different characteristics like transparency and interaction, this does result in a large variance in experiment duration. As our experiment is relatively short, it is unlikely that experiment fatigue played a role in our results as would be common in experiments with large task variances. We will clarify these details regarding the experiment duration in our paper.

**Recommendations** -- We also thank the reviewer for the additional recommendations. We will include the following additional citations within our Future Work section as directions to expand this work toward collaborative robotics applications that utilize behavior-tree-based policies.
- Ghadirzadeh, Ali, et al. "Human-centered collaborative robots with deep reinforcement learning." IEEE Robotics and Automation Letters 6.2 (2020): 566-571.
- Iovino, Matteo, et al. "A framework for learning behavior trees in collaborative robotic applications." 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE). IEEE, 2023.
- D'Ambrosio, David B., et al. "Achieving Human Level Competitive Robot Table Tennis." arXiv preprint arXiv:2408.03906 (2024).

Furthermore, in the future work section, we will add "designing alternate metrics to analyze agent collaborativeness, both quantitatively and qualitatively". This may require a deeper assessment of state occupancy and resource sharing between agents for this specific domain.
Thanks for taking the time to clarify those points! I think that is much improved from the version in the paper. Given this and the other information presented in the rebuttal and this discussion, I have increased my score to a 6. I do not expect it will increase further as the remaining concerns are difficult to impossible to address in the rebuttal period, namely the relatively small participation pool/reference points for both the teaming gap studies and final human subject study.
Thank you very much for getting back to us and allowing us to clarify this disparity.  In clarifying this specific statement, we propose the following to better ground this claim.

"Overall, participants generally assessed higher-performing agents more positively in their subjective ratings. In considering conditions that utilized a tree-based model (IV1-C1, IV1-C2, IV1-C3, and IV1-C4), we see the addition of interaction with the tree policy provides significant subjective benefits in positive teaming traits and collaborative fluency (defined within the Human-Robot Collaborative Fluency Assessment [12]). In including the remaining condition which utilizes a black-box model, IV1-C5: Fictitious Co-Play, and comparing it to IV1-C1: Human-Led Policy Modification, we see that even though Fictitious Co-Play outperformed Human-Led Policy Modification in terms of team reward (though not significantly in the domain of Optional Collaboration), no significant subjective differences were observed between these two conditions. This presents an interesting relationship between transparency, interaction, and performance in its relation to subjective perception that warrants future research."




We will also update the first design guideline to the following (the change is bolded). This change should also contribute to reducing the previous overstatement.

1. *The creation of white-box learning approaches that can produce interpretable collaborative agents that achieve competitive initial performance to that of black-box agents.* This guideline is critical to providing humans with the subjective benefits **obtained from interactivity with** white-box models, objective benefits of black-box models, and the ability to interact with policies to facilitate team development. 


Again, thank you for helping us improve our manuscript.
I apologize for the delay in my response. My primary concern, per my initial review is "As seen through these findings, the ability to interact with an interpretable model is perceived significantly better across several measures". I recognize the comparisons being made, but my concern is that this overstates the outcomes, which appear to be more mixed. I recognize the authors have said they plan to revise the claims but there was no clarification as to how. Clarifying this point would be beneficial for my evaluation.
After considering all the feedback and responses, while I am not fully convinced of the paper's methodological strength, the idea and its execution are both original and solid. The paper offers insightful findings, which motivated me to raise my score.
Thanks for the responses from the author(s). The responses from the author(s) clarified some of my concerns to some extent. I increased my score from 4 to 5.
Thank you for reading our rebuttal and replying so quickly. We would love to improve our paper by adjusting the language around unsupported claims or better tying certain results to these claims. Above, in our reply, we discussed how our results directly relate to our guidelines. Are there any other specific unsupported claims you would be able to highlight so that we could improve our paper?

This would be much appreciated. Thank you in advance!
Thanks to the authors for their detailed response. My concerns around the claims and the results not supporting them, remains unchanged and so I am choosing to maintain my score.
Thank you for the rebuttal and additional results. I think this is a good paper. I maintain my decision.