CREW: Facilitating Human-AI Teaming Research
Authors: Lingyu Zhang
Zhengran Ji
Boyuan Chen
Venue: Accepted by TMLR ()

Abstract: With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce \textbf{CREW}, a platform to facilitate Human-AI teaming research in real-time decision-making scenarios and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.

Published in Transactions on Machine Learning Research (11/2024)
CREW: Facilitating Human-Al Teaming Research
Lingyu Zhang Duke University
bingggu.heang@idaake.oo edu
Zhengran Ji Duke University
zhengran.jj@dduke.com edu
Boyuan Chen Duke University
wwy.un..ceenniiee edu
Project Website: http://wwwgggraaoooooaaaaaaooooa
Reviewed on http::///pppnrroo....////////rua
Abstract
With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and Al agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AT teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-tcaming research or multi-agent AI algorithms. We introduce CREW, platform facilitate Human-AT leaming research in real-time decision-making scenarios and engage collaborations from multiple scientific disciplines, with strong emphasis human involvement. It includes pre-built tasks for cognitive studies and Hluman-Al teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Morcover, CREW benchmarks rcal-time human-guided reinforcement learning agents using state-of-the-arl algorithums and well-tumed baselines. With CREW, we were able to conduct 50 human subject studies within a week verify the effectiveness of our benchmark.
1 Introduction
Over the past decade, significant progress in machine learning has increased the potential and necessity for humans to collaborate and interact with Artificial Intelligence (AT) agents. Human-AT teaming has emerged as rescarch paradigm to explore the dynamic interactions and collaborative processes between humans and AI. By leveraging the complementary strength of both humans and AI, advancements can significantly enhance the overall team performance.
Unlike traditional AI rescarch, which typically focuses on machine lcarning algorithms in isolation, Human- AI learning requires multidisciplinary approach to incorporate insights from various scientific domains. Numerous studies have examined human-human teaming Klimoski & Mohammed (1994) with cognitive seience, neuroscience, and psychology. Machine learning and robotics communitics have extensively rescarchod multi-agent machine learning Zhang al. (2021), while team dynamics Salas (2018) has been explored in complex systems, social science, and network science. Despite the importance and potential of this research

---

Published in Transactions on Machine Learning Research (11/2024)
paradigm, there is still lack of comprchensive and unified platform to benefit research on joint efforts across disciplines and scalable hypothesis verification.
Developing a comprehensive platform for Human-Al teaming research presents several unique challenges. Firstly, the platform needs to support diverse tasks with varying complexities with easy extensions for new tasks or modifications. While reinforcement learning research platforms (Towers et al. (2023) have widely adopted this practice, current Human-Al teaming platforms remain limited to single tasks Carroll al. (2019) Vinyals et al. (0117). Secondly, enabling real-time communicttin through various modalities between multiple humans and heterogeneous AI policies is essential for effective collaboration. However, existing solutions typically support human feedback only through replaying offline trajectorics and do not implement real-time feedback mechanisms. Understanding how to build AI that can team with, learn from, be guided by, align with, and augment humans is as crucial as modeling human behavior during interactions with AI. Therefore, the platform must provide interfaces for recording human physiological data alongside agent data, tailored for cognitive science and neuroscience studies. Furthermore, current studies often involve small participant groups, making it difficult to derive generalizable conclusions. Lastly, the absence of unified platform has limited fully open-sourced studies and the establishment of appropriate benchmarks.
We present CREW, platform designed to facilitate Human-Al tcaming research aiming to address the above challenges. We develop CREW around key principles such as extensible and open environment design, real- time communication support, hybrid Human-Al teaming modes, parallel sessions for scalable experiments, and comprehensive human and agent data collection. Additionally, CREW incorporates highly modular algorithm components compatible with practices in the machine learning community. We demonstrate CREW's potential by benchmarking real-time human-guided reinforcement learning (RL) algorithms alongside various RL baselines. With CREW, were able to conduct 50 human subject studies within a week. Moreover, CREW includes a set of cognitive tests to explore how individual differences among humans impact their effectiveness in training Al agents. To our knowledge, CREW is the first platform to unify the desired features for Human-AI teaming research across multiple disciplines. We aim for CREW to serve as an infrastructural foundation for multidisciplinary, reproducible, and scalable Auman-AI leaming research.
2 Related Work
Human-AI Teaming Research Extensive research has explored Human-Al teaming across various domains. Machine learning studies have developed algorithms to leverage human expertise to improve the accuracy
Lertvittayakkujjrn et al. (2020), robustness Bao et al. (2018) Ziegler et al. (2019), and interpretability Lertvittayalumjon et al. (2020;; Bao et al. (018)): Zhou & Chen (2019) of models. Integrating human
feedback can not only improve performance Ibarz et al. (2018);
Christiano et al. (2017) but also align the
models with human preference Lertvittayakumjor et al. (2020): Ouyang ct al. (2022): Ziegler et al. (0019)
Human-computer interaction research has created interlaces Weisz el al. (2021] ; Neerinex el al. (2018) and
workflows [Bau et al. (2020) that enhance collaboration between humans and AI, combining their strengths to achieve superior performance. Ethical rescarch focuses on understanding and mitigating the socictal Hemmer
et al. 2023 ;
Chowdhury et al. (2022). ethical Ezer et al. (2019): Yin et al. (2019); Pflanzer et al. (0023), and technical Green & Chen (2019): Stahl et al. 2021 challenges of the rapid advancement and wide adoption
of A1. Many fields, including neuroscience, healthcare, robotics, transportation, education, security, and
accessibility, have shown growing interest Nourbakhsh et al. (2005): Henry et al. (022)): AAaaiihhyyv et al.
(2021): ; Nwana (1990); Pazho et al. (2023): : Kumar & Jain (2022) in Human-AT teaming. Overall, the broad
spectrum of interests highlights the need for multidisciplnnary collaboration to drive further advancements.
Human-AI Teaming Platform While significant progress has been made in Human-AI teaming research, there remains an absence of comprehensive research platform. Overcooked Environment Carroll et al. [2019 is a simplified version of the original popular game to challenge human agents and AI agents in tasks that require close coordination and strategic teamwork. StarCraft II Learning Environment (SC2LE) Vinyals el al. (017) supports adversarial settings to allow Human-Al interaction and learning from human demonstrations. Rapid Integration and Development Environment(RIDDE USC Institute for Creative Technologics (2024) focuses on defense-related scenarios, emphasizing rapid development and integration of AI systems for operational purposes. In addition to real-time decision-making tasks, previous research has developed
2

---

Published in Transactions on Machine Learning Research (11/2024)
Extenaible moce
Real-tine Interaction
Maltiplayer
Multimodal fredlback
Physiological data
Fully openssouroed
CREW
Limit
Overooled Carroll
2010
1+
Viavas
RIDE
CSC indt Tale for Cracive Technologes
a24
Limit
Metta
W
Table. 1: Human-AT teaming platforms. *Instead of real-time feedback training, RLHF-Blender and Uni- RLHF support online training where humans are presented with data from the replay buffer instead of the current experience.
platforms Preire el al. (2020) Metz et al. (2023): Yuan el al. (2024) that focus oflline preference rating settings where humans can provide offline evaluations or corrections with imitation learning or reinforcement learning.
Overall, existing platforms have more than one of the following limitations that constrain Human-Al AT leaming research as summarized Table. [ Most environments only support one type of task that can be difficult extend, and the interactions between humans and machine learning agents are limited mouse and keyboard operations. Moreover, most of the environments only support collecting game data, such state, action, reward focusing on the machine learning algorithm development, but none of them support the collection and analysis of human physiological data (eye movement, pupillometry, brain activity, heart impulse, or natural language) to understand human cognitive states and different effects on human subjects, as often required in the cognitive studies involved Human-Ar Al teaming Thakur et al. Dikker et al. (2017): Heinisch et al. (202]]; Gordon et al. (2020); Koorathota
al. (2023); Xu et al. (2021). Additionally, the scale of collaboration has been limited to two agents in collaborativ or competitive setting. Notably, most existing Human-AI teaming studies have only been evaluated on very small number of subjects among the authors (N < 10, mostly N < 5), which greatly limits our understanding of the state-of-the- art performance. Furthermore, common previous multi-agent environments such as OvercookedCarrool et al. (2019) and Starcraft Vinyals et al. 2017) are closed-sourced, making it unrealistic to build additional features on top of them. This also makes it impossible for researchers to modify the tasks to their own needs. There remains a large gap between the existing platforms and the desired environment to facilitate future research.
Real-Time Human-Guided Learning Most real-world decision-making processes require rapid decisions and adaptation in real time. Therefore, real-time human-guided learning is essential topic in Human-Al leaming research. Previous research has focused algorithm design to integrate real-time human feedback into policy training. TAMER and Deep TAMER Knox & Stone (2009); Warnell et al. (2018) learn to predict human discrete feedback and utilize the feedback model as the one-step myopic value function for policy learning. COACH and DeepCOACH AAruuuam et al. (2019); MacGlashan et al. (2016) addressed the inconsistency of human feedback by modeling il as the advantage Tunction under an actor-critic framework. Recent progress has further refined these algorithms, addressing various challenges in human feedback intogration, such as different feedback modalities feodback stochasticity Arakawa et al. (2018). and continuous action spaces Sheidlower et al. (2022) Other works have explored indirectly inferring human objectives from
feedback or preference Huang et al. 2023 Xiao et al. (2020); Christiano et al. (2017). CREW provides an extensive environment for real-time human
nuuann-uuded guided learning with online training and feedback integration, human physiological data collection, and parallel distributed experiment support.
3
CREW: Design and Components
3.1 Platform Vision and Design Philosophy
We design CREW to facilitate Human-Al teaming research. Our vision to construct a unified platform for rescarchers from diverse backgrounds, allowing them join forces from human-Al interaction, machine learning algorithms, workflow design, as well as human analysis and training. To achieve this, CREW incorporates the following capabilities, illustrated in Figure. 1
3

---

Published in Transactions on Machine Learning Research (11/2024)
Collect
Provide
aucio
dsccpta scala
continuuus scalar
take control
gare pupl
EEG
ECG
Real-time Feedback
Physiological Data
Play
Interact
Engine
Al Agent
Human-guided ML
Human analysis
Muiti-player Human-Al Teaming
Figure. 1: CREW a platform to facilitate Human-AI teaming research. CREW designed under the vision of multidisciplinarry collaboration research from both human and Al science. CREW allows real-time interaction among multi-human and multi-agents while enabling extensive data collection on both AI agents and human agents.
Extensible and open environment design. CREW is fully open-sourced. Our key contribution is building the infrastructure to allow the development of scalable environments with supported Human-AT research features that scale up to very complex settings. We implemented set of built-in example tasks described in Section 3.2 for rapid development. Many additional features and game logic can be added without modifying complex infrastructure-levd components.
Real-time communication. While some Human-AT interaction tasks, such as human preference-based fine-tunin,, can be performed offline, many applications require online real-time interaction. Whether is training preision-makinn models with real-time human guidance or general human-Al collaboration tasks, the ability to convey messages with minimum delay is essential. Synchronizing data flow between human interfaces, AI algorithms, and simulation engines necessitates the establishment of real-time communication channel.
Hybrid Human-Al teaming support. Teaming is an essential aspect of our daily jobs. Our vision extends this concept to Human-AI teaming, where both humans and AI operate in teams. Increasing interest in the organization, dynamics, workflow, and trust in multi-human and multi-Al teams highlights the need for a platform capable of distributing and synchronizing tasks, states, and interactions across multiple environment instances and even across physical locations.
Parallel sessions support. A key bottleneck for human-involved AI research is the requirement to conduct experiments with dozens or hundreds of human subjects to obtain trustworthy and reliable conclusions. Such process can be tedious and time-consuming, To enhance efficiency and scalability, CREW supports multiple independent parallel sessions of the same setting, unconstrained by geographical locations, to obtain the "crowd-sourcing" effects of large-scale experiments. This capability enables experimenters collectively share experimental data and results.
Comprehensive human data collection. Though human plays important role in Human-Al teaming, our understanding of human behaviors remains limited and under-explored in existing studies. Therefore, CREW offers interfaces to simultaneously collect multi-modal human data, ranging from active instructions and leedback to passive physiological signals.
ML community-friondly algorithm design. The choice of programming language and librarics should align with the customs and preferences of the ML community. The system design should be modular allow seamless transitions between tasks and algorithms.
4

---

Published in Transactions on Machine Learning Research (11/2024)
3.2 Environments
Tasks We select Unity the simulation engine for CREW due to its popularity in game design and AI research to allow extensible and open environment design. Unity, as used by many popular commercial games as has a high ceiling for the complexity of game development. It also has a large user community which makes customized task development more accessible. Adding new components to an environment can be done through simple atag-and-drog (on the left of Figure. 2
We have implemented four challenging tasks as examples. Multi-player tasks are designed for multi-agent and multi-human leaming research, and single-player tasks are designed for human-guided AT agent learning studies. For each task, provide both visual and structured state input options. The detailed settings are summarized in Table. 2
Bowling is a modified version of Atari bowling where each round consists of 10 rolls and the score for each roll the number of pins hit. Bowling is designed to have the simplest dynamics among our tasks to serve as rapid test for training a single agent. Find Treasure (Figure. 34) is a single-player embodied navigation task where the agent's goal is to explore a maze and reach the treasure with randomized initial and goal locations. Ivl Hide-and-Seck Chen et al. (2020 2021a) (Figure. 3B) is onc-on-one hido-and-soek task where the seeker learns to explore the maze and catch moving hider that follows a heuristic policy for obstacle avoidance, and run away from the seeker within line of sight. We introduce this task adversarial competition setting. NyN Hide-and-Seek (Figure. 3C) is a multiplayer version where multiple seekers and hiders can coordinate, collaborate, and compete. The hidors and seekers can either be controlled by humans or heterogeneous AI policies.
Table 2: Task Specifications
Tasks
Visual Observation
State Observation
Action Space
Reward
Bowling
grayscale rgb
ball pos, pos, status
release pos, length before steer, steer direction
# pias lis
Find Treasure
agent pos, treasure pos
next loc x, next loc y next x, next y nexct loc next
-1 step, +10 treasure found strp,, hider euught dedine
Hido-and-Srek.
rgb
pos, pos
NyN Hide-and- Seel
sockurs pos, hiders pos
Customizing new tasks or easily extend current tasks. Researchers can build tasks of much higher flexibility and complexity in CREW than existing platforms due to our efforts in setting distributed multi- agent systems mixing human and Al agents. We chose the of tasks above only for benchmarking purposes for hunan-guided reinforcement learning experiments, and it is far below the complexity limit of game design easily scale up the difficulty of Hide-and-Seek game to complex map. Another example is recent work Ji et al. (2024) that shows the usage of CREW for human-guied multi-agent Hide-and-Seck.
that CREW support. As shown in Figure. 2 on the right side,
Figure. Left: Example of adding objects to an environment CREW through drag-and-drop. Right: Example of scaling up the complexity of hide-and-seek search-and-rescue task.
Visual Observations Different visual observations create various challenges in visual embodinent learning for both humans Vander Heyden et al. (2017); Fishbein et al. (1972) and Al agents Bandini Zariffa (2000); Tarun et al. (209)). We provide various camera configurations support perceptual-motor research. As shown in Figure 3 for all our navigation and competitive tasks, we implemented visual observations from multiple views for the users' selection: top-down fully observable view, top-down accumulated partially
5

---

Published in Transactions on Machine Learning Research (11/2024)
Top-Down
Top-Down Partial
Top-Down Ego
First-Person
A
B
C
Figure. 3: CREW supports multiple tasks from single agent (A: Find Treasure) to multi-agent competitive setting (B: 1v1 Hide-and-Soek), and multi-agent collaborative and competitive setting (C: NvN Hide-and-Soek) We also show different camera views supported by CREW for perceptual-motor research.
observable view similar to SLAM in robotics Smith & Cheeseman 1986), a top-down egocentrie view, and first-person view.
Procedural Generation Learning robust, generalizable, and scalable AI agents requires diverse training environments. Procedural generation allows for the creation of wide range of environment patterns and terrain types. We provide randomized maze patterns where the grids are guaranteed to be connected (Figure. AA) and procedural-generated terrains for more complex simulations as in Figure. 4B.
A SEE
B
GS
Figure. 4: Environment generation in CREW. (A) Randomized maze. (B) Procedure generated terrains.

---

Published in Transactions on Machine Learning Research (11/2024)
3.3 Human and Agent Role Assignment
Humans and Al agents often have complementary strengths. For example, humans generally better at exploring and adapting Al agents are good at repelilive exploitation and precise caleulation. Naturally, team consists of humans and AT agents should have various roles to be effective. Different roles can also be assigned within Al agents to study multi-agent machine learning with heterogencous policies. To facilitate these experiments, we provide the role assignment feature CREW. "Player" allows humans directly control an agent. "Viewer" allows humans to observe and provide feedback signals as guidance an AI agent. "Server" role allows humans to host the entire task by monitoring it without direct participation. "AI Agent" assigns different learning policics to cach agent.
Positive
situations, while
Positive
Negative
Take Control
Negative
Figure. 5: (A) Discrete
scalar feedback. take control of the agent and teleoperate. (C) Con- tinuous scalar feedback: the human can hover the mouse over this window to provide per-step feedback.
(B) Option
Human Feedback Interface We provide a user interface to allow the Viewer role to provide direct feedback to Al agents shown in Figure. Scalar feedback is the most common foedback type used in human-guided RL Knox Stone (2009); Warnell et al. (2018) Xiao al. (2020). Conventionally, a human chooses to provide positive negative rating to the state-action pair of agent policy. Additionally, CREW offers an interface that allows humans to provide foodback that is continuous in value and time, enabling granular feedback information. Moreover, our interface also allows humans to directly take control over agents and perform teleoperation.
3.4 Multiplayer and Parallel Sessions
Enabling multi-human multi-agent sessions requires robust networking solutions (Figure. 6 Netcode jntt for game state synchronization, and Nakama makk as the networking server. In CREW, a server instance hosts the task, runs the simulation, and handles agent policy training, which can be executed powerful headless GPU server. Human participants can connect via client instances on less powerful machines, which display synchronized game states and collect human input. CREW is cross-platform, allowing participation from Linux, Windows or MacOS machines.
We use Unity
Match List
Match ID
a6ecfa71-c8708 0707--8877-731---00---2000
Active Players
Num Cilents
Match ID
bfd 222--22----22 --w-dd----3---00 8-885-8556
Active Playars
Num Clients
Match
New Match
Figure. 6: In CREW, it is possible to connect and join multiple instances of tasks. It is simple typing in IP address and selecting the task one wants to join.

---

Published in Transactions on Machine Learning Research (11/2024)
3.5 Human and Agent Data Collection
Data collection is at the of Human-AT leaming research. CREW includes a pipeline for thorough data collection on both the human side and Al agent side.
Human data Prior research in Human-Al AI teaming and human-human teaming has leveraged physiological
features in various ways. Thakur et al.] uses predicting human eye gaze as an auxiliary task for guiding imitation learning. Dikker et al.
2017) conducts brain synchrony with EEG in classroom multi-human
learning setup. Qin et al. (022 leverages pupil dynamics to predict the team performance in a multi-hunan virtual reality task. Heinisch et al. (2024) uses physiological data for human-robot interaction with service robots. Gordon et al.
20200) explored physiological synchrony in teaming. Koorathota et al. (2023) used gaze data to enhance vision transformers.
Xu et al. (2021) uses EEG data implicit human feedback for accelerating RL. We implemented the relevant features to support future research along these lines. Besides the feedback interfaces that collect feedback signals of multiple modalities and teleoperation actions, we also provide interfaces for collecting audio, eye gaze, pupillometry, electroencephalogram (EEG), and electrocardiogram (ECG) physiological responses as in Figure. with lime synchronizations from each machine with Lab Streaming Layer Kothe al. (2024).
Agent data including the policy weights, observations, actions, rewards, feedback reccived, and loss values at every time step can all be saved for further analysis. Users also have the option to enable experiment monitoring and logging by Weights & Biases Biewald (2020). As all of our tasks include vision-based settings, we also provide implementations of a set of vision encoder architectures.
Agent
EEG
Audio
ECG
A00
Gaze & Pupil
Mouse & Keyboard
50003
3040
1066
16/00
11200
17030
1790
370
Time
Figure. 7: Data collection in CREW. All human and agent data, including game states, feedback, mouse and keyboand board, audio, gaze and pupil data, EEG and ECG data are all streamed and synchronized through Lab Streaming Layer.
3.6 Designing Algorithms
A Algorithms research is crucial for Human-AI teaming. We designed the algorithm component of CREW to be extensible and accessible to the ML community. Algorithms are implemented in Python with PyTorch Paszke et al. (019 and TorchRI. Bou et al.] (2023) which part of the PyTTrch ecosystem, making it easy for researchers to design and deploy new algorithms. We provide implementations of state-of-the- art human-guided machine learning algorithms and strong reinforcement learning baselines. The API for communication between Unity instance and Python algorithm is implemented with MLAgents Juliani et al. (2018). All environments follow a uniform communication protocol for the observation and action data across lasks. Our modular design allows smooth switching between lasks and algorithms. Real human experiments are sime-consuminn and costly. To ease algorithm debugging, we implemented simulated human feedback providers as surrogates for real humans for complex tasks. These simulated feodback uses houristics basod on prior task knowledge and not available in novel tasks in the real world; hence, they should be used solely debugging tools, not as replacements for real human evaluations.
8

---

Published in Transactions on Machine Learning Research (11/2024)
Table. 3: Cognitive Tests Specifications
Tests
Rule
Seare Metric
Align ball the left side of the screen with targtt on the right side as accurately as possible within five scconds.
Negutive average of the distances between ball and target's horizoutal positions across trinls.
Fge Aligument (199) Scharre 2014
Click the screen as quickly as possible when it changes from yellow to green.
Reflex (reflese) Does 2014
Negutive average of the response times.
Obeerve red ball moving in au unknown but fixed pattern for Ss. When the hall patsess, predict the
Negative average of the distances between the ball's actual predicted posiliona acrues trials.
Theory of Behavtor (theory' Chen al.
20215
ball's position one second after is resumrss moving.
Ideucify the piece amoug thrre similar pieces that connot be rolated to match the targel piecs.
Mental Rotacion (potation)
Shepard Metzler
1971
Accuracy of the subject's idrutifictiious across all trials.
Mental Fitling (fiaees) Shepard Metzler
1971
Edeucify the only piece among three similar pieces that cau with the largel piece.
Accuracy of the subject's identificaiions across all tridss
A video of ageal navigating mase with restriesed field view presented. Identify the from selection of thre similar anaree.
Spatial Mapping (spatial)
Berkowitz et al. 2021
Accuracy of the subject's idmutiãcctions across all triels.
3.7 Modular Pipeline Design for Quantifying Human Characteristics
Individual differences among humans can significantly affect their tearnnng with Al agents. To support research along this line, CREW supports set of cognitive tests to quantify these differences shown in Table. 3 We provide modular and convenient pipeline (Figure. for executing cognitive tests and Human-Al experiments. The framework integrates various media files (e.g., instruction videos or pictures), inter-trial intervals, executable Python scripts, and Unity builds, ensuring a smooth and effective workflow. The pipeline requires minimal effort from researchers during proctoring, as a single click initiates the sequential execution of experiments. The pipeline allows restart from interruption points.
X
0.20x
S BV
Human-Al Experiments
B
Cognitive Teots
4 Benchmarking Study
Figure. 8: Modular pipeline for experi- ment setup. Experimenters can freely select and organize the order of cogni- tive tests and Human-AI experiments with CREW's deployment pipeline.
As an example of running experiments with CREW, we bench- mark a state-of-the-art real-time human-guided RL framework, Deep TAMER Warnell et al. (2018), along with strong RL baselines. In the original Deep TAMER, the framework was only tested on Atari Bowling with 9 human subjects. With CREW, this is the first time it is possible to systematically conduct human-guided RI. benchmarking across multiple environments on a larger population. We summarize our findings as well as insights on the scalability of the framework in this section. We also discuss the relationship between human characteristics and guided agent performance.
4.1 Experiment Setup
Tasks We selected 3 singlo-player games: Bowling, Find Treasure, and 1v1 Hido-and-Seck for this benchmark. For Find Treasure and Hide-and-Seek, each episode has a time limit of 15 seconds. All algorithms directly Iearn from visual inputs with the top-down accumulatod partially observable view.
Human Trainers We recruited 50 human subjects for the experiments under the approval of the Institutional Review Board. We highlight that experiment is the largest-scale study so far on real-time human guided AI training. For every human subject, the experiment time is approximatcly 40 minutes without interruptions. The experiment starts with cognitive tests (10 minutes) and is followed by the human guiding the agent using the c-Deep TAMER (which will be discussed in the next section) framework (30 minutes). The order of the cognitive tests Eye Alignment, Reflex, Theory of Behavior, Mental Rotation, Mental Fitting, and Spatial Mapping. There are detailed instruction videos for each test before the test starts. As for the human guiding agent component, each human subject guides a single agent to play 3 tasks for total of 30 minutes(5
9

---

Published in Transactions on Machine Learning Research (11/2024)
minutes for Bowling, 10 minutes for Find Treasure, and 10 minutes for 1v1 Hide-and-Seek). Before each task, there is detailed instruction video that describes the rule of the task and how human subjects can give feedback to the agent. Benefiting from CREW's unique feature to host experiments with parallel sessions, we were able to conduct all 50 experiments within one week. Prior to guiding the agents, the participants were asked to complete all our cognitive lests Table.
Deep TAMER Warnell et al. (2018) well-established baseline that leverages human feedback as myopic value functions. During training, human trainer provides positive negative discrete feedback based on the agent's behavior. This feedback is assigned to relevant state-action pairs through credit assignment window. A neural network is trained to predict human feedback given state-action pairs, and the policy chooses actions that maximize this predicted feedback. Originally, Deep TAMER was limited to discrete actions. We extend it to continuous action spaces using an actor-critic framework, termed as c-Deep TAMER, specified in Alg [ For bowling, the agent trained for minutes, and for minutes Find Treasure and Hide-and-Sock.
Algorithm The c-Deep TAMER algorithm.
Require: initial policy parameters 0, human feedback model H parameters é, empty replay buffer D, step size ? buffer updale interval b, credit assigmment window w
Init: j = =0
1: for i 1,2, do
observe state 8, time t
execute action
clip(Ao(s:) + aLow, alligh), where N
4: X; = as, t; t++1)
if new feedback y then
6:
j=j+1
y; =y
D, (x, wxx +0}
8:
D DDUD,
update H by one step of gradient descent using 10:
Σ
[[H((s, 5/12
Dj
(x,))E))
11:
update A by step of gradient ascent using
V [Dj]
Σ (x.y)E
Ho(s;, Ao(s:))
update larget networks
13:
k == le
14:
end if
15:
If mod(i,6)==) and 2> * 0 then
16:
Randomly sample a batch of transitions, B = {(s, a, y)} from D
17:
update by step of gradient descent using
V
Σ (xyjEB
[Ho((,, - - y|1a
18:
update A by step of gradient ascent using
V
Σ (xy)eB
Ha Ao(s:))
update target networks
20:
k = k
end if
22: end for
10

---

Published in Transactions on Machine Learning Research (11/2024)
Baseline RL We selected two state-of-the-art RL algorithms: Deep Deterministic Policy Gradient (DDPG) Lillicrp et al. 2015 and Soft Actor-Critic (SAC) Haarnoja et al. (2018).
Heuristic feedback We also evaluate simulated feedback-guided RL. We simply add the feedback signals as additional dense rewards to the environment reward. DDPG is selected as the backbone RL algorithm as the state-oftthe-art transitioned from SAC to DDPG Yarats et al. (2021) in visual control tasks. The hyperparameterss for the heuristic baseline is set to the same as DDPG baseline.
Evaluation We checkpoint model weights every 2 minutes and evaluate unseen test environments. For bowling, every checkpoint is evaluated for 1 game (10 rolls). For Find Treasure and 1v1 Hide-and-Seck, the checkpoints are evaluated for 100 episodes.
Processing Inputs All tasks in our experiments directly learn from pixels where humans and Al agents share the same input modality. The frames rendered from the environment are first resized to 100x100 pixels. For Find Treasure and iv1 Hide-and-Seck, we stack the past frames as input include the history information. We then apply a simple random shift to the frame stack, as it has been shown to be an effective augmentation strategy for visual reinforcement learning. The shifting factors along the height and width are uniformly sampled from [0, 0.08).
Model architecture We use a 3-layer CNN as the vision encoder, each having 64 channels and followed by batch normalization and ReLU activation function. All actor-critic frameworks use a 3-layer MLP with 256 neurons in each layer for both the actor and critic network.
Computational Resources All human subject experiments were conducted desktops with one NVIDIA RTX 4080 GPU. All evaluations RTX 3090 Ti. We note that used larger GPU servers due to the need for parallel evaluation resulting from the large scale of our study, but this is not requirement to run CREW.
run hcadless server with 8 NVIDIA RTX A6000 and NVIDIA
4.2 Results
Agent training performance We hypothesize that subjects with higher cognitive tests can load to higher- performing agents. Therefore, we show the agent performances guided by the 15 subjects who scored the highest in our cognitive tests side by side with the performance of all 50 subjects in Figure. As shown in the results, the agents guided by the top 15 subjects exhibit higher performance than the overall average. This is most prominent in Find Treasure, where c-Deep TAMER (blue curve) is significantly higher in the Top 15 plot than the All 50 plot. As shown in Figure. TO the correlation between cognitive test scores and guided Al performance is positive for all tasks, where Find Treasure had a strong statistical significance (**, p < 0.01). We do not draw a general conclusion from these early explorations. We hope to provide a useful metric and experiment to study individual differences for future work, exemplified by our benchmarking results. We highlight our concurrent work Zhang et al. (2024) to use them in human-guided RL evaluation.
Dowling (Tpp 151
Dowlngg [N] 50)
Find Treasue (Top 151
Rho Treasure (AI] 5D)
1Ide and Seek: Ivl CTop 15)
Hide and Seek Iv1 (AI) 50)
o-Dwep TAMER
c-Deso TAMER BBRS
pdeep TAMER
c-weo TAMER
pCeep TAMER
<-Deep "AMER
SAC
SAL
03
Succoss Roto (96)
Successs 196)
Score
32
32
20-
2.5
3.0
255
3.0
0.0
Time
Figure. 9: The average result of the human subjects with the top 15 cognitive test scores and the average result across all 50 subjects. We also find that e-Deep TAMER has difficulties scaling to more complex tasks.
11

---

Published in Transactions on Machine Learning Research (11/2024)
In particular for the top 15 subjects, the simple bowling task, c-Deep TAMER surpasses RL baselines by an average of 10 scores given the same training time. On Find Treasure, heuristic feedback achieved the highest performance as expected, showing the upper bound performance with accurate and non-delayed dense rewards. c-Deep TAMER also demonstrated strong performance with faster learning trends than RL basclines. On Ivl Hide-and-Seek, c-Deep TAMER performed similarly to RT baselines, suggesting that c-Deep TAMER has difficulty scaling tasks with higher complexity. Similar conclusions still hold for all 50 subjects. We also include a comparison between the top and bottom 25 cognitive test scorers in Appendix 6.5
Analysis of Individual Differences Due to the cognitive test feature and emphasis on Human-AI teaming, we can deepen our understanding of how individual human differences can affect the perfor- mance of human-guided agents. We calculated the correlation between human subjects' cognitive test scores and c-Deep TAMER training results in Fig- ure. 10 The cognitive test scores are normalized by the mcan and variance over the subjects through and-Seek
Bowling
-fn.s.)
-(n.s.)
+fn.s.)
+(n.s.)
-nn.s.)
+(n.s.)
Find Treasure
+In.s.]
+(-)
+(*
+(*)
+(n.s.)
+(-*)
Hide-
+In.s.1
-(n.s.)
-(n.s.)
+(n.s.)
+(+)
+hn.s.)
z-score. Following Tukey 1977), we remove outliers with scores 1.5x Interquartile Range above the third quartile or below the first quartile are removed. We found that non-outlier subjects performed equally well the spatial mapping test we do not include this for correlation analysis. The heatmap shows the sign of correlation coefficients from linear regression and the statistical significance (+).
Total
+nn.s.]
+(n.s.)
+(*]
+(*)
+nn.s.)
+(**)
Eye
Theory
Rotation
Fitness
Reflex
Overall
Correlation between cognitive test scores and c-Deep TAMER training performance. The darker the color, the more statistically significant the correla- tion. "+" or "_" mcans positive or negative correlation. Overall, the total score (i.e., the sum of three game scores) has significant positive correlations with rota- tion, fitness, and overall score. This likely a result of the strong contribution of Find Treasure since other tasks only show positive correlations with certain cog- nitive scores but not statistically significant.
Figure. 10:
The Find Treasure score has a significant positive correlation with theory of behavior, mental rota- tion, and fitness scores, suggesting that subjects who performed better on these tests also trained better agents. This is likely due to the need for good spa- tial reasoning and future behavior estimation in Find Treasure. The lvl Hide-and-Sen nificant positive corrclation with reflex score, likely
score shows a sig-
due to the rapid changes in environment and lask dynamics the complex adversarial setting. Subjects with faster reaction speeds provided timely foedback align with relevant state-action pairs. Overall, the total score (i.e., the sum of three game scores) has significant positive correlations with rotation, fitness, and overall score, suggesting that these cognitive skills are most relevant to the performance of c-Deep TAMER-guidee agents.
5 Conclusion, Limitation, and Future Work
We introduce CREW for facilitating Human-Al teaming research from diverse human and machine learning scientific communities. CREW offers extensible environment design, enables real-time human-AI communi- cation, supports hybrid Human-AT teaming, parallel sessions, multimodal feedback, and physiological data collection, and fcatures ML community-fricndll algorithm design. We also provide a set of built-in tasks and baseline algorithm implementations. Using CREW, we benchmarked state-of-the-art human-guided RL algorithm against baseline methods involving 50 human subjects and provided insights into the relationship between individual human differences and agent-guiding performance.
CREW is still in the early efforts among several critical aspects. Future work will explore building more diverse and challenging tasks, including multiplayer tasks with complex strategies and robotics environments requiring an understanding of physics. While we have only benchmarked several algorithms, we hope CREW can help benchmark many existing algorithms that were not fully open-sourced in unified environment. Finally, more supports human physiological data processing and analysis shall be investigated and supported in CREW.
12

---

Published in Transactions on Machine Learning Research (11/2024)
References
Nakama.
Unity netcode. URL attpss///ailyy.cc///ppddccct/sss
Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, and Shin-ichi Maeda. Dqn-tamer: Human-in- the-loop reinforcement learning with intractable feedback. ar preprint urXiv: 11748, 2018.
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint or Xix: 04257, 2019.
Shahin Atakishiyev, Mohammad Salameh, Hengshuai Yao, and Randy Goebel. Explainable artificial intelli- gence for autonomous driving: A comprehensive overview and field guide for future research directions. arXiv preprint 2xtu:2112211166 2021.
Andrea Bandini and José Zarilla. Analysis of the hands in egocentric vision: A survey. IEEE Iransactions on pattern analysis and machine intelligence, 45(6):6846 6866, 2020.
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay, Deriving machine attention from human rationales. Xiv preprint orx:i:1806 180000988.977 2018
David Bau, Hendrik Strobelt, William Peobles, Jonas Wulff, Bolci Zhou, Jun-Yan Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.072: 2020.
Michal Berkowitz, Andri Gerber, Christian M Thurn Beatrix Emo, Christoph Hoelscher, and Elsbeth Stern. Spatial abilities for architecture: Cross sectional and longitudinal assessment with novel and existing spatial ability tests. Frontiers in psychology, 11:609363, 2021.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL htttp:://www.wndb... available from wandb.com.
Software
Christopher Boes. The history of examination of reflexes. Journal of neurology, 261 (12):2264 2274, 2014.
Albert. Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision-making library for pytorch. arXiv preprint Xix::2300 00577, 2023.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Grilfiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Boyuan Chen, Shuran Song, Hod Lipson, and Carl Vondrick. Visual hide and seek. In Artificial Life Conference Proceedings 32, pp. 645-655. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info.... .... 2020.
Boyuan Chen, Yuhang Hu, Robert Kwiatkowski, Shuran Song, and Hod Lipson. Visual perspective taking for opponent behavior modeling. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13678-13685. IEEE, 2021a.
Boyuan Chen, Carl Vondrick, and Hod Lipson. Visual behavior modelling for robotic theory of mind. Scientific Reports, (1(1):224, 2021b.
Soumyadeb Chowdhury, Pawan Budhwar, Prasanta Kumar Dey, Sian Joel- Edgar, and Amelie Abadie. Ai-employee collaboration and business performance: Integrating knowledge-based view, socio-technical systems and organisational socialisation framework. Journal of Business Research, 144:31 49, 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Suzanne Dikker, Lu Wan, Ido Davidesco, Lisa Kaggen, Matthias Oostrik, James McClintock Jess Rowland, Georgios Michalareas, Jay J Van Bavel, Mingzhou Ding, et al. Brain-to-brain synchrony tracks real-world dynamic group interactions in the classroom. Current biology, 27(9):1375 1380, 2017.
Neta Ezer, Sylvain Bruni, Yang Cai, Sam J Hepenstal, Christopher A Miller, and Dylan D Schmorrow. Trust engineering for human-ai leams. In Proceedings of the haman factors and eryonomics sociely anmual meeting, volume 63, pp. 322 326. SAGE Publications Sage CA Los Angeles, CA, 2019.
Harold D Fishbein, Susan Lewis, and Karen Keiffer Children's understanding of spatial relations: Coordination of perspectives. Developmental psychology, 7(1):21, 1972.
Pedro Freire, Adamn Gleave, Sam Toyer, and Stuart Russell. Derail: Diagnostic environments for reward and imitation learning. arXiv preprint arxiv:2012 2. 2.0100111.1333. 0656, 2020.
Ilanit Gordon, Avi Gilboa, Shai Cohen, Nir Milstein, Nir Haimovich, Shay Pinhasi, and Shahar Siegman. Physiological and behavioral synchrony predict group cohesion and performance. Scienlific reports, 10(1): 8484, 2020.
Ben Green and Yiling Chen. The principles and limits of algorithm-in-the--lop decision making. Proceedings of the ACM Stuaan-Compull Interaction, 3(CSCW):1 24, 2019.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861 1870. PMLR, 2018.
Judith S Heinisch, Jérôme Kirchhoff, Philip Busch, Janine Wendt, Oskar von Stryk, and Klaus David. Physiological data for alfective computing in hri with anthropomorphic service robots: the affect-hri data set. Scientific Data, 11(1):333, 2024.
Patrick Hemmer, Monika Westphal, Max Schemmer, Scbastian Vetter, Michael Vössing, and Gerhard Satzger. Human-ai collaboration: The elfect of delegation on human lask performance and task satislaction. Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 453 463, 2023.
Katharine Henry, Rachel Kornfield, Anirudh Sridharan, Robert C Linton, Catherine Groh, Tony Wang, Albert Wu, Bilge Mutlu, and Suchi Saria. Human machine teaming key ai adoption: clinicians' experiences with deployed machine learning system. NPJ digital medicine, 5(1):97, 2022.
Jie Huang, Jiangshan Hao, Rongshun Juan, Randy Gomez, Keisuke Nakamura, and Guangliang Li. Gan-based interactive reinforcement learning from demonstration and human evaluative feedback. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 4991-4998. IEEE, 2023.
Borja Tbarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Zhengran Ji, Lingyu Zhang, Paul Sajda, and Boyuan Chen. Enabling multi-robot collaboration from single- human guidance. arXiv preprint arxix:2409 19831, 2024.
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. Unity: A general platform for intelligent agents. arXie preprint Xr0: 02667, 2018.
Richard Klimoski and Susan Mohammed. Team mental model: Construct or metaphor? Journal of management, 20(2):403 437, 1994.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Sharath Koorathota, Nikolas Papadopoulos, Jia Li Ma, Shruti Kumar, Xiaoxiao Sun, Arunesh Mittal, Patrick Adelman, and Paul Sajda. Fixating on attention: Integrating human eye tracking into vision transformers. arXiv preprint arXic:2308.1188 2023.
Christian Kothe, Seyed Yahya Shirazi, Tristan Stenner, David Medine, Chadwick Boulay, Matthew I Crivich, Tim Mullen, Arnaud Delorme, and Scott Makeig. The lab streaming layer for synchronized multimodal recording, bioRriv, pp. 2024-02, 2024.
Nitin Kumar and Anuj Jain. A deep learning based model to assist blind people in their navigation. J. Inf. Technol. Educ. Innov. Pract., 21:95-114 114, 2022.
Piyawat Lertvittayakumijon, Lucia Specia, and Francesca Toni. FIND: human-in--the-lopp debugging deep text classifiers. CoRR, abs/ abss/2010.04900 2020. URL http:ss//aratv.orr////////////// 04987
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint MKK::150900097.. 2015.
James MacGlashan, Michael L Littman, David L Roberts, Robert Loftin, Bei Peng, and Matthew E Taylor. Convergent actor critic by humans. In International Conference on Intelligent Robots and Systems, 2016.
Yannick Metz, David Lindner, Raphaël Baur, Daniel Keim, and Mennatallah El-Assady. Rlhf-blender: A configurable interactive interface for learning from diverse human feedback. arXiv preprint Xrxvv:2308 04332, 2023.
Mark A Neerinex, Jasper der Waa, Frank Kaptein, and Jurriaan van Diggelen. Using perceptual and cognitive explanations for enhanced human-agent team performance. In Engineering Psychology and Cogmitive Ergonomics: 15th International Conference, EPCE 2018, Held as Part of HC1 International 2018, Las Vegus, NV, USA, July 15-20, 2018, Proceedings 15, pp. 204 214. Springer, 2018.
Illah R Nourbakhsh, Katia Sycara, Mary Koes, Mark Yong, Michael Lewis, and Steve Burion. Human-robot teaming for search and rescue. IEEE Pervusive Compuling, 4(1):72 79, 2005.
Hyacinth S Nwana. Intelligent tutoring systems: an overview. Artificial Intelligence Review, 4(4):251 277, 1990.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in neurol information processing systems, 35:27730 27744, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advonces in neural information processing systems, 32, 2019.
Armin Danesh Pazho, Christopher Neff, Ghazal Alinezhad Noghre, Babak Rahimi Ardabili, Shanle Yao, Mohammadreza Baharani, and Hamed Tabkhi, Ancilia: Scalable intelligent video surveillance for the artificial intelligence of things. IEEE Internet of Things Journal, 2023.
Michael Pflanzer, Zachary Traylor, Joseph B Lyons, Veljko Dubljević, and Chang S Nam. Ethics in human ai teaming: principles and perspoctives. AT and Ethics, 3(3):917-983) 2023.
Yinuo Qin, Weijia Zhang, Richard Lee, Xiaoxiao Sun, and Paul Sajda. Predictive power of pupil dynamics in a tcam based virtual reality task. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops ( (VRW), pp. 592 593. IEEE, 2022.
Eduardo Salas, Denise L Reyes, and Susan H McDaniel. The science of teamwork: Progross, reflections, and the road ahead. American Psychologist, 73(4:5993, 2018.
D Scharre. Sage: A test to detect signs of alzheimer's and dementia. The Ohio State University Werner Medical Center, 2014.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Isaac Sheidlower, Elaine Schaertl Short, and Allison Moore. Environment guided interactive reinforcement learning: Learning from binary feedback in high-dimensional robot task environments. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 1726 1728, 2022.
Roger N Shepard and Jacqueline Metzler. Mental rotation of three-dimennional objects. Science, 171(3972): 701 703, 1971.
Randall C Smith and Peter Cheeseman. On the representation and estimation of spatial uncertainty, The international journal of Robotics Research, 5(4):56 68, 1986.
Bernd Carsten Stahl, Andreas Andreou, Philip Brey, Tally Hatzakis, Alexey Kirichenko, Kevin Macnish, S Laulhé Shaelou, Andrew Patel, Mark Ryan, and David Wright. Artificial intelligence for human Hourishing-beyone principles for machine learning. Journal of Business Research, 124:374-388, 2021.
Aneesh P Tarun, Nauman M Baig, Jack Chang, Rabia Tanvir, Sumaiyah Shihipar, and Ali Mazalek. Third eye: Exploring the affordances of third-person view in telepresence robots. In Social Robotics: 11th International Conference, ICSR 2019, Madrid, Spain, November 26-29, 2019, Proceedings 11, pp. 707-716. Springer, 2019.
Ravi Kumar Thakur, MD Sunbeam, Vinicius G Goecks, Ellen Novoseller, Ritwik Bera, Vernon Lawhern, Greg Gremillion, John Valasek, and Nicholas R Waytowich. Imitation learning with human eye gaze via multi-objective prediction.
Mark Towers, Jordan K. Terry Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL
https:://eenddo.orgge 81100////0///0
John W Tukey. Exploratory data analysis. Reading/ Addison- Wesley, 1977.
USC Institute for Creative Technologies. Rapid integration & development environment (ride), 2024. URL
https://ride wttrriieeeiittuu.. iiciiiitusc.. usc.uusc.ddu/ Accessed: 2024-05-27.
Karin M Vander Heyden, Mariette Huizinga, Maartje EJ Raijmakers, and Jelle Jolles. Children's 's represen- tations of another person's 's spatial perspective: Different strategies for different viewpoints? Journal of experimental child psychology, 153:57-73, 2017.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vozhnevets, Michelle Yco, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, el al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 100..708.00 04 782, 2017.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dinensional state spaces. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. Perfection not required? human-ai partnerships in code translation. In 26th International Conference on Intelligent User Interfaces, pp. 402-412, 2021.
Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Poovendran. Fresh: Interactive reward shaping in high-dimensional state spaces using human feedback. arXiv preprint arXiu:2001.066 106178,,....1,, 2020.
Duo Xu, Mohit Agarwal, Ekansh Gupta, Faramarz Fekri, and Raghupathy Sivakumar. Accelerating reinforce- ment learning using eag-based implicit human leedback. Neurocomputing, 460:139 153, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXio preprint or 2in: 2021.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Chunyong Yin, Biao Zhou, Zhichao Yin, and Jin Wang. Local privacy protection classification based human-centric computing. Human-centric compuling and information sciences, 9(1 ((1):33, 2019.
Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. arXiv preprint arXix:240. 02423, 2024.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321-384, 2021.
Lingyu Zhang, Zhengran Ji, Nicholas Rt Waytowich, and Boyuan Chen. GUIDE: Real-time human-shaped agents. In The Thirty-eighth Annual Conference on Neurul Information Processing Systems, 2024. URL sttpss//openreewe
Jianlong Zhou and Fang Chen. Towards trustworthy human-ai teaming under uncertainty. In IJCAT 2019
workshop on erplainable AI (XAI), 2019.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fino-tuning language models from human preferences. arXiv preprint or Xix: 1909.099.000 08593, 2019.
17

---

Published in Transactions on Machine Learning Research (11/2024)
6
Appendix
6.1
Accessing CREW
Our fully open-sourced code base and detailed documentations be found at https: http://gttuub..oo.
@gueralroboticssaa//CRH.
6.2
Platform details
The environments of CREW is implemented using Unity 2021.3.2417 with packages Agents m.3.0-exp... Juliani et al. 2018). Netcode for GameObjects 1.3. Inet and Nakama Unity 3.6.0. nak Algorithms are developed with torchrl 0.3.0 Bou et al. (2023).
6.3
Hyperparameters
The hyperparameter settings for our experiments summarized Table.
Table. 4: Hyperparameters
c-Deep TAMER
DDPG
SAC
0.99
0.99
0.99
learning rale
1e-4
le-4
1e-4
max grad norm batch size
0.1
0.1
0.1
16
240
240
frames per batch
8
240
240
alpha init target entropy
0.1
-6.0
actor scale_ lb
le-4
# Q value nets target update polyak
2
2
0.995
0.995
0.995
actor exploration noise
N(0, 0.1)
(0, 0.1)
credit assignment window
0.22, 4), others 1]
6.4 Algorithm Statistics
In Table. 5 we report the average number of updates of cach algorithm. In the task that c-DeopTAMBR showed the most advantage, Find Treasure, c-Deep TAMER had relatively lower number of average updates than the RT. baselines.
Table. Average number of updates
Bowling
Find Treasure
Hide and Seek
c-Deep TAMER
1357.54
1114.72
1224.0
DDPG
600
1200
1200
SAC
600
1200
1200
6.5
Additional Individual Differences Results
6.6
Full Cognitive Test Analysis
We include the full results for the cognitive test and c-Deep TAMER score analysis. All linear regression plots including coefficients and p-values and summarized in Figure. 12
18

---

Published in Transactions on Machine Learning Research (11/2024)
Bowling (Bottem25)
Hide and Seek (Tpp2)) lidd and Seek Ivl (Bottom22)
C-Oess
C.Daup
DGP3 SNC
DGPG SBC Hourz::
DDPO
6AC Hour St.
62-
Heuttalc
Pacuraaic
30
62
Surcvese Rate (K6]
42
28-
23
10-
0
30
25
3/5
10.00
000
3.5
0.5
20.0
25
3.0
5.0
1.3
10.0
Time (01
Time (m18))
Figure. 11: Guided Al performance of the top and bottom 25 cognitive test scorers.
6.7 Instruction Video Script
General Introduction:
Welcome to the XXX Human AI Collaboration Experiment. Today, our session will start with preliminary cognitive and gaming proficiency tests to gauge your initial skills. Following this, we will delve into main experiments where you will interact with AI algorithms through series of engaging games. Our session will conclude with short survey to capture your feedback the experience. Your participation is invaluable in advancing understanding of human AI interactions. Thank you for joining today, and let's begin.
Cognitive Test Introduction:
Welcome to the Cognitive Test segment our experiment. this session, you will participate in five interactive games designed to assess various cognitive skills for about seven minutes. These tests will challenge your precision, reflexes, predictive abilities, problem-solving skills, and spatial awareness through engaging activities. Each test is brief, and you'll receive clear instructions before each one begins.
Between each trial of each game, there will be a three-second intertrial interval where the computer screen will turn white with a Gray cross in the middle. Please focus on the center of the cross as much as possible during this time. Let's get started see how you do.
Eye Alignment Instruction:
In this experiment, your goal is to align the ball positioned on the left side of the screen with the square on the right as accurately as possible. Each trial lasts for five seconds, and you'll have a total of six trials. Use your mouse to drag the ball during each trial. The time for each trail. As time goes down, the bar gets smaller and changes color from green to red.
the top center of the screen shows much time you have
Reflex Instruction:
In this experiment, after three-second countdown, the screen turns yellow. You must click as quickly as possible when the screen changes from yellow to green. Clicking during the yellow phase will result in a failure, and failing to click within two seconds after screen turns green will also fail. You'll have total of trials.
Theory of Behavior Instruction:
19

---

Published in Transactions on Machine Learning Research (11/2024)
In this experiment, watch the red ball moving for five seconds, then guess where it will be one second after it pauses. Click on the map to mark your prediction. You have only one chance to click. The closer you are, the higher your score. Remember, you only have two seconds to make your guess after the ball pauses. There will be trials in total. The time at the top left shows time for each trail's observation and prediction as time decreases. The bar shrinks and changes from green to red.
Mention Rotation and Mental Fitting Instruction:
In this experiment, you will need to answer 12 questions in total, each with only one correct answer Click the button to choose the option that you think best answers the question. For each question, you have seconds to view and respond. The time bar at the top right of the screen indicates how much time you have for each question. As time decreases, the shrinks and changes color from green to red.
Spatial Mapping Instruction:
In this experiment, you will need to answer six questions in total, each with only correct answer. Watch the video and click the button to choose the option that you think best answers the question. For each question, you are free to answer while the video is playing, and you will also have three extra seconds after the video is paused to answer the question. The time bar at the top right of the screen indicates how much time you have for each question. As time decreases, the shrinks and changes color from green red.
Human Guiding AI Introduction:
Welcome to the Human-Guiding AI section of the experiment. You will guide the AI to play three games for about 30 minutes in total in this section. You'll give feedback to the agents based on their performance in each game. Before starting each game, please enter your name the name box and click on the Connect button. match list will appear. Click the Match button in the list. It will turn brown when clicked. Then click the Join Match button to enter the match. Once you've joined the match, select the AI agent in the Active Clients panel at the top right corner of the screen by clicking it. The agent button will turn brown to indicate that you've selected it successfully. Now you can observe the agents and provide feedback. You'l1 constantly click on the positive negative buttons to inform the agent of its performance. Positive means it's doing well. Negative means it's doing poorly.
Bowling Instruction:
Each episode of this bowling game consists of 10 rolls. At the start of each roll, there are 10 pins positioned on the right side of the screen. The agent then launches ball in an attempt to knock down the pins, with the number of pins knocked down determining the score for that roll. The total score for the episode is calculated as the sum of the scores from all rolls.
rolls. Please guide the AI agent to maximize the total score across
Find Treasure Instruction:
In this game, the AI agent, represented by the green character, begins with a limited field of view, only able to perceive surroundings, while the rest of the map is obscured by shadow. As the agent navigates the map, areas it has visited become revealed within its field of view. Please guide the AI agent to locate the treasure and navigate to it, represented by the brown chest, quickly as possible.
1v1 Hidc-and-Seck Instruction:
In this game, the AI agent, represented by the red character, begins with a limited field of view, only able to perceive surroundings, while the rest of the map is obscured by
20

---

Published in Transactions on Machine Learning Research (11/2024)
shadow. As the agent navigates the map, areas it has visited become revealed within its field of view. Guide the AI agent to chase the hider, represented by the green character, and catch it as quickly as possible.
6.7.1 Compensation
We pay each human subject $20 for participation.
21

---

Published in Transactions on Machine Learning Research (11/2024)
(ilt pabker E/00
EAREP
Kap 11!
Sippl EXTT,
bet Bnd
240
959
19.0
Shoe 1.80 pubko E557
Skpt: paRke DE0R
Skpp: piokec
Suos LPI palle
Fidd
240
Pope 3.35
Sape:2 puRke:00
Saze: CAAE prsser
Duse produe
20
240
152
J0L0
2500
Fipe
Stpe 3,305
Syper poskee
Stipet Exaiee
raloco, badinruaas
rclo Malaaoo
Siye: p
Zie
Dea
Dee
Ceral ye Joaling sen
Osant WUT scew
Oseral JUMM so29
Dike
Figure. 12: Linear Regression Plots of Correlation Analysis
22


Reviews:
[{"review_id": "WP57qqUeD7", "reviewer": "Reviewer_o49T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For context, I had not previously read about human-AI teaming, though I'm of course familiar with the general concept. So some of my comments here might be a result of that ignorance.\n\nTo me the most interesting results were the results on training performance and individual differences. However, the statistics used in this section seemed somewhat arbitrary to me: maybe this is just a result of me not knowing the research field very well, but is it normal to remove datapoints that are more than 1.5x the IQR away from the mean (in section 4.2)? Also, it seems arbitrary to compare the top 15 to all trainers; why not compare top 25 to bottom 25? This seems especially dubious because of the fact that you use correlation coefficients as the metric in Figure 10; why switch statistical method like that?\n\nI am very confused about the evidence for the claim that \"Subjects scoring higher on cognitive tests are generally better at guiding RL\": in Figure 9, I don't see that effect except for c-Deep TAMER on Find Treasure. Figure 10 seems consistent with my interpretation here: overall cognitive ability only correlated significantly with agent performance on Find Treasure.\n\nI found the submission somewhat confusing to understand based on the abstract. For example, I didn't understand what kind of tasks the human-AI teams would be doing (game tasks? programming tasks? NLP tasks?). The abstract also doesn't mention the empirical results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a platform for research where humans and AIs collaborate to solve tasks in various game environments, which is useful for research into human-AI teaming. This platform has a variety of features, e.g. allowing multiplayer environments and human data collection. The paper also investigates whether humans who are better at cognitive tasks train AIs faster, and finds that they do. It also introduces a novel RL algorithm for training AIs with humans, that seems to be better than baselines in some settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please either remove the claim that \"Subjects scoring higher on cognitive tests are generally better at guiding RL\" or justify that that is true in more than c-Deep TAMER on Find Treasure. This is necessary for my recommendation for acceptance,\n\nPlease consider using more consistent statistical methods for the empirical results. This is not necessary but I recommend it.\n\nPlease consider rewriting the abstract to be clearer to more generalist ML researchers. This is not necessary.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "npWM351pP3", "reviewer": "Reviewer_F7Uw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Comprehensive Design: CREW is designed to be extensible and modular. It is also open-source and cross-platform.\n- Real-Time Interaction: The platform supports real-time communication between humans and AI agents, with real time feedback during training.\n- Data Collection: CREW collects a variety of data, including physiological signals.\n- Human Study: The authors conducted the largest study with 50 persons for real-time training, demonstrating the platform's practical usability.\n\nWeaknesses:\n- it is not clear if the benefits of c-Deep TAMER come from the higher number of network updates or the actual human feedback (the batch size is 15 times smaller than DDPG/SAC, which let me think the number of network updates is also 15 times bigger for c-Deep TAMER)\n- the experiment section should show the importance of the framework, I believe the real-time training is a key contribution but those experiments only show that offline training is more beneficial (\"heuristic\" vs \"c-Deep TAMER\" in Figure 9.)\n- the authors claims that other works support only a single task, but the 4 proposed tasks in CREW are also a bit limited and do not really show the need for real-time human feedback\n- CREW seems to offer visual and low level observations but nothing for textual information\n- 5 to 10 minutes of training seems to be very small for RL training", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces CREW, a platform designed to facilitate Human-AI teaming research. The platform aims to address several challenges, including the limited number of tasks, real-time human feedback during training, hybrid Human-AI teaming support, parallel sessions for scalable experiments, and comprehensive human and agent data collection. The authors demonstrate the platform's capabilities through a benchmarking study involving 50 human subjects.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical requested changes:\n- please highlight somewhere the number of network/policy updates for each algorithm, if those are different, you should also report the results of the baselines with equals number of updates\n- the \"heuristic\" baseline should be better explained as well as why it is expected to has the highest performance, I understood it as an online DDPG with offline human feedbacks/reward function (unlike \"c-Deep TAMER\" that would receive the feedback incrementally)\n- the \"heuristic\" baseline is missing on Bowling\n- except if I misunderstood, the \"heuristic\" baseline should not be the same across \"top 15\" and \"all 50\", I assume what is displayed now is \"all 50\"\n\nNon critical ones:\n- it would be nice to clearly state that each trained RL agent is only receiving the feedback of a single human\n- what are the hyperparameters for the heuristic DDPG?\n- please mention how long the humans have to wait for the update step?\n- \"this positive correlation between cognitive test scores and guided AI performance holds true on all tasks, as shown in Figure. 9\" I failed to see that on Bowling (except for the checkpoint at 1 min. all the others look like the opposite)\n- it would also be interesting to know the upper bound performance of SAC and DDPG if we trained them longer", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ABS1WGZvh9", "reviewer": "Reviewer_5saz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe work is both interesting and valuable to the community. The limitations of previously developed platforms are clearly articulated, and the authors provide a thorough yet concise description of the strengths of their novel platform in relation to the existing literature.\n\nSeveral of the proposed features are particularly noteworthy for the machine learning community, especially in the area of human-assisted algorithms.\n\nOverall, the paper is well-written and easy to follow.\n\n\n# Weaknesses\nThe work defines CREW as a Human-AI teaming research platform. Given this definition, I would have expected a platform that facilitates true collaboration between human and AI agents, where each plays distinct roles to tackle complex tasks that would otherwise be extremely difficult to address.\n\nUnfortunately, I don't believe this work fully lives up to that premise. Despite the authors' efforts to emphasize Hybrid Human-AI teaming, the tasks and study presented in the paper do not clearly demonstrate this collaborative aspect. Instead, the platform seems primarily geared toward scaling the collection of human behavioral and physiological data for human studies and improving the integration of human feedback and prior knowledge into machine learning algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces CREW, a platform to facilitate Human-AI teaming research. The authors design a novel platform for human-AI collaboration which tackles some of the limitations of the existing platforms. Furthermore, the paper contains a few developed environments and a benchmarking study.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The introductory section places significant emphasis on the teaming aspect, yet the rest of the paper seems more focused on human-guided learning for AI algorithms. I would like to ask the authors for clarification on this choice. While it's clear to me how human data and feedback can be beneficial in the AI space, I find it less clear how AI can contribute to human studies. Specifically, what value does this platform offer to neuroscientists or other researchers in human-centered fields?\n\n2. The paper provides almost no information about the hardware requirements necessary for successfully using the platform. Figure 7 depicts a setup for human data collection, but I believe additional details about the hardware used to gather these data would help other labs adopt the platform more easily. I also have a few questions regarding the computational resources mentioned in your experiments. The statement, \"All evaluations were run on a headless server with 8 × NVIDIA RTX A6000,\" is somewhat confusing. What exactly do you mean by \"evaluation\"? What resources were required to train the reinforcement learning algorithms? Given that your networks do not appear large enough to justify the use of 8 GPUs, this point needs further clarification.\n\n3. I quickly went through your anonymized code base. I believe that in these types of works, the code is very important. The code is not well documented and does not seem ready for publication.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the discussions with reviewers, the authors have agreed to applying numerous changes, many of which are quite important. Some of these changes involve more accurate and careful claims surrounding e.g. statistical significance, and fixing cases where the incorrect figures were referenced. Therefore, I request the authors to very carefully make sure that they apply all the changes they have agreed to in their discussions with the reviewers.

Aside from that, the authors should also adjust any references to anonymity (e.g., for the github repo) and update them to no longer be anonymised for the camera-ready version.
Decision: Accept (poster)