SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Adarsh Pyarelal<sup>1</sup>, Eric Duong<sup>2</sup>, Caleb Jones Shibu<sup>2</sup>, Paulo Soares<sup>2</sup>, Savannah Boyd<sup>3</sup>, Payal Khosla<sup>4</sup>, Valeria Pfeifer<sup>3</sup>, Diheng Zhang<sup>3</sup>, Eric Andrews<sup>3</sup>, Rick Champlin<sup>1</sup>, Vincent Raymond<sup>5</sup>, Meghavarshini Krishnaswamy<sup>6</sup>, Clayton Morrison<sup>1</sup>, Emily Butler<sup>4</sup>, Kobus Barnard<sup>2</sup>
$^{1}$ School of Information,  $^{2}$ Department of Computer Science,  $^{3}$ Department of Psychology,  $^{4}$ Norton School of Human Ecology,  $^{5}$ Lum AI,  $^{6}$ Department of Linguistics University of Arizona adarsh@arizona.edu https://tomcat.ivilab.org
We present a rich, multimodal dataset consisting of data from 40 teams of three humans conducting simulated urban search-and-rescue (SAR) missions in a Minecraft-based testbed, collected for the Theory of Mind-based Cognitive Architecture for Teams (ToMCAT) project. Modalities include two kinds of brain scan data—functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), as well as skin conductance, heart rate, eye tracking, face images, spoken dialog audio data with automatic speech recognition (ASR) transcriptions, game screenshots, gameplay data, game performance data, demographic data, and self-report questionnaires. Each team undergoes up to six consecutive phases: three behavioral tasks, one mission training session, and two collaborative SAR missions. This dataset will support studying a large variety of research questions on topics including teamwork, coordination, plan recognition, affective computing, physiological linkage, entrainment, and dialog understanding. We provide an initial public release of the de-identified data, along with analyses illustrating the utility of this dataset to both computer scientists and social scientists.
Teams of the future will increasingly involve humans and AI agents working together as trusted partners, leveraging their complementary skills to achieve shared goals. The efficacy of AI teammates will be enhanced if they are able to understand the beliefs, desires, and intentions of their human teammates, i.e., if they have a machine theory of mind (MToM) [1, 2]. However, this capability alone is not sufficient—they will need to understand the interpersonal dynamics between their human teammates, that is, they will need a machine theory of teams. A natural first step for constructing a (computational) theory of teams is to draw upon the vast amount of existing literature on teamwork in purely human teams. However, as Roberts et al. [3] note, significant additional work is needed to extend existing models of teamwork to human-machine teams.
We present the ToMCAT (Theory of Mind-based Cognitive Architecture for Teams) dataset—a rich, multimodal dataset developed to significantly advance our understanding of teaming in both purely human and hybrid human-machine teams. The dataset contains data from experiments in which teams of three humans (and optionally, an AI advisor) execute complex collaborative tasks—specifically,
urban search-and-rescue (USAR) missions—in a virtual Minecraft-based testbed [4]. However, we emphasize that the primary focus of this paper is not on the USAR missions themselves, but rather the complex social behaviors that the testbed is designed to elicit. Additionally, we instrument the participants with additional sensors that capture data via the following modalities: spoken communications, gaze, facial image captures, galvanic skin response (GSR), electroencephalography (EEG), and functional near-infrared spectroscopy (fNIRS), and have them perform a set of novel behavioral baseline tasks ( $\S 4$ ).
Main contributions. 1) A rich multi-person, multimodal dataset for teams collaborating on complex, time-constrained tasks, 2) Three structured behavioral baseline tasks designed to ground the physiological data, as well as serve as independent multi-person, multi-task, multimodal datasets; and 3) Exploratory analyses suggestive of the huge space of inquiry possible with this data.
This paper builds upon multiple lines of research related to human-machine teaming, synthetic task environments, and interpersonal coordination as evidenced by physiological linkage. The primary motivation for this dataset is to accelerate the development of effective artificial agent teammates with artificial social intelligence through a deeper understanding of purely human and human-machine teaming in cognitively complex, time-constrained scenarios.
Machine Theory of Mind Given the core role played by theory of mind (ToM) [5] in human social intelligence, there has been steadily increasing interest in developing artificial agents with MToM [1, 2, 6-22]. A significant portion of the literature on MToM, however, evaluates it in contexts that are either disembodied (e.g., image classification and purely language-based tasks [6, 12, 16]) or 'lightly-embodied' (e.g., text adventure games [13] and small 2D gridworlds [1, 2, 14, 18]). Voxel-based environments such as Minecraft [23] represent a natural step up in complexity.
Minecraft for AI research Minecraft is an open-world adventure game that is gaining popularity as an AI testbed due to its ability to support diverse tasks [24], modifiability, and large user base. Of the projects that use Minecraft for AI research, a large number use it as a testbed for reinforcement learning [25-27], or for training AI agents to perform tasks based on natural language commands [28-32]. However, following others [8, 9, 11, 15, 21, 33-39], we are interested in using Minecraft as a testbed for ToM and human-machine teaming—that is, rather than having AI agents execute tasks within a Minecraft-based environment, we have humans executing the tasks, with AI agents (if present) acting as passive observers or advisors.
USAR tasks in synthetic task environments The environment we use [4, 40] is an example of a synthetic task environment (STE) [41]—a medium-fidelity simulation environment that strikes a pragmatic balance between highly abstracted lab settings (more controlled, cheap, less likely to generalize to real-world situations) and high-fidelity simulations/real-world environments (less controlled, expensive, more representative of real-world scenarios). The USAR task was chosen due to its time-constrained, cognitively demanding nature, coupled with the potential for humans and robots to perform complementary team roles in real-world USAR scenarios [33, 42]. The use of Minecraft-based USAR STEs to study human-machine teaming is relatively well-established—they have been used for small-scale studies [33, 43] as well as large-scale datasets [38, 39] upon which numerous analyses have been performed [8, 10, 11, 15, 17, 21, 44].
The dataset most closely related to ours is the ASIST Study 3 dataset [39]. We use the same STE [4] and Minecraft USAR tasks [45]. However, our data differs from theirs in several ways, the most important of which is the inclusion of a number of additional sensing modalities in our experiments: gaze, facial image capture, EKG, GSR, EEG, and fNIRS (see § 2.1 for more details), which opens up a number of additional research opportunities.
First, since we are concerned with theory of mind, recording EEG and fNIRS signals provides us a way to ground inferences about cognitive and affective components of human mental state in rich data that reflects the actual underlying brain activity of the participants—in a sense, getting us closer to the 'ground truth'. Notably, the study of the affective component of human mental states (i.e., emotions) is conspicuously absent from existing works on MToM, despite affect playing a crucial role in human social interactions and decision making. Additionally, affect is reflected in other forms,
e.g. facial expressions and changes in heart rate, neither of which are represented in the ASIST Study 3 dataset. While we do not expect, say, USAR team members to wear fNIRS/EEG caps in the field, we believe that the modalities in this dataset will allow us to create mappings between surface-level indicators of affect and coordination (e.g., facial expressions, tone of voice) and deeper underlying affect and coordination as detected from fNIRS/EEG signals.
Second, physiological linkage (PL)—i.e., statistical association between the physiological markers of two or more people over time [46]—has been shown to be predictive of performance and attributes [47, 48]. Furthermore, PL may depend on the interaction of individual differences and context—e.g., differences in individual social skills and attachment styles have been found to be associated with qualitatively distinct forms of PL in competitive and collaborative contexts [49]. Therefore, PL may be a promising predictor and outcome of team dynamics. Through the addition of physiological sensing modalities, our dataset enables the study of PL in the context of human-machine teams.
Third, unlike in ASIST Study 3, the participants in our study perform a set of tasks designed to compare physiological changes and phonetic entrainment resulting from performing team tasks. Finally, instead of running multiple AI advisor experimental conditions, we only have one (the ToMCAT agent), resulting in a much larger amount of data for this single advisor, which will enable increased statistical power for analyses that are not focused on comparing the outcomes of interventions by different AI advisors (one of the primary goals of ASIST Study 3).
Open-access fNIRS datasets Notably, to the best of our knowledge, our dataset is also the largest open-access fNIRS dataset to date. It is approximately 13.5 times the size of the fNIRS2MW [50] dataset (  $1.5 \times$  more subjects and  $\approx 9 \times$  more fNIRS data per subject).
2.1 Physiological measures
We simultaneously record multiple subjects' neural activities (hyperscanning) during real-time social interactions [51]. Based on previous literature on team cognition and hyperscanning [52], we selected EEG, fNIRS, EKG, GSR, and gaze as our main physiological measurement modalities. These were collected for each participant for the duration of the group session (see § 3). This approach provides opportunities for data analysis at various time resolutions [52], optimal variable control by using data from different modalities for denoising, and higher-level feature selection/construction [53]. Details on the equipment and procedures for data acquisition and signal processing are provided in the appendices.
EEG EEG is a non-invasive measure of the scalp electrical activity generated from the cerebral cortex. It provides data on brain activity with temporal resolution on the order of a millisecond [54]. This high temporal resolution affords us opportunities for event-related (task-based) analysis (event-related potential, ERP), which has been widely adopted in cognitive science research on various topics including decision-making, emotion elicitation, and team cognition [55]. EEG hyperscanning has yielded fruitful results for human-human social interaction research [52]. Sinha et al. [51] found that inter-brain synchrony calculated from simultaneous EEG recordings of paired subjects was found to be significantly higher when the subjects were in a cooperative scenario compared to when they were in a competitive scenario. EEG has also been used to study human-machine teaming scenarios—e.g., Shayesteh, Ojha, and Jebelli [56] measured EEG signals from subjects performing a collaborative construction task with a virtual robot in an immersive environment, and found that a  $k$ -nearest neighbors model (kNN) trained on EEG signals was able to predict the human's level of trust in the robot with an accuracy of  $\approx 88\%$ .
fNIRS fNIRS is an non-invasive optical brain imaging technology that assesses the contrast between oxygenated and de-oxygenated hemoglobin in the cortex, and uses the hemodynamic fluctuation as an indirect measure of brain activity in targeted brain areas [52]. While fNIRS has a lower temporal resolution than EEG [57] (on the order of  $10\mathrm{ms}$ ), it is highly portable and less susceptible to motion, making it an increasingly popular modality for social interaction experimental settings [58]. For this study, we use optodes that mainly cover the frontal lobe area, based on previous research [52, 59, 60] that found that greater interpersonal brain synchronization occurs at the frontopolar area, indicating better coordination performance. Oxygenation changes in the prefrontal cortex (PFC) have also found to be related to performance on various individual cognitive tasks [61], including language translation and switching [62], verbal fluency [63], and mental manipulation [64].
(a) Participant layout
(b) Individual participant
Figure 1: Our experimental setup for data collection. Figure 1a shows the layout of the participants. The 'Lion', 'Tiger', and 'Leopard' stations are for regular participants, while the 'Cheetah' station is used for the experimenter that joins the group session for the competitive ping-pong task (§ 4.3). The 'Lion' and 'Tiger' stations are separated from the 'Leopard' and 'Cheetah' stations by a divider, in order to reduce audio cross-contamination between the participants' microphones. Figure 1b shows a more detailed view of an individual participant, who is instrumented to record EEG, fNIRS, GSR, EKG, and gaze data and in the midst of a Minecraft SAR mission.
EKG An electrocardiogram (EKG) measures heart activity over time and offers high temporal resolution. Common EKG signal derivatives include inter-beat interval (i.e., the time between heartbeats) and respiratory sinus arrhythmia (i.e., variability in heart rate due to breathing). These signals result from coordinated biological activity within a person and are commonly used to model coordination between people, as reflected by physiological linkage. Additionally, EKG data can be used to filter out systematic cardiac activity noise (1-1.5 Hz [65]) from fNIRS data.
GSR Galvanic skin response (GSR), is a measure of electrical conductivity on the surface of the skin [66]. Sweat gland activity varies unconsciously and automatically, peaking approximately 1-5 seconds after stimulus onset. GSR is commonly used in the study of emotion processes and teamwork. For example, studies have found that GSR activity is associated with team performance [67], mental effort [68], and self-reported emotion during team tasks [69]. For our study, we were interested in the peak amplitudes (i.e., change from stimulus onset to highest peak), which can be used to examine sweat gland activity within and between teammates following a stimulus presentation or during team-based tasks.
Gaze An eye tracker works by shining infrared light onto the eyes of a participant, creating reflections on the corneae that are then used to identify the locations of their pupils. By capturing their eye/pupil movements, the eye tracker software can infer the point of gaze (where the participant is looking) in real time [70, 71]. Eye-tracking is widely used in a variety of disciplines, including psychology [70], marketing [72], and UI/UX research [73]. Additionally, eye-tracking can provide event markers for other modalities such as EEG, fNIRS, and EKG.
The study was held at the University of Arizona. Individuals were deemed eligible if they were at least 18 years of age, read and spoke English, and did not have any major physical limitations that would interfere with completing tasks on a computer. Interested individuals contacted the research team via e-mail, text, or phone. Details on the ethical review and the recruitment process are provided in  $\S$  A.3 and Appendix B respectively. All participants were compensated with either an Amazon gift card or course credit. Participants were asked to complete a 30-minute individual 'pre-session' and a 3-hour 'group session'.
3.1 Pre-session
Participants started by completing the online consent form if they had not already done so. We then measured their heads to select an appropriately sized EEG/fNIRS cap for them.
Speech elicitation tasks We conducted two speech elicitation tasks, collecting speech data for each participant prior to their interaction with their teammates. This created a speech baseline for each player prior to their interactions with their teammates and was used to study phonetic entrainment between teammates during the course of the Minecraft missions. For more details, see Appendix K. Entrainment is a useful method for assessing the dynamics of a social interaction and levels of rapport [74, 75]. Prior work has found strong correlations between entrainment and success in group tasks [74, 76, 77].
Questionnaires The following questionnaires were administered: (i) a COVID-19 health screener, (ii) a survey that collected information about basic demographics (e.g., sex, racial background, household income, highest level of education), experience with video games, and health (e.g., speech/hearing, language, impairments, diagnoses, psychoactive medication, etc.), (iii) the Big Five Inventory-2 Short Form (BFI-2-SF) personality questionnaire [78], and (iv) the Attachment Style Questionnaire that assessed adult attachment [79].
3.2 Group session
When required, the participants interacted with each other through a keyboard and mouse located at each experimental station. Each participant was positioned in front of a computer monitor, with dividers used to increase physical separation between the participants (see Figure 1). If only two out of the three planned participants showed up to the group session, or if one of the participants dropped out in the midst of it, a confederate (i.e., a member of our research team) would step in to take their place. Out of the 1014 task instances for which data was supposed to be recorded for regular participants, 147 (i.e.,  $14.5\%$ ) had experimenters filling in for participants.
Baseline tasks Participants started by conducting a set of behavioral baseline tasks (§ 4).
Search-and-rescue missions Next, participants conducted the Minecraft-based SAR missions described by Huang et al. [45]. Each team conducted a 20-minute tutorial mission, followed by two 17-minute main missions: Saturn A and Saturn B. The tutorial mission consists of a series of tasks designed to familiarize participants with the game environment and their avatars' specific abilities. The first 2 minutes of the main missions were devoted to planning—participants were encouraged to discuss strategies and review good and bad practices they adopted in the previous mission. In the next 15 minutes, access to the building is unblocked and participants can effectively start to earn points, which happens after they find, treat and move victims of an in-game building collapse to assigned safe areas. A subset of the teams were advised by the ToMCAT AI agent [80], which was designed to improve team coordination by intervening on team communication.
Post-game survey After completing the Minecraft tasks, participants completed a brief post-game survey. They were asked to rate their emotions due to (i) their AI agent teammate, (ii) how the game went, and (iii) the other team members, on a scale of 'not at all' to 'a very large amount'. In addition, participants indicated their impression of the agent and other team members by sliding a bar between a pair of adjectives (e.g., intelligent-unintelligent, inexpert-expert, etc.). Lastly, participants responded on the extent to which they disagreed or agreed to general statements along with statements about other team members and the agent (e.g., "It seemed like my emotional reaction was wrong or incorrect because of the agent's response").
4 Baseline tasks
We collected rest state physiological data and conducted three behavioral baseline tasks to ground complex physiological signals expected in collaborative missions to simpler, well-studied settings. For example, we can map patterns in the Minecraft tasks to patterns of coordination established in a simple task. These tasks closely resemble well-established ones, facilitating connections with
existing research. Our approach extends prior studies by (i) addressing the underexplored area of emotion in hyperscanning [81] through affective behavioral tasks and (ii) expanding from the prevalent two-person paradigm to experiments involving three participants.
4.1 Rest state and finger tapping tasks
Rest state task This task was designed to collect baseline physiological data from the participants while they were in a resting state. This is consistent with standard neurophysiological research [82] and important for comparison purposes—by establishing a resting baseline, we can better understand an individual's functional baseline in the absence of exposure to the stimuli in the other tasks. In this task, participants sat quietly for 5 minutes without engaging in any activity, with their monitor displaying a countdown timer showing them the time remaining in the task.
Finger-tapping task This task was designed to record physiological data during team synchronization in a cooperative activity. It allows us to observe neural processes during synchronization. Results from Tognoli et al. [83] suggest certain neural correlates when participants are tasked with finger-tapping with and without visual cues from each other. Furthermore, hyperscanning [52] studies have shown an association between EEG signals and behaviors [81, 84]. Our finger-tapping task is a variant of the one proposed by Tognoli et al. [83].
4.2 Affective task
In this task, participants viewed a curated set of images (see Appendix L) designed to elicit various emotions. We employed Russell's valence-arousal scale [85]—a widely recognized tool in affective research—to quantitatively assess emotions. The task aims to collect physiological and emotional data for interpreting emotional experiences based on physiological responses in subsequent tasks. Prior studies indicate a connection between fNIRS, EEG, and autonomic functioning during the processing of emotions—specifically, PFC activation [86] and potential dual motive systems in the brain [87]. These findings align with literature on the PFC's role in memory, emotion regulation, and cognition [88-92]. The affective task includes 'individual' and 'team' affective subtasks.
Individual Affective Task For each image, the participant was shown the following sequence: (i) a black screen for one second, (ii) a  $+'$  icon at the center of the screen (guiding the participant's attention to the center) for 0.5 seconds, (iii) the image itself for 5 seconds. After viewing each image, participants had 20 seconds to rate the emotions they experienced during their observation. The rating screen presented a 5-point valence scale (-2 for upset to +2 for happy) and a 5-point arousal scale (-2 for 'calm' to +2 for 'excited'), with 0 denoting 'neutral'. These scales were adapted from the Self-Assessment Manikin (SAM) [93] pictorial rating—a non-verbal technique for gauging individual affect. Participants were prompted to register their emotional responses and submit them before the onset of the subsequent image.
Team Affective Task This task is similar to the individual affective task, except that participants viewed each image together instead of separately, after which they discussed their emotional experience and submitted a single rating representing their collective emotional experience in response to the image.
4.3 Ping-pong task
The primary objective of this task is to collect neurophysiological data when participants are engaged in competitive and cooperative scenarios. A range of tasks including card games, ping-pong, and music and rhythm synchronization exercises have been used in the hyperscanning literature—e.g., studies have found evidence of inter-brain synchronization among participants collectively perform a piece of music [94, 95] and in a cooperative task based on the Prisoner's Dilemma [96]. The ping-pong task is divided into two subtasks: competitive and cooperative.