SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $R^2$  of fNIRS for Ping Pong Cooperative linear regression: Mean 0.0915, Var 0.007
-  $R^2$  of fNIRS for Minecraft Saturn A linear regression: Mean 0.069, Var 0.010
-  $R^2$  of fNIRS for Minecraft Saturn B linear regression: Mean 0.059, Var 0.006
-  $R^2$  of EEG for Ping Pong Cooperative linear regression: Mean 0.060, Var 0.001
-  $R^2$  of EEG for Minecraft Saturn A linear regression: Mean 0.045, Var 0.002
-  $R^2$  of EEG for Minecraft Saturn B linear regression: Mean 0.058, Var 0.008
✓ The average runtime for each result. To synchronize the data with the computing infrastructure below, the synchronization of physio data and task data took one hour to complete. Then, from the output of the physio synchronization program, the physio correlation analysis took under 10 minutes to complete.
$\checkmark$  The computing infrastructure used. The models were trained on a server with dual AMD EPYC 7542 32-Core processors, 1 TB of RAM and dual Nvidia A100 GPUs.
G Author statement
The authors bear all responsibility in case of any violation of rights during the collection of the data or other work, and will take appropriate action when needed, e.g., to remove data with such issues.
H Data collection
H.1 Questionnaires
Questionnaire data were collected and managed using REDCap electronic data capture tools hosted at the University of Arizona [134-136]. REDCap (Research Electronic Data Capture) is a secure, web-based software platform designed to support data capture for research studies, providing 1) an intuitive interface for validated data capture; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for data integration and interoperability with external sources.
Figure 5: The montage we used for combined EEG/fNIRS data acquisition. EEG electrodes were located in the anterior frontal (AFF1h, AFF2h), frontal (F7, F8), frontocentral (FC5, FCz, FC6), central (C3, Cz, C4), occipital (O1, O2), temporal (T7, T8) and parietal (P7, P3, Pz, P4, P8) regions. fNIRS optodes were located in frontal (Fz, F1, F2, F3, F4, F5, F6), anterior frontal (AF3, AF4, AFz, AF7, AF8), frontal polar (FP1, FPz, FP2). The line is the channel formed when fNIRS source optode (red) and fNIRS detector (blue) optode are combined.
H.2 EEG
Equipment We used actiCHamp Plus (Brain Vision, LLC), which records and streams EEG signals at  $500\mathrm{Hz}$ .
Montage We used an adapted, 32-electrodes, 10-20 electrode placement system [137], to provide adequate coverage of all regions of the brain [54]. The actual placement of the electrodes (i.e., the montage) used for our experiment is shown in the combined EEG/fNIRS montage in Figure 5.
For experiments prior to November 22, 2022, our dataset contains the following EEG channels: AFF1h, F7, FC5, C3, T7, TP9, Pz, P3, P7, O1, O2, P8, P4, TP10, Cz, C4, T8, FC6, FCz, F8, AFF2h, AFF5h, FC1, CP5, CP1, PO9, Oz, PO10, CP6,CP2, FC2, AFF6h. For experiments after November 22, 2022, our dataset contains the following EEG channels: AFF1h, F7, FC5, C3, T7, TP9, Pz, P3, P7, O1, O2, P8, P4, TP10, Cz, C4, T8, FC6, FCz, F8, AFF2h (omitting AFF5h, FC1, CP5, CP1, PO9, Oz, PO10, CP6,CP2, FC2, AFF6h) to meet the time constraint of each experiment.
Software EEG data was streamed using LSL-actiCHamp [138], streamed over the local area network using Lab Streaming Layer [139] and recorded using LabRecorder [140] to XDF files [141].
H.3 fNIRS
Equipment We used NIRSport2 (NIRx Medical Technology LLC), which records and streams fNIRS signals at  $10.2\mathrm{Hz}$ .
Software fNIRS data was streamed using Aurora fNIRS [142], streamed over the local area network using Lab Streaming Layer (LSL) [139] and recorded using LabRecorder [140] to XDF files.
Montage See Figure 5.
H.4 EKG
We measured heart activity continuously throughout the experiment. We connected EKG leads to the Aux channel of the EEG/fNIRS caps. The EKG data was streamed alongside EEG and GSR data using LSL and written to XDF files.
H.5 GSR
GSR was recorded continuously throughout the experiment at a sampling rate of  $500\mathrm{Hz}$ . Two bipolar electrodes were placed on the ulnar border of the palm on the participants' right hand. We used an electrode paste that mimicked the salt concentration of sweat (i.e.,  $0.5\%$  saline) to improve signal quality. Participants' hands were wrapped in medical tape to secure the electrodes. Leads connected the electrodes to the Aux channel of the EEG/fNIRS cap. The GSR data was streamed alongside EEG and EKG data using LSL and written to XDF files.
H.6 Eye tracking
Equipment The eye tracking system used in this study was Pupil Core (Pupil Labs GmbH). The eye tracker was connected to a computer via USB and eye tracking data was streamed using LSL at  $250\mathrm{Hz}$  and written to XDF files.
**Calibration** The eye tracker is positioned in front of the participant's eyes, then eye tracker was calibrated using a 5-point calibration procedure (the participants were asked to look at the target on the screen without moving their head). The calibration procedure gave a confidence score and we made sure it was over  $90\%$ .
I Data Extraction and Labeling
The data extraction process encompassed retrieving diverse stream types such as EEG, EKG, GSR, fNIRS, and Gaze from a consolidated XDF file, which had been stored using LSL. For enhanced analysis and processing capabilities, each stream was then extracted from the xdf file and converted into separate Python dataframes utilizing pyXDF [143].
To label the stream data frames with task-specific labels, the start and stop times of each task were obtained from timestamps assigned by the task itself. For instance, let us consider the rest state. Its start and stop timestamps were compared with the timestamps in the EEG data frame. Whenever a match was found, the corresponding section in the EEG data was labeled as "start/rest_state" and "stop/rest_state" accordingly. This process was repeated for the remaining streams, associating the start and stop times of each task with their respective sections in the corresponding data frames. By labeling the data in this manner, it becomes easier to identify and analyze specific segments of interest within the recorded streams.
The extracted and labeled dataframes can be either saved as a CSV file or forwarded to the subsequent stage of processing, known as signal processing, which will be described in the following section.
- For fNIRS dataframes, they consist of various components including raw signals representing changes in attenuation values detected from two wavelengths (W1: 760 nm and W2: 850 nm), as well as HbO (oxygenated hemoglobin) and HbR (deoxygenated hemoglobin) signals. When saving the data as a CSV file, typically only the HbO and HbR signals are preserved. However, if further signal processing is required, we retain both the raw signals and the HbO and HbR signals for subsequent analysis.
- For EEG, EKG, and GSR dataframes, they were converted to voltage values since the data was recorded in microvolts. The conversion was performed by multiplying the data by the corresponding scale factor of  $1 \times 10^{-6}$ .
J Signal Processing
J.1 Synchronization of EEG, EKG, GSR, and fNIRS Signals
In multimodal neuroimaging studies, synchronizing signals from multiple modalities is a crucial step to conducting comprehensive studies on all these modalities together. The discrepancy between EEG, EKG, GSR, and fNIRS signals' time series is an issue that researchers encounter frequently due to the limitations of recording hardware or the necessity to remove invalid signals. Additionally, these two modalities have distinct recording rates, further complicating their alignment. To facilitate comprehensive evaluation of EEG, EKG, GSR, and fNIRS signals, it is essential to synchronize these signal modalities.
The synchronization process is a two-step approach involving noise artifacts removal of EEG and fNIRS, followed by the synchronization process of the signals to the desired sampling rate,
J.1.1 Removing Noise in EEG Signals with Notch Filter
EEG signals often exhibit susceptibility to artifacts, an interference that can be attributed to several sources. For instance, physiological factors such as eye movements or blinks can induce such artifacts [125], as can environmental elements like fluorescent lighting or grounding complications [126].
Upon thorough examination and visualization of the raw EEG data, we identified a consistent  $60\mathrm{Hz}$  electrical disturbance within the signal, along with corresponding harmonics. An anomalous peak was also noted around the  $5\mathrm{Hz}$  mark, potentially attributable to a grounding irregularity or an other environmental factors.
With the aid of MNE-Python [127], we efficiently mitigated these intrusive noises by deploying a notch filter. The filter was configured with a frequency of  $60\mathrm{Hz}$ , a transition bandwidth of  $9\mathrm{Hz}$ , and notch widths of  $2\mathrm{Hz}$ . The filter is applied to each channel separately.
J.1.2 Mitigating Artifacts in fNIRS Signals Utilizing Bandpass Filter
fNIRS signals are often susceptible to motion artifacts (MA) stemming from physiological activities, including cardiac and respiratory disturbances. These artifacts become particularly noticeable in the measurement of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations within the signal channels.
To address these challenges, we employed a bandpass filter as an effective noise reduction strategy. The filter was calibrated in line with the recommendations provided by [128]. With a low cutoff bandwidth of  $0.01\mathrm{Hz}$  and a high cutoff bandwidth of  $0.2\mathrm{Hz}$  for the  $4^{\mathrm{th}}$  order Butterworth method, the filter was tailored to selectively allow signal components within this frequency range while attenuating components outside the range. The filter is applied to each channel separately.
J.1.3 Pre-processing EKG and GSR Signals
To remove noise and improve peak-detection accuracy for EKG signals, we employed a finite impulse response (FIR) filter with  $0.67\mathrm{Hz}$  low cutoff frequency,  $45\mathrm{Hz}$  high cutoff frequency, and order of  $1.5\times$  sampling rate (where sampling rate is  $500\mathrm{Hz}$ ) implemented by NeuroKit2 [144] on top of the BioSPPy package.
We removed noise and smoothed the GSR signals using a low-pass filter with a  $3\mathrm{Hz}$  cutoff frequency and a  $4^{\mathrm{th}}$  order Butterworth filter, both implemented by Neurokit2.
J.1.4 Synchronization of EEG, EKG, GSR, and fNIRS Signals
To obtain the EEG, EKG, GSR, and fNIRS signals at the desired sampling rate, we resampled the signals using the FFT-based resampling method mne.filter.resample available in the Python
MNE library [127]. Then, we used linear interpolation to synchronize all the signals to a time series with a regular interval matching the desired sampling rate.
J.2 Synchronizing Task Data with EEG, EKG, GSR, and fNIRS Resampled Signals
Understanding the relationship between participants' behaviors, environmental stimuli, and neuroimaging data requires a precise synchronization of task data with the corresponding EEG, EKG, GSR, and fNIRS signals. By aligning these data streams, we can examine the influence of environmental stimuli on the participants' neuroimaging signals, which in turn, impact their behavior and task performance.
The process of integrating EEG, EKG, GSR, and fNIRS signals with task data starts with grouping of signals by the tasks during which they were recorded, followed by the synchronization of the task data to the corresponding EEG, EKG, GSR, and fNIRS signals.
J.2.1 Grouping EEG, EKG, GSR, and fNIRS Signals by Task
The preliminary step in our approach to synchronizing EEG, EKG, GSR, and fNIRS signals with the task data involves the grouping of EEG, EKG, GSR, and fNIRS signals by the tasks during which the signals were recorded. The task data can be categorized into two distinct types: status-based and event-based data.
Status-based task data This type of task data represent the current state of the task, such as task score. For each task, the grouping process of these data begins by including the EEG, EKG, GSR, and fNIRS signal recorded immediately before the task initiation and immediately following task completion. This ensures no data is overlooked at the boundaries of the task. Subsequently, all EEG, EKG, GSR, and fNIRS signals recorded between these two points are included, forming a complete set of signals associated with the task.
Event-based task data This type of task data, on the other hand, correspond to specific events that occur during the task, such as affective task arousal or the submission of a valence score. For each task, we determine the EEG, EKG, GSR, and fNIRS entry associated with the first event and the last event. These signal entries, as well as all entries recorded between these points, are included into the data set related to the task.
J.2.2 Synchronizing Task Data with EEG, EKG, GSR, and fNIRS Signals
Having grouped the EEG, EKG, GSR, and fNIRS signals according to task type, we then proceed to synchronize these signal entries with their respective task data.
Status-based task data The synchronization is accomplished by assigning the status data recorded closest in time to each EEG, EKG, GSR, and fNIRS signal entry. This method ensures that each EEG, EKG, GSR, and fNIRS entry is paired with the most representative status data.
Event-based task data We assign each event data to the EEG, EKG, GSR, and fNIRS signal entry recorded at the time closest to the occurrence of the event. Those EEG, EKG, GSR, and fNIRS signal entries without a corresponding event data are left unassigned, signifying that no specific event occurred during these recordings.
K Speech Elicitation Task
K.1 Reading Task
In the reading task, participants were shown a short piece of text describing the Minecraft environment used in the experiment and instructed to read it aloud with a natural tone and pace. The purpose of this task was to elicit an identical set of specialized Minecraft terminology that the participants were likely to use during the experiment, in order to control for possible word-frequency and other linguistic effects. This task also creates a fallback audio recording in case participants are not sufficiently
Figure 6: Top-down view of a building floor. Each room has a unique label. The map also contains three places of interest, marked with overlaid graphics.
verbose during the spontaneous elicitation (i.e., during the Minecraft tasks), and gives them a short introduction to the Minecraft environment.
Participants read out the following description of the Minecraft environment and game objectives:
Minecraft is a sandbox construction game in which players can control the environment. The Minecraft world for this game will require players to pick roles. You are tasked with search and rescue in this scenario. Keep the channels of communication open. Since this is a rescue site, be careful as you clear rubble and identify the critical victims.
Some of the tools you will have access to include a medkit, a stretcher, and a hammer. Using the map, identify the zone the team wishes to work on, and proceed to clear the floor and the rooms. As you orient yourself in the environment, pay attention to doors and windows.
Player roles include a medic, and engineer and a searcher. You can pick your roles based on the game's requirements. Pay attention to the clock. Keep the game's objectives in sight as you proceed.
K.2 Map Task
Next, we used a map task in order to capture audio containing speech representative of the participant's typical speaking style. Participants were shown a building floor plan with several labelled regions and icons indicating locations of interest (see Figure 6).
Participants were instructed to imagine a friend standing at one of these locations and provide descriptive verbal instructions directing their friend to the other two locations, using as many details from the map and its labels as they could. This task also served as a way for participants to practice giving navigational guidance to their teammates prior to doing so in the Minecraft tasks.
In the instructions, they were informed that their friend was located at the building entrance on the mid-left of the map(indicated by the person icon) and needed to first travel to an office located in room 203 E (top left, box icon) and then to Storage 242 (bottom-right, indicated with a 'stop' icon). The instructions were as follows:
For the following map task, you will be helping your friend reach their destination on a building floor. They are located on the entrance on the left side of the map, at the point with the person emoji. Your friend does not have access to the map. Therefore, you will need to give them directions so that they can:
Reach the room with the cardboard box (top left). Travel to the room with the stop sign (bottom right)
Please make sure that your friend has as much information and details as possible so that they can navigate the floor without getting lost.
L IAPS Stimuli for the affective task
For the affective task, we used stimuli from the International Affective Picture System (IAPS) [145] following Balconi, Grippa, and Vanutelli [86]. Stimuli were selected from the neutral, pleasant (high and low), and unpleasant (high and low) categories.
The IAPS stimuli numbers chosen are listed below.
- Individual task: 1525, 2025, 2352.2, 2487, 2521, 2635, 3019, 5260, 5900, 7270, 7405, 8034, 8485, 8531, and 9910.
- Team task: 1019, 1710, 1947, 2208, 2683, 2703, 3005.2, 5621, 6415, 6930, 7021, 7360, 7484, 8490, and 8502.
We note the following limitations.
The following factors limit the generalizability of the dataset:
- The dataset is primarily comprised of English-speaking undergraduates at a university in the US.
- Minecraft is known for low visual fidelity, so it is possible that results from these experiments may not generalize well to real-life SAR scenarios.
Additionally, due to the complexities of collecting this kind of data, there are a fair number of instances of missing data for a subset of modalities for a subset of sessions.
Finally, while we provide an initial release of the data with this paper, we have not been able to provide some of the planned modalities and documentation by the camera-ready deadline. However, we will continue to work on providing these at a rapid pace, and expect to have all the planned data modalities and documentation available by the time of the NeurIPS 2023 conference.
N Affect prediction score changed in camera-ready manuscript
The results for the experiment on affect prediction in this camera-ready version of the manuscript are significantly different (i.e., the accuracy is lower) from the results in the accepted version of the paper (i.e., after the rebuttal period). This is because our intention was to test on individual-image pairs not seen in training, which is very challenging. However, there was a bug in the splits, where information was leaked from training to testing.
The results in this camera-ready manuscript are generated by code that incorporates a fix for the data leakage.
O Consent form
Consent Version: 02/25/2023
Page 1 of 5
University of Arizona Consent to Participate in Research
Study Title: ToMCAT: Theory of Mind-based Cognitive Architecture for Teams
Principal Investigator: Adarash Pyarelal
Sponsor: This research is funded by the U.S. Department of Defense.
Summary of the research
This is a consent form for participation in a research project. Your participation in this research study is voluntary. It contains important information about this study and what to expect if you decide to participate. Please consider the information carefully. Feel free to ask questions before making your decision whether or not to participate.
The purpose of this research is to develop an artificially intelligent computer agent that can help human teams perform better. If you choose to participate, you will play a Minecraft game, while the computer agent watches and sometimes provides advice. During the game we will measure your brain activity, eye gaze, heart rate, muscle activity and sweating and the computer agent may ask you questions about how you are feeling. We will also videotape you while you play the game. Your participation in the study will take place during two lab sessions and will require about 3.5 hours of your time. There is some risk that someone could recognize you in the videotape, or that you will be upset by the game. There is no direct benefit to you of participating.
Why is this study being done?
This research study is being done so that we can develop an artificially intelligent (AI) agent that can participate in human teams to improve the team's performance. The first goal of the research is to train the agent to observe the team (including video of the team, chat messages, brain activity, heart rate, sweating and eye gaze) and predict from those observations what the humans are trying to do in the game, how coordinated they are as a team, and how they feel about the game and about each other. The second goal of the research is to train the agent to communicate with the humans and take actions that could improve the team's performance.
What will happen if I take part in this study?
If you choose to participate you will do the activities listed below during two lab sessions.
- At the pre-session, you will be asked to answer some brief socio-demographic questions (e.g., your age, ethnicity, etc.) and do a speech/language task where you will read some sentences out loud and provide spoken directions for someone else to navigate from a start to an end point on a map that will be given to you. You will then be asked to schedule a time within 1-3 weeks to complete the second testing session.
- At the testing session, sensors will be placed on your torso and head that will measure heart rate, muscle activity, sweating, brain activity and eye gaze. You will also wear a headset with speakers and a microphone.
HSPP Use Only: Consent Form
Fed non-medical v2019-02
Approved by University of Arizona
Date Approved: 3/31/2023
- The session will be videotaped.
- You will be asked to complete the following three baseline tasks on the computer: finger tapping, looking at emotional pictures, and a ping-pong game.
- After the baseline tasks you will play a game in Minecraft. You will be given a tutorial about how to play the game and a chance to practice the game.
- You will then complete a 5-minute competency test so that we can better understand video-gaming proficiency across individuals.
- The Minecraft game will either involve: 1) saving villagers, who are hiding in closed rooms within a building, from various creatures that appear in Minecraft (e.g., zombies, creepers, etc.), or 2) a search and rescue mission where you must navigate through a building and rescue victims and remove hazards. There will be a 20-minute time limit on the game. Your performance during the game will be recorded.
- Every 3 minutes, the computer agent may pause the game and ask you to answer some questions about what you are thinking and feeling. The agent may also offer you some information or advice. The agent may be acting in a truly "intelligent" way, coming up with its own information and advice, or it may be preprogrammed or controlled by the research assistant.
- After the game, all sensors and the headsets will be removed. You will answer another set of questions about your feelings about the game and the agent.
- Finally, the game may be played back on the computer screen and you may be asked to pause the play-back (by pushing a computer key) at any point that you remember having a plan in mind (e.g., an idea about what you were trying to do, or a plan about a sequence of actions that you intended to take), or changing your plans, or feeling confused about what to do. You will be asked to tell the research assistant what you remember, and the research assistant may ask clarifying questions. This will be audio-recorded and may be translated into text using natural language processing algorithms.
How long will I be in the study?