SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Competitive Ping Pong Task Inspired by Sinha et al. [97], our competitive ping-pong task has participants compete against each other in a 2-minute 1-on-1 computer-based ping-pong game. Typically, two of the three participants would compete against each other, while the third competed against a confederate. Players controlled an on-screen paddle with a mouse. Paddles were positioned
on the left and right sides of the screen, and constrained to move solely vertically. Participants scored a point whenever the ball hit the wall of the side opposite to that of their paddle. After hitting the wall, the ball would ricochet back with the vertical component of its velocity being randomized. Prior to the match, a 10-second familiarization phase was provided during which the participants could practice moving their paddles while the ball remained stationary at the center of the screen.
Cooperative Ping Pong Task This task was similar to the competitive version, except that all three participants were on the same side, playing against an AI agent instead of against each other. The horizontal component of the ball's velocity during the cooperative task was higher than that in the competitive task. The participants' paddles could move through each other and were on the left side of the screen, while the AI agent's paddle was on the right. Similar to the competitive task, a 10-second familiarization phase was provided before the start of the match.
5 Exploratory experiments
We developed two simple experiments to illustrate the large scope of new studies our data set can support. The first is designed to compare the power of EEG and fNIRS data to predict self-reported affect, and the second explores whether synchronization of EEG and fNIRS signals among team members is predictive of team performance. Note that while we do not fuse the data from the two modalities, these experiments are intrinsically multimodal since the EEG and fNIRS modalities are recorded simultaneously for each participant, thus enabling us to compare their predictive power.
We emphasize that these experiments are exploratory (i.e., meant to recognize novel patterns in the data, which can potentially lead to the generation of new hypotheses) rather than confirmatory (i.e., testing existing hypotheses). Both types of experiments are required for scientific progress [98]. Exploratory experiments are especially appropriate when the topic of research—in our case, machine learning based on brain data—is relatively less well-studied compared to other modalities (due to the inherently challenging nature of collecting brain data) such as images, text, and audio.
5.1 Predicting affect from brain scan data
Predicting affect from brain scan data is a challenging, yet intriguing endeavor in the realm of neuroscience and affective computing. In this study, we use a multimodal dataset comprising EEG and fNIRS data to examine the feasibility of predicting individuals' self-reported valence and arousal using data from the individual affective task (§ 4.2). We focus on specific regions of the head to ensure spatial alignment of EEG and fNIRS data, enabling the comparison of these two modalities.
Most work on mapping brain scan data to valence and arousal uses EEG data. This includes Rayatdoost et al. [99], who developed a deep domain adversarial neural network (DANN) to link EEG data to valence and arousal, achieving average classification accuracies of  $72.8\%$  and  $65.0\%$  for valence and arousal respectively on the MAHNOB-HCI database [100], and accuracies of  $69.8\%$  and  $57.6\%$  for valence and arousal classification on the DAI-EF database [101]. Galvão, Alarcão, and Fonseca [102] used kNN on features derived from the EEG frequency domain to predict valence and arousal values, achieving accuracies of  $79.4\%$ ,  $83\%$ , and  $80.6\%$  on the DEAP [103], AMIGOS [104], and DREAMER [105] datasets, respectively. For all five datasets, participants watched videos and subsequently rated their emotions using SAM scales [106] for valence and arousal (a 9-point scale for MAHNOB-HCI, DAI-EF, DEAP, and AMIGOS, and a 5-point scale for DREAMER)
Bandara et al. [107] used a support vector machine (SVM) to predict valence and arousal scores from fNIRS data, achieving an  $F_{1}$  score of 0.74 on the DEAP dataset. For images, Trambaiolli, Biazoli, Cravo, et al. [108] achieved a classification accuracy of  $89\%$  in discerning positive from negative valence using fNIRS signals and a linear discriminant analysis (LDA). Finally, Sun, Ayaz, and Akansu [109] combined EEG and fNIRS data, and used an SVM to classify with  $75\%$  accuracy.
While many studies use a SAM scale with ranges of 1-9, 1-5, or 1-10, we use a scale with a range of  $-2$  to  $+2$  for a more compact scale and a clear neutral reference. We use brain scan data obtained from EEG and fNIRS recording trimmed to sets whose EEG and fNIRS samples overlap as best as possible. Specifically, for EEG we selected channels FC5, FCz, FC6, F7, F8, AFF1h, and AFF2h, whereas for fNIRS we selected channels Fz-F1, Fz-F2, F3-F7, F3-F1, F4-F2, F4-F6, AF3-F7, AF3-Afz, AF4-F6, and AF4-Afz. We trained separate CNNs [110] for fNIRS and EEG data. The fNIRS data were kept
Table 1: Accuracy and loss (mean ± standard error of the mean, computed over 5 folds) for classification of valence and arousal scores.
Model	Offset (s)	Window size (s)	Accuracy (%)	Loss
Valence	Arousal
CNNfNIRS	0	2	26.8 ± 1.2	30.1 ± 0.8	3.02 ± 0.012
CNNfNIRS	2	2	30.4 ± 1.2	28.0 ± 0.8	3.02 ± 0.008
CNNfNIRS	5	2	28.4 ± 1.2	30.3 ± 0.8	3.01 ± 0.007
CNNEEG	0	1	29.3 ± 1.4	29.8 ± 1.3	2.80 ± 0.004
BaselinefNIRS	N/A	N/A	29.0 ± 0.9	29.7 ± 0.1	N/A
BaselineEEG	N/A	N/A	27.5 ± 0.7	30.3 ± 0.5	N/A
in the spatial domain, while EEG data were segmented into frequency bands (theta, alpha, beta, and gamma) and their wavelet features were extracted with four levels of decomposition.
fNIRS signals exhibit a phenomenon known as the hemodynamic response factor (HRF) [111, 112] which represents the relationship between neural activity and the corresponding changes in blood oxygenation levels that occur in response to that activity (when a participant views a specific image). The HRF consists of two phases: (i) the initial dip, which typically lasts for 1–2s and involves an initial drop in the concentration of oxygenated hemoglobin (HbO) and a simultaneous increase in deoxygenated hemoglobin (HbR) shortly after neural activation, and (ii) the hemodynamic response peak, in which there is an increase in HbO concentration and a decrease in HbR concentration, resulting in a peak in HbO concentration that occurs  $\approx$  4–6s after activation.
We present our results in Table 1. We tried several offsets for the fNIRS data to see if there was a noticeable effect due to the HRF as discussed above. We note that accuracies for EEG and fNIRS are not precisely comparable due to the disparity in participant numbers—97 for EEG and 102 for fNIRS. Unfortunately, our attempt at using basic CNNs for classification did not perform any better than the baseline. In contrast to prior work, we held out data in units of participant-image pairs, rather than predicting solely within participants. The relationship between functional brain regions and cap location varies among participants. While the training data did contain some data for held out participants looking at other images, most of the training data was for other participants. Success on this task will likely require addressing functional regions to individualized cap locations, as well as more effort on neural network design. Given basically baseline performance, we are not surprised that accounting for the HRF with an offset did not make any real difference. Further details and quantitative results can be found in the confusion matrices provided in § F.1.
5.2 Linking temporal correlation of brain signals with scores
Shared cognition in team environments is gaining interest, particularly in understanding social dynamics that lead to successful performance, and in developing intervention techniques to enhance collaboration [113, 114]. Research has consistently shown that cooperation plays a vital role in influencing overall task performance [113], enabling coordination and information sharing, which enhances the effectiveness of the team [115, 116]. Studies have found behavioral and neurological synchronization between subjects during cooperative tasks [115-118], reinforcing the idea that cooperation may go beyond mere action coordination or knowledge sharing and may suggest a shared mental model within cooperative settings that includes coordination and sharing of social content like emotion and intentions [113, 119].
Research efforts on these fronts will benefit from more comprehensive data, specifically, data containing a larger set of modalities, and from multiple interacting participants. Existing studies mostly focus on limited brain regions with a single modality (i.e., either fNIRS or EEG but not both) during a single task [116, 117], which undermines their ability to capture the complex nature of shared mental models. Our research extends previous methods of classifying cognitive processes in single-participant studies [120] to identifying cognitive processes in teams, thus enabling the study of shared mental models of teams.
In this second experiment to illustrate the potential of our dataset, we study whether synchronization of EEG and fNIRS signals between team members can predict team performance. Building on research establishing a connection between synchronization and higher cooperation levels [115-117], we examine the brain holistically for associations between the correlation of EEG and fNIRS data
Table 2: Linear regression coefficient  $\beta$  (slope) and associated significance ( $p$ -value), cross-validation with leave-one-out mean error ( $\bar{e}$ ) with standard error of those means ( $\sigma_{\bar{e}}$ ) in parentheses for each fNIRS and EEG channel over average correlation from all experiments. fNIRS (EEG) channels with  $R^2 < 0.165(0.075)$  for all three tasks are excluded from the table. Blue and red numbers indicate positive and negative slopes respectively.  $R^2$  of the null model for all channels is 0.0. The values under the  $\bar{e}^*$  and  $\beta^*$  columns are in units of  $10^2$ .  $p$ -values less than 0.05 are in bold.
Channel	Ping Pong Cooperative	Minecraft Saturn A	Minecraft Saturn B
R2	p	ē (σē)	β	R2	p	ē* (σē)	β*	R2	p	ē* (σē)	β*
fNIRS
F3-F5 (HbO)	0.03	0.30	2.3 (.3)	2.45	0.10	0.07	1.5 (.2)	-2.99	0.27	0.01	1.1 (.1)	-7.60
F3-F1 (HbO)	0.00	0.89	2.3 (.3)	0.43	0.07	0.12	1.6 (.2)	-2.40	0.17	0.03	1.2 (.2)	-9.13
Fz-AFz (HbO)	0.02	0.35	2.2 (.3)	2.11	0.23	0.00	1.3 (.2)	4.56	0.02	0.44	1.2 (.2)	-2.73
FPz-FP1 (HbO)	0.19	0.01	2.1 (.3)	6.43	0.09	0.08	1.6 (.2)	-2.64	0.01	0.62	1.3 (.2)	-2.66
FPz-AFz (HbO)	0.00	0.68	2.3 (.3)	0.77	0.26	0.00	1.3 (.2)	4.49	0.02	0.45	1.2 (.2)	-3.45
F4-F2 (HbO)	0.00	0.74	2.3 (.3)	-0.77	0.01	0.50	1.7 (.3)	1.02	0.20	0.02	1.2 (.2)	-10.32
AF3-FP1 (HbR)	0.11	0.04	2.2 (.3)	3.90	0.19	0.01	1.4 (.2)	-3.94	0.05	0.27	1.2 (.2)	-3.41
FPz-FP2 (HbR)	0.00	0.78	2.3 (.3)	-0.58	0.21	0.00	1.3 (.2)	-4.32	0.01	0.56	1.2 (.2)	2.16
AF8-F6 (HbR)	0.00	0.94	2.4 (.4)	0.17	0.29	0.00	1.3 (.2)	-4.99	0.10	0.10	1.2 (.2)	4.13
EEG
FCz	0.09	0.08	2.3 (.3)	-4.20	0.27	0.00	1.2 (.2)	-3.66	0.12	0.08	1.2 (.2)	-2.77
AFF1h	0.14	0.02	2.3 (.3)	-5.79	0.00	0.75	1.6 (.2)	0.60	0.02	0.53	1.3 (.2)	0.79
O2	0.04	0.24	2.4 (.3)	-2.54	0.24	0.00	1.4 (.2)	4.44	0.02	0.54	1.3 (.2)	0.88
TP9	0.13	0.03	2.3 (.3)	-6.98	0.02	0.43	1.6 (.2)	-1.06	0.17	0.04	1.2 (.2)	3.88
P3	0.00	0.70	2.4 (.3)	-0.90	0.00	0.76	1.6 (.2)	0.56	0.10	0.11	1.2 (.2)	1.81
P8	0.06	0.16	2.3 (.3)	-3.30	0.25	0.00	1.2 (.2)	-4.53	0.00	0.80	1.3 (.2)	-0.51
C4	0.00	0.91	2.4 (.4)	-0.31	0.07	0.13	1.5 (.2)	3.03	0.09	0.14	1.3 (.1)	-4.56
AFF2h	0.03	0.32	2.4 (.3)	-2.17	0.17	0.02	1.4 (.2)	4.45	0.02	0.54	1.3 (.2)	-0.98
T8	0.03	0.33	2.4 (.3)	-2.29	0.00	0.79	1.6 (.2)	-0.47	0.08	0.18	1.3 (.2)	-1.88
channels between participants with scores on the ping-pong tasks (a more constrained laboratory setting), and the two Minecraft-based SAR missions (a more naturalistic setting). We expect that stronger linkage between certain parts of the brain will be associated with better performance. Our analysis methods are similar to those in existing studies [116, 117]. Specifically, we are looking at whether the temporal correlation across participant brain signals can predict team performance.
For each of the three tasks, for each of the EEG and fNIRS channels, we computed the Pearson correlation of the brain data between the three participants in a team by taking the average of their pairwise correlations. If there were only two participants, we simply used their correlation. We compute correlations by synchronizing the measurements to a common start time and frequency (500 Hz for EEG and 10 Hz for fNIRS) using linear interpolation. We then used the correlation for each EEG and fNIRS channel as the predictor for the final task performance score as the outcome variable in a linear regression model fit to these variables over all teams. We evaluated the performance of the resulting models using the  $R^2$  measure, which quantifies the proportion of variance in the task performance that can be accounted for by the EEG and fNIRS correlations, and cross-validation with leave-one-out mean error, where we fit the regression model over all but one team and evaluate prediction error on the one held-out team in each fold. Our results are shown in Table 2.
For the Ping Pong Cooperative baseline task, many fNIRS channels had a positive slope for the fit of the task performance predicted by fNIRS synchronization. Conversely, the linkage between EEG-measured brain regions among participants and task performance often had negative slopes. In our Minecraft Saturn A mission, participants were in the process of familiarizing themselves with the task's challenges. Here, the EEG/fNIRS correlation between participants and task performance was varied, without a clear pattern. However, in the Minecraft Saturn B mission, in which participants were now familiar with the mission and each other, there was a larger number of fNIRS and EEG channels with a negative correlation between brain data linkage and task performance when compared to Minecraft Saturn A. Understanding what is going on will require more thorough analysis on where we suspect spurious results, mapping to brain function as seen in the baseline tasks, and accounting for the differences in what we expect EEG and fNIRS signals to tell us.
6 Dataset usage
Accessing the dataset The dataset and its documentation is available at https://tomcat.ivilab.org. We provide access to the data in two ways. The first is through a Datasette [121] instance, which provides graphical and programmatic interfaces for users to explore the data, retrieve subsets of it that they are interested in, or simply download the backing SQLite database. The second is in the form of pre-built files containing subsets of data that we (i) used for the experiments in this paper, and (ii) expect will be commonly requested by other researchers.
A key issue with using the raw data is that it comprises data from multiple asynchronous data streams. Hence, aligning the data for multiple participants entails interpolation. A second issue is recording specific issues encountered during data collection, such as an experimenter stepping in for a participant. Finally, derived data will also include standard data transformations and cleaning.
Continued engagement The scale and complexity of this dataset make it infeasible for us to annotate every type of label that might be of interest. Rather, we hope that the release of the dataset will seed the development of a community that works together to fully explore this data. We envision a process by which researchers build upon this dataset by adding layers of annotations for labels of interest, unlocking the ability to answer additional research questions. We are also happy to include pointers on our website to papers that use this dataset, in order to facilitate connections between researchers working with this dataset.
We highly encourage users to sign up for our mailing list to get updates on data issues and annotation layers, benchmarks, and documentation.
6.1 Additional usage examples
In § 5, we presented illustrative experiments in that study the affective component of ToM and the relationship between brain signal correlations and team performance on a collaborative task. One could study other aspects of ToM as well.
Intention detection For example, intentions are commonly considered part of human mental states. Consider a researcher who is interested in developing algorithms to infer participant intentions from observed behavior (i.e., plan recognition [122]). This could be done by encoding the observations as sequences of discrete actions—either manually or semi-automatically, depending on the temporal, spatial, and semantic granularity of interest. Data that could be used for this include the participant's in-game position and velocity, as well as the semantic contents of their utterances.
Utterance classification Another example use case involves labeling participant utterances with sentiment, emotion, and dialog act labels will enable developing models for sentiment, emotion and dialog act classification in task-related dialog. Unlike previous dialog datasets, the ToMCAT dataset contains text, speech, and physiological data that can be simultaneously leveraged for these classification tasks.
In this transdisciplinary work, we integrate numerous threads of research in computer science, psychology, and cognitive science to present the ToMCAT dataset, which to our knowledge is the only dataset that contains both (i) data on human-machine teaming in a complex synthetic task environment and (ii) rich physiological data from a number of sensors, including fNIRS and EEG, thus enabling the exploration of fundamental research questions related to the interplay of competition, cooperation, and neurophysiological responses in human-machine teams. Furthermore, to our knowledge, this is the largest open-access fNIRS dataset currently available. Finally, we conduct exploratory experiments linking valence, arousal, and team performance to brain signals, illustrating the potential of the dataset for exploring a variety of research questions. We are excited to share this dataset with the community and look forward to seeing the research findings it enables.
Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-20-1-0002. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. We would also like to acknowledge intramural funding from the University of Arizona's SensorLab.
[129] Florian Eyben, Martin Wöllmer, and Björn Schuller. Opensmile: The Munich Versatile and Fast Open-Source Audio Feature Extractor. In: Proceedings of the 18th ACM International Conference on Multimedia. MM '10. Firenze, Italy: Association for Computing Machinery, 2010, pp. 1459–1462.
[130] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. OpenFace 2.0: Facial Behavior Analysis Toolkit. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, Xi'an, China, May 15-19, 2018. IEEE Computer Society, 2018, pp. 59-66.
[131] Recommended Formats Statement - Datasets. URL: https://www.loc.gov/preservation/resources/rfs/data.html.
[132] The pandas development team. pandas-dev/pandas: Pandas. Version latest. Feb. 2020.
[133] Wes McKinney. Data Structures for Statistical Computing in Python. In: Proceedings of the 9th Python in Science Conference. Ed. by Stefan van der Walt and Jarrod Millman. 2010, pp. 56-61.
[134] Paul A. Harris, Robert Taylor, Robert Thielke, Jonathon Payne, Nathaniel Gonzalez, and Jose G. Conde. Research electronic data capture (REDCap)A metadata-driven methodology and workflow process for providing translational research informatics support. eng. In: Journal of biomedical informatics 42.2 (2009), pp. 377-381.
[135] Paul A. Harris et al. The REDCap consortium: Building an international community of software platform partners. In: Journal of Biomedical Informatics 95 (2019), pp. 103208-103208.
[136] Colleen E Lawrence et al. A REDCap-based model for electronic consent (eConsent): Moving toward a more personalized consent. In: Journal of Clinical and Translational Science 4.4 (2020), pp. 345-353.
[137] Richard W Homan, John Herman, and Phillip Purdy. Cerebral location of international 1020 system electrode placement. In: Electroencephalography and Clinical Neurophysiology 66.4 (Apr. 1987), pp. 376-382.
[138] LSL-actiChamp. URL: https://github.com/brain-products/LSL-actiChamp.
[139] Lab Streaming Layer. URL: https://github.com/sccn/labstreaminglayer.
[140] LabRecorder. URL: https://github.com/labstreaminglayer/App-LabRecorder.
[141] The XDF file format. URL: https://github.com/sccn/xdf.
[142] NIRxfNIRS Software. URL: https://nirx.net/software.
[143] pyXDF. URL: https://github.com/xdf-modules/pyxdf.
[144] Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lepinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. NeuroKit2: A Python toolbox for neurophysiological signal processing. In: Behavior Research Methods 53.4 (Feb. 2021), pp. 1689-1696.
[145] Peter Lang and Margaret M Bradley. The International Affective Picture System (IAPS) in the study of emotion and attention. In: Handbook of emotion elicitation and assessment 29 (2007), pp. 70-73.
A Datasheet 19
A.1 Motivation 19
A.2 Composition 19
A.3 Collection process 23
A.4 Preprocessing/cleaning/labeling 26
A.5 Uses 27
A.6 Distribution 28
A.7 Maintenance 29
B Recruitment 30
C Data format 30
D Structured metadata 30
E Long-term preservation 31
F Reproducibility 31
F.1 CNN experiments 31
F.2 Correlation experiments 34
G Author statement 36
H Data collection 36
H.1 Questionnaires 36
H.2 EEG 37
H.3 fNIRS 37
H.4 EKG 38
H.5 GSR 38
H.6 Eye tracking 38
I Data Extraction and Labeling 38
J Signal Processing 39
J.1 Synchronization of EEG, EKG, GSR, and fNIRS Signals 39
J.2 Synchronizing Task Data with EEG, EKG, GSR, and fNIRS Resampled Signals. 40
K Speech Elicitation Task 40
K.1 Reading Task 40
K.2 Map Task 41
L IAPS Stimuli for the affective task 42
M Limitations 42
N Affect prediction score changed in camera-ready manuscript 42
O Consent form 43
PRecruitment materials 48
The dataset was created to significantly advance our understanding of teaming in both purely human and hybrid human-machine teams. While there exist prior datasets involving human-AI collaboration in complex open-ended environments, none include the large number of physiological measures that we provide.
This dataset was created by members of the ToMCAT research project (https://ml4ai.github.io/tomcat), spanning five departments (School of Information, Computer Science, Norton School of Human Ecology, Psychology, and Linguistics) and three labs (ML4AI, IVILab, and the TIES Lab) at the University of Arizona.
The creation of this dataset was funded by the Army Research Office and was accomplished under Grant Number W911NF-20-1-0002. The grant was awarded through the Defense Advanced Research Projects Agency (DARPA). We would also like to acknowledge intramural funding from the University of Arizona's SensorLab.
The dataset consists of data from experiments in which teams of three humans (and optionally, an AI advisor) execute urban search-and-rescue (USAR) missions in the ASIST Study 3 testbed [4], which is based on Minecraft and instrumented to capture in-game states, actions, and events with a high level of granularity. Additionally, we instrument the participants with additional sensors that capture data via the following modalities: spoken communications, eye tracking, facial image captures, galvanic skin response (GSR), electroencephalography (EEG), and functional near-infrared spectroscopy (fNIRS).
We provide data from 40 teams.