SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The dataset is a sample of the subset of the human population that meet the following criteria: (i) over 18 years of age, (ii) able to read and speak English, and does not have any major physical limitations that would interfere with completing tasks on a computer (e.g., limited vision or hearing, problems with fine motor control).
The sample is weighted heavily towards younger participants (ages 18-22, corresponding to the typical age range for undergraduates at the university where the study was conducted). The ratio of
female to male participants is representative of the sex distribution of the students at the university [123].
Of the participants whose race was recorded,  $48\%$  identified as 'Non-Hispanic White' or 'European American', which is lower than the proportion of enrolled students at the university that identified as 'White'  $(66\%)$ .  $7\%$  identified as 'African American' and  $13\%$  as 'Asian American', which is roughly consistent with the proportion of students at the university who identify as 'Black or African American'  $(6.7\%)$  or 'Asian'  $(10.6\%)$  of the student population at the university.  $27\%$  of the participants identified as Hispanic, which is consistent with the portion of students at the university who identify as 'Hispanic or Latinx', and greater than the percentage of 'Hispanic or Latino' people in the US population in the 2020 census  $(18.7\%)$  [124]—this can be attributed to the geographic location of the university (the southwestern US). Most deviations from representativeness of the sample from the larger population can be attributed primarily to the sampling methods we used (convenience and snowball sampling, described later in this document).
Modalities include two kinds of brain scan data—functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), as well as skin conductance, heart rate, eye tracking, face images, spoken dialog audio data with automatic speech recognition (ASR) transcriptions, game screenshots, gameplay data, game performance data, demographic data, and self-report questionnaires.
We provide both nearly-raw and derived data products.
There are many potential prediction tasks that can be performed with this dataset. Some that may be of interest to researchers are provided below:
- Individual and team scores on the baseline tasks.
- Post-game survey questionnaire responses.
There are instances of missing data in the dataset. Some of the instances are expected, as it is built into the design of the experiment (e.g., when participants skip optional questions in questionnaires). However, as is the case with any complex study involving human subjects, there are many ways for data collection to not proceed as originally expected, resulting in missing data. We have instances of missing data due to a variety of reasons:
- When an experimenter sat in for a no-show participant or a participant that left in the middle of a group session, most of the time, the experimenter did not wear an EEG/fNIRS cap or eye tracker. This is because properly setting up the cap and calibrating the eye tracker for a participant is a time-consuming process, and so in such cases, we chose to prioritize the number of tasks for which we had data for the group session over ensuring the presence of the EEG/fNIRS/gaze modalities. Thus, we do not obtain physiological measures for them for the whole group session.
- There were a few group sessions where a participant either refused to wear their EEG/fNIRS cap (and/or eye tracker) or had to take them off due to physical discomfort, usually caused by the cap not setup right for the participant's head size, or glasses not fitting right on their head or over their own eyeglasses. In these cases, the physiological measures were not obtained for the entire group session for that participant.
- Participant takes cap or glasses off sometime in the middle of the experiment after tasks have been completed: Again, this was usually caused by the participant not feeling comfortable with the device on, the cap not setup right for the participant's head size, glasses not fitting right on head or over personal glasses, or the participant feeling sick. Therefore, the EEG,
NIRS, and/or gaze data would be good for completed tasks up to the point where the participant took the device off.
- The following group sessions had no EEG amplifier at the 'lion' station: exp_2023_04_17_13, exp_2023_04_18_14, exp_2023_04_20_14, exp_2023_04_21_10, exp_2023_04_24_13, exp_2023_04_26_10, and exp_2023_04_27_14. We had to send the amplifier in for repair around 2023-04-17 and did not receive a loaner amplifier until 2023-04-28. Therefore, the EEG data for the 'lion' station for those group sessions is either missing or invalid.
- In earlier group sessions, we attempted to run the NIRx Aurora fNIRS data acquisition software on an iMac. However, it would crash sometimes. If we noticed the crash, in most cases, we would immediately go to the iMac and restart it. However, there were some sessions where we did not find out that Aurora had crashed until after the session had ended, leading to missing fNIRS data for these sessions. In later group sessions, we switched to using a Windows computer, which resolved the problem.
Each participant has a unique ID, as does each group session. These can be used to link data from different modalities and tasks.
We do not provide recommended data splits.
Sources of noise EEG signals are susceptible to artifacts from eye movements/blinks [125] and external sources like fluorescent lights or grounding problems [126]. From plotting the raw EEG data, we observed a predictable  $60\mathrm{Hz}$  electrical noise in the signal (and related harmonics). We also observed a peak around  $5\mathrm{Hz}$  that may be due to a grounding issue or some other environmental influence. Using MNE-Python [127] we eliminated noise from these sources using a notch filter with frequency set to  $60\mathrm{Hz}$ , trans bandwidth set to  $9\mathrm{Hz}$  and notch widths set to  $2\mathrm{Hz}$ .
fNIRS signals are known to have motion artifacts (MA) (e.g., cardiac and respiratory artifacts). MA of all fNIRS channels' oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations were then filtered using a bandpass filter with bandwidth set to  $0.01 - 0.2\mathrm{Hz}$  [128].
EKG data may contain high-frequency noise and artifacts due to muscle movements. These were filtered using a filter with highpass:  $0.0\mathrm{Hz}$  lowpass:  $250.0\mathrm{Hz}$  using MNE-Python [127].
The GSR data may contain high-frequency noise and rapid-transient artifacts, these were filtered using a filter with highpass:  $0.0\mathrm{Hz}$  lowpass:  $250.0\mathrm{Hz}$  using MNE-Python [127].
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
The dataset is self-contained. However, documentation for the schemas of the messages that comprise the data from the testbed is not on our website but is available in the repository for the ASIST Study 3 testbed [4].
The dataset does not contain data that might be considered confidential.
The data contains screenshots of a Minecraft-based urban search-and-rescue mission involving rescuing victims that were injured in a building collapse. While we expect that these are unlikely to cause anxiety due to the low visual fidelity of Minecraft, we are not able to rule it out.
Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset. Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
The dataset includes questions on demographics, gaming experience, and health. Depending on the variables on interest, it is possible for researchers to look at data by gender, race/ethnicity, and age, among other variables. However, no one was purposefully excluded based on any of these criteria.
To the best of our knowledge, it is not possible to identify individuals either directly or indirectly from the publicly released portions of this dataset.
Aggregated demographic data for selected dimensions are provided in Table 3. Note that demographic data is missing for two regular participants and the 9 experimental confederates that filled in for missing participants.
Table 3: Aggregated demographic data for selected dimensions.
(a) Age distribution	(b) Sex	(c) Race	(d) Hispanic
# Participants	40	Female	54	Other	32	No	73
30	Male	45	Non-Hispanic White	28	Yes	27
20	Prefer not to say	1	European American	20
10	Other	0	Asian American	13
0	e) Videogaming experience	African American	7
Have played them occasionally	38	Have played it occasionally	42
Have played them fairly often	30	Have played it fairly often	24
Have played them regularly for years	28	Have played it regularly for years	18
Never played them	4	Never played it	16
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
All publicly released data in the dataset has been deidentified.
Questionnaires While we collected socio-demographic and health data through self-report, any identifiable information has been removed so the data we provide does not include personal information (e.g., e-mail, phone number, etc.) for any of the participants.
Audio We recorded spoken audio data from participants during the study. While participants have given their consent for their audio recordings to be shared publicly, we do not do so at this time since we believe that the audio data is still fairly sensitive. We will publish derived features from the audio data (e.g., vocalic features extracted using openSMILE [129]). We will support researchers who want to perform alternative processing on the raw audio files by working with them to run their processing code on our machines and sharing the resulting derived data with them.
Face images We do not plan to release the face images that we captured during the experimental sessions, as they contain personally identifiable information (PII). We plan to provide automated FACS codes for the images by processing them using OpenFace [130], contingent on the codes passing a manual validation/spot-checking procedure<sup>1</sup>. Similar to the raw audio, we are willing to work with researchers to run their custom face image processing pipelines on our machines and share the (de-identified) results with them.
A.3 Collection process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
The data was directly observable in some case (e.g., raw audio/images/physiological measures), and reported by subjects for other cases (e.g. self-report questionnaires). For the data that was reported by subjects, we used questionnaires that were previously validated in the literature.
See Appendix H for details on how the data was collected.
The sampling strategy for this study was a convenience sample, which is a non-probabilistic sampling strategy where participants are included due to ease of access. We also used snowball sampling as participants were asked to pass along our study contact information to any of their friends, peers, co-workers, and other people who they thought might be interested in participating.
An interdisciplinary research team comprised of individuals from Computer Science (CS), Human Development & Family Science (HDFS), Psychology, and Speech, Language, and & Hearing Sciences (SLHS) was involved in the data collection process. A postdoctoral research associate in HDFS recruited and scheduled participants for the study, among other study-related responsibilities. Data collection was conducted by a team of graduate students who working hands-on and oversaw undergraduate research assistants (RAs). All graduate students were financially compensated as this project provided funding in the form of a research assistantship. However, undergraduate students were compensated in the form of independent research study credit with the study PI. Undergraduates
<sup>1</sup>Participants wore masks throughout the experiment, which poses a significant challenge for automated FACS coding.
Figure 2: Workflow showing the IRB process for human subjects research.
enrolled in the independent study as part of their semester coursework and received 3 credits at the end of the semester in a graded format (pass/fail).
Participants were compensated in the form of an electronic Amazon.com gift card. Compensation was calculated based on time spent in the study— $5.00 for the 30-minute pre-session and$ 10.00 for each hour of the testing session for a maximum of $35.00.
Participants were further incentivized through gamification—they had a chance of winning an additional $20.00 Amazon gift card if they obtained the highest score on the Minecraft missions out of 20 group sessions.
Participants who enrolled in the study through the SONA system received compensation in the form of SONA credit. For the pre-session and testing session, participants received 1 credit and 3 credits, respectively.
The data was collected between September 9, 2022, and March 5, 2023.
Yes, the project (IRB 2001272896) was approved by the institution (see approval below).
Prior to beginning the study, an in-depth ethical review process was conducted by the University of Arizona's Institutional Review Board (IRB). This step requires the study PI(s) to complete the application for human subjects research and provide any documentation that will be used during the study including, but not limited to, the following:
- List of all members of the research team
- Updated Collaborative Institutional Training Initiative (CITI) and Conflict of Interest (COI) trainings for all members of the research team.
- Informed consent forms
E-mails to participants
- Advertisements
- Phone call/e-mail script
- Debriefing
- Study questionnaires
Research design and methods
The human subjects research application is submitted to our electronic IRB system (eIRB) where it undergoes the workflow shown in Figure 2.
The IRB reviewer and the study PI and/or primary contact for IRB on the research team's side are always in communication to address requested edits. When all edits have been completed, the project goes to the full IRB for review. At this stage, more clarifications may be requested or the project goes to post-review. Once everything is finalized and the IRB approval is on all documents, then those
documents are used until they are updated (e.g., when modifications are necessary, continuing review, etc.).
The recruitment materials submitted to the IRB are included in this document (see Appendix P).
We collected the data from the individuals in question directly.
Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to or otherwise reproduce, the exact language of the notification itself.
The informed consent form explained to participants the data collection process in depth. In addition, the research team would always ensure that the participants were aware of the various parts of data collection (e.g., fNIRS, EEG, REDCap survey, etc.) and had opportunity to ask questions at any point in time. At the beginning and throughout each session, the research team went over what will be asked of the participants. The research team were always communicating with the participants and vice versa about what data we were going to collect in a specific part of the study.
Participants also were able to contact the research team whenever they had questions prior to coming in for the in-lab sessions. The questions would range from ensuring they understood the duration of the pre-session and testing session to questions about the consent form to scheduling needs and verifying study location.
Informed consent was obtained in-person when participants visited the University for their initial lab session (referred to as the pre-session). When participants arrived at the research lab a trained undergraduate RA or graduate student went over the consent form which covered all study-related information. Then the RA asked the participant to read/look over the consent form to ensure that the participant understood the study and was able to ask any questions or concerns. The RA reminded the participant that they are not required to participate and can refuse to consent without penalty.
We also conducted informed consent online through REDCap. Participants who were interested in the study and agreed to participate were sent an individualized link for the consent form in REDCap. The consent form was housed in the File Repository in REDCap which is secure. When participants came for the in-lab sessions, they were again asked if they had any questions or concerns about the study. If participants had not completed the consent form prior to their in-lab sessions, then their individualized link was opened on the computer and they completed the consenting process then. The experimenter conducting the consent reminded the participants that they were not required to participate and could refuse to consent without penalty.
For any in-person informed consent, the research team left the participant alone in the room so that the participant could carefully read through the consent form and would not feel pressured to respond due to timing, presence of others, etc.
The consent forms are included in this document (see Appendix O).
Participants were notified throughout the consent and during the in-lab sessions by the research team that they could opt out of the study at any point in time. In such circumstances, the participants had access to the research team through e-mail, phone, and in-person so that they could withdraw from the study.
Consent for future research was requested on the informed consent form for the present UA research team to store the identifiable data indefinitely (video and audio recording; eye tracking data) and to use it for future research without obtaining additional consent.
We recently had to change the PI of record on the IRB protocol since the previous PI of record unexpectedly passed away. We are preparing to submit a modification to contact all study participants and notify them of the contact information for the new PI. Once this has been approved, we will be updating the participants with this contact information.
While we did not perform a formal data protection impact analysis, the IRB approval process required us to describe impacts on study participants. The UArizona IRB deemed the risks to participants from our study to be minimal.
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remaining questions in this section.
EEG electrode setup alteration In our study, we made an alteration to our EEG electrode setup for experiments on or after November 11, 2022. Prior to this date, our EEG data comprised several channels, including AFF5h, FC1, CP5, CP1, PO9, Oz, PO10, CP6, CP2, FC2, and AFF6h. However, due to the prolonged calibration time required for these electrodes, we opted to exclude them from our experimental sessions starting November 11, 2022. The montage in Figure 5 reflects the updated setup.
For experiments prior to November 11, 2022, our dataset contains the following EEG channels: AFF1h, F7, FC5, C3, T7, TP9, Pz, P3, P7, O1, O2, P8, P4, TP10, Cz, C4, T8, FC6, FCz, F8, AFF2h, AFF5h, FC1, CP5, CP1, PO9, Oz, PO10, CP6,CP2, FC2, AFF6h. For experiments on or after November 11, 2022, our dataset contains the following EEG channels: AFF1h, F7, FC5, C3, T7, TP9, Pz, P3, P7, O1, O2, P8, P4, TP10, Cz, C4, T8, FC6, FCz, F8, AFF2h (omitting AFF5h, FC1, CP5, CP1, PO9, Oz, PO10, CP6,CP2, FC2, AFF6h) to meet the time constraint of each experiment.
Note that while the raw EEG data collected before November 11, 2022 contains a greater number of channels, for the purposes of consistency and comparative analysis, we confined our analysis to the channels that were present in the data both before and after the mentioned date. The channels we retained and used in the valence/arousal prediction benchmark analysis are the following: AFF1h, F7, FC5, C3, T7, TP9, Pz, P3, P7, O1, O2, P8, P4, TP10, Cz, C4, T8, FC6, FCz, F8, AFF2h. By streamlining our channels, we aimed to ensure more efficient data collection and analysis, while maintaining the rigor and validity of our findings.
The following preprocessing and cleaning steps were performed for the data in the SQLite database:
Minecraft issues For instances where a Minecraft mission has to be restarted due to technical issues, we remove the data from the previous (interrupted) mission, and only keep the data from the new (uninterrupted) mission.
Physiological measure data for confederates We remove physiological measure data from group session tasks in which a confederate stepped in to fill the role of a no-show participant or a participant that left in the midst of a group session if that confederate did not put on an EEG/fNIRS cap or eye tracker.
The raw data is saved on our lab servers. While we do not rule out sharing it (sans the identifiable data) with the public in the future, we believe that its size (just shy of 20 TB), complexity, and unclean nature makes its utility to downstream consumers limited enough to not justify the considerable logistical effort required to share it at the moment.
We do provide cleaned data though through our SQLite database, which consists of non-PII data that has been checked to be valid. We also provide the scripts that were used to clean the data.
The software used to preprocess/clean label the data is available at https://github.com/ml4ai/tomcat.
- The code for building the SQLite database from the raw data is in the human_experiments/datasette_interfacedatabase.
- The code for creating the processed datasets for the benchmark analyses are in the human_experiments/lab/software/tomcat-physio-data-extraction and tomcat-physio-data-extraction-v2 directories. The results from these are then further processed by the code in the data_products/scripts directory. This code effectively segmented the data into manageable chunks, specifically isolating the task data relevant to our benchmark analyses. Also, the code for constructing the synchronized physio data from the raw data is in the human_experiments/lab/software/tomcat-physio-synch directory. This synchronizes all the physio data.
The dataset has not been used for any tasks already (besides the benchmark analyses in the main paper).
Links to papers and systems that use the dataset are listed at https://tomcat.ivilab.org/papers-and-systems.