SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With some additional annotations effort, the dataset could also be used for the following tasks:
Sentiment analysis
- Emotion recognition
- Plan recognition
- Entrainment detection
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)? If so, please provide a description. Is there anything a dataset consumer could do to mitigate these risks or harms?
Future analyses of the derived data products may be affected by the specific filtering/interpolation strategies used. We do not currently know of specific uses of the dataset that could result in unfair treatment of individuals or groups.
We do not currently know of specific tasks for which the dataset should not be used, but users should in general strive to make sure that their usage of the dataset does not perpetuate systemic inequalities. The dataset should not be used to attempt to re-identify individual participants.
Will the dataset be distributed to third parties outside of the en- tity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
The dataset will be distributed publicly.
The dataset is available at https://tomcat.ivilab.org. See Appendix C for more details.
We have not set up a DOI for the dataset yet, since the size of the dataset is too large for commonly used DOI providers (e.g., Zenodo, Figshare). We are working with CyVerse to set up a DOI. In any case, we do not expect the current URL (https://tomcat.ivilab.org) to change in the foreseeable future.
The initial release of the dataset is already public at https://tomcat.ivilab.org. We will add additional raw and derived data in the near future.
The data is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license (https://creativecommons.org/licenses/by-nc-sa/4.0). The software is licensed under the MIT License.
No third parties have imposed restrictions on the data associated with the instances.
The dataset is not subject to export control or other regulatory restrictions.
The responsibility for supporting the dataset will lie with the ToMCAT project principle investigators Adarsh Pyarelal, Clayton Morrison, and Kobus Barnard. The data will be initially hosted on a web server in the Computer Science department at U. Arizona, and will be mirrored by CyVerse as curated data. CyVerse (https://cyverse.org) is a cyber infrastructure system led by U. Arizona, initiated by NSF in 2008, and has external and institutional support for curating data indefinitely.
The maintainers can be contacted by email: tomcat-dataset-mainainers@list.arizona.edu.
Errata will be published at https://tomcat.ivilab.org/errata.
The dataset will be updated periodically by the maintainers, with three types of updates.
Type of update	Frequency
Adding new raw data (e.g., from additional modalities as they get processed)	We expect a number of updates to be published in summer and fall 2023 as we process more of the raw data and upload data from additional modalities. It is unlikely (but not impossible) that we will add data from new participants to this particular dataset, post fall 2023.
Fixing previously uploaded data.	This is likely to occur often in summer and early fall 2023, and very infrequently post 2023.
Adding new derived data (e.g., from improved interpolation procedures)	We expect this to happen relatively infrequently, triggered by either (i) a new publication on the data or (ii) a request from a collaborator for a particular form of derived data.
Updates will be communicated to dataset consumers via the mailing list tomcat-dataset-updates@list.arizona.edu. They will also be published at https://tomcat.ivilab.org/updates.
There are no limits on the retention of the data.
We will rename and/or move older versions of the data to an archive location. As appropriate, we will communicate such changes to dataset consumers via the mailing list tomcat-dataset-updates@list.arizona.edu and at https://tomcat.ivilab.org/updates. We will continue to host older versions for a minimum of two years, and we will only remove data if it is judged to have no value moving forward. Obsolescence of such versions will be communicated to dataset consumers as above.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for commu- nicating/distributing these contributions to dataset consumers? If so, please provide a description.
Others wishing to extend/augment/build on/contribute to the dataset will be encouraged to email us at tomcat-dataset-mainainers@list.arizona.edu about their plans.
We will validate proposed contributions to some extent before integrating them into the hosted data. One possible category of contribution is a new way distill, transform, or represent the raw data, leading to a new derived data tables. Here we will require the code to replicate producing the new data. Another possible contribution would be a labeling of the data, such as manual transcription of speech data, or human evaluation of emotive speech. Here we will compare with existing versions and/or perform spot checks.
B Recruitment
Participants were recruited at the University of Arizona. All advertisement and recruitment materials provided contact information for the Temporal Interpersonal Emotion Systems (TIES) Lab and the corresponding Gmail address and Google Voice phone number. IRB-approved flyers were posted across campus and shared on social media and the project's website (see Appendix P).
To further reach the undergraduate student body at the university, study information was placed on the SONA system for Psychology as the project's PIs were affiliate faculty in the department. This allowed students to either receive credit for research experience in their lower-level Psychology courses and/or receive extra credit for participation in research experience in their Family Studies & Human Development (FSHD) courses.
Lastly, we also recruited through word-of-mouth and sending e-mails with IRB-approved language to undergraduate, graduate student, and faculty listservs and/or newsletters.
C Data format
The dataset uses two open and widely used data formats—namely, SQLite and CSV, both of which are recommended storage formats for datasets according to the Library of Congress [131].
We provide access to the data in two ways. The first is through a Datasette [121] instance, which provides graphical and programmatic interfaces for users to perform SQL queries to retrieve subsets of data they are interested in, or simply download the backing SQLite database. The second is through links to pre-built files containing selected parts of the data that we (i) used for the experiments in this paper, and (ii) expect will be commonly requested as good starting points for analysis. We provide all valid raw data and several derived data products. One key issue with using the raw data is that it is it is comprised of data from multiple asynchronous data streams. Hence, aligning the data for multiple participants entails interpolation, which we typically do for most of the derived data sets. A second issue is recording specific issues encountered during data collection, such as an experimenter stepping in for a participant who decided to leave in the midst of a group session. Finally, derived data will typically also include standard data transformations and cleaning. More details are available on the website itself.
D Structured metadata
Structured metadata using web standards (schema.org) has been added to the index page for https://tomcat.ivilab.org.
We have also added a route to the Datasette instance to enable quick inspection and programmatic access to the structured metadata: https://tomcat.ivilab.org/~/structured_metadata.json.
E Long-term preservation
We are committed to long term preservation and availability of this data. We have redundancy in our maintainers, all three of whom are faculty at the University of Arizona. We are able to host the data on new servers in two different academic units. Initially, we will host the primary data source on a web server maintained by the U. Arizona Computer Science Department. We will also mirror the data on the CyVerse curated data repository. CyVerse (https://cyverse.org) is a cyber infrastructure system led by U. Arizona, initiated by NSF in 2008, and has external and institutional support for curating data indefinitely.
F Reproducibility
We use the ML reproducibility checklist v2.0 (https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf). Since we do not present any new models, algorithms, or theoretical claims in this work, we skip the corresponding items in the checklist.
F.1 CNN experiments
fNIRS and EEG were trained and tested separately using Convolutional Neural Networks (CNN) models.
For all datasets used, check if you include:
$\checkmark$  Relevant statistics We use the data from the individual affective task, encompassing five distinct classes  $(-2, -1, 0, 1, 2)$  that represent valence and arousal scores. The dataset consists of fNIRS recordings obtained from 102 subjects and EEG recordings obtained from 97 subjects. The disparity in subject counts can be attributed to instances where the EEG amplifier encountered technical issues, resulting in a failure to record data for certain individuals.
$\checkmark$  Details regarding train/validation/test splits. To ensure robust model evaluation, the dataset is divided into an  $80\%$  training set and a  $20\%$  testing set, employing a 5-fold cross-validation approach, where each fold acts as a validation set for training the model.
$\checkmark$  An explanation of any data that were excluded, and all pre-processing steps. Within the affective task dataset, encompassing both EEG and fNIRS data, we find two essential columns: valence_score and arousal_score. These columns capture the participant's subjective ratings of images during specific time intervals. However, it is important to note that numerous entries within these columns may currently lack values and remain empty.
We used the pandas [132, 133] fillna function with the bfill method in this code to fill those empty entries with appropriate values. By using bfill, the missing values in these columns were filled with the next available value from the same column, going backwards. This ensures that the missing values are replaced with values that are likely to be similar to the previous ratings given by the participants.
To transform signals into images for CNN processing, we may need to perform an offset, particularly when accounting for the Hemodynamic Response Factor in fNIRS data. This offset determines the starting point for data extraction corresponding to each image viewed by the participant. For example, setting an offset of 10 means that the first 10 rows of data for each unique stimulus-response instance are skipped, and extraction begins from the 11th row. By default, the offset is set to 0.
To create images, we must consider multiple rows of data, defined by the window size parameter. Setting a window size results in the extraction of a data segment for the length of the window size following the stimulus presentation. For example, if the window size is set to 10, it would extract 10 rows of data from the segment in which the participant is rating an image.
The images generated are subsequently annotated based on the predominant valence and arousal scores observed within their respective windows.
$\checkmark$  A link to a downloadable version of the dataset or simulation environment.
The dataset used for these experiments is available at https://tomcat.ivilab.org/
Table 4: CNN architecture for fNIRS data.  $W$  is the size of the window.
Layer (Type)	Output shape	# of parameters
Conv2d-1	[-1, 16, W, 22]	160
BatchNorm2d-2	[-1, 16, W, 22]	32
MaxPool2d-3	[-1, 16, W/2, 11]	0
Conv2d-4	[-1, 32, W/2, 11]	12,832
BatchNorm2d-5	[-1, 32, W/2, 11]	64
MaxPool2d-6	[-1, 32, W/4, W/4]	0
Conv2d-7	[-1, 64, W/4, W/4]	100,416
BatchNorm2d-8	[-1, 64, W/4, W/4]	128
MaxPool2d-9	[-1, 64, W/8, W/8]	0
AdaptiveAvgPool2d-10	[-1, 64, 1, 1]	0
Linear-11	[-1, 128]	8,320
Dropout-12	[-1, 128]	0
Linear-13	[-1, 64]	8,256
Linear-14	[-1, 5]	325
Linear-15	[-1, 5]	325
derived-data-products. After extracting the data, you will find two folders named fnirs_10hz and eeg_500hz. Within each of these folders, there will be an experiment folder. Inside the experiment folder, you will find files named affectiveindividual\*.csv.
$\checkmark$  For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control. The data collection process is detailed in the main paper and in Appendix H.
For all shared code related to this work, check if you include:
$\checkmark$  Specification of dependencies. The dependencies are listed in the requirements.txt file of the directory code/ToMCAT-ML-Modeling in the supplementary material.
$\checkmark$  Training code. The training code is included with the supplementary materials (see the directory code/ToMCAT-ML-Modeling)
$\checkmark$  Evaluation code. The evaluation code is included with the supplementary materials (see the directory code/ToMCAT-ML-Modeling).
$\checkmark$  (Pre-)trained model(s). We do not provide pre-trained models, since our CNN models are relatively small and quick to train, and since we provide the training data as well.
$\checkmark$  README file includes table of results accompanied by a precise command to run to produce those results.
For all reported experimental results, check if you include:
$\checkmark$  The range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper-parameters used to generate results.
The architecture and configuration of the CNN models we used are summarized in Table 4 and Table 5. Notably, the EEG CNN model incorporates an input shape of [W,56]. Here,  $W$  represents the window size. The value 56 is derived from 7 EEG channels, each divided into 4 frequency bands (Theta, Alpha, Beta, Gamma). Thus, we obtain 28 values (4 frequency bands per channel). From these, wavelet features are extracted and then horizontally appended, resulting in 56 columns of EEG data. In contrast, the fNIRS CNN model has an input size of [W,22], which represents the 11 HbO and 11 HbR fNIRS channels considered. Both the EEG and fNIRS CNN models share a common number of classes, set to 5.
For the fNIRS CNN model with a 2-second window and a 2-second offset, the optimal configuration was found with 15 epochs and a batch size of 50. This configuration, specifically for the 0-second offset, achieved an accuracy of  $30.4 \pm 1.2$  for valence and  $30.3 \pm 0.8$  for arousal, with a loss of  $3.02 \pm 0.008$ . The confusion matrices for fNIRS arousal and valence are presented in Figure 3.
Table 5: CNN architecture for EEG data.  $W$  is the size of the window.
Layer (Type)	Output shape	# of parameters
Conv2d-1	[-1, 16, W, 56]	2,000
BatchNorm2d-2	[-1, 16, W, 56]	32
Conv2d-3	[-1, 32, W-27, 56]	896
BatchNorm2d-4	[-1, 32, W-27, 56]	64
ELU-5	[-1, 32, W-27, 56]	0
AvgPool2d-6	[-1, 32, W-27, 14]	0
Dropout-7	[-1, 32, W-27, 14]	0
Conv2d-8	[-1, 32, W-27, 15]	16,384
BatchNorm2d-9	[-1, 32, W-27, 15]	64
ELU-10	[-1, 32, W-27, 15]	0
AvgPool2d-11	[-1, 32, W-27, 3]	0
Dropout-12	[-1, 32, W-27, 3]	0
Linear-13	[-1, 128]	11,956,352
ELU-14	[-1, 128]	0
Linear-15	[-1, 5]	645
Linear-16	[-1, 5]	645
(a) fNIRS Arousal Confusion Matrix
(b) fNIRS Valence Confusion Matrix
Figure 3: Confusion matrices for arousal and valence prediction based on fNIRS data with a 1-second window and a 2-second offset.
Regarding EEG CNN model with a 1-second window and a 0-second offset, the optimal configuration was found with 15 epochs and a batch size of 50. This configuration, specifically achieved an accuracy of  $29.3 \pm 1.4$  for valence and  $29.8 \pm 1.3$  for arousal, with a loss of  $2.8 \pm 0.004$ . The confusion matrices for EEG arousal and valence are shown in Figure 4. The hyperparameters used for the fNIRS and EEG CNN models include a learning rate of 0.01, a batch size of 50, and training for 15 epochs.
✓ The exact number of training and evaluation runs. We conducted a total of 7 training and evaluation runs for each model. These runs were performed using a 5-fold cross-validation strategy, ensuring comprehensive and robust assessment. During each run, we evaluated the models based on two key metrics: classification accuracy and loss.
$\checkmark$  Definition of the specific measure or statistics used to report results.
- The average accuracy and error of the mean are calculated from all folds for the arousal score.
- The average accuracy and error of the mean are calculated from all folds for the valence score.
- Average loss per fold and error of the mean are calculated from all folds for the valence score.
(a) EEG Arousal Confusion Matrix
(b) EEG Valence Confusion Matrix
Figure 4: Confusion matrices for arousal and valence prediction based on EEG data with a 1-second window and a 0-second offset.
- Confusion matrix for valence score and arousal score.
These measures provide a clear understanding of the reported results, including accuracy and variability for both arousal score and valence score, as well as the average loss and its variability across different folds. Additionally, the confusion matrix provides insights into the classification performance for valence score and arousal score.
$\checkmark$  A description of results with central tendency (e.g.mean) & variation (e.g errorsbars). For arousal score, the average accuracy is reported as the mean, indicating the central tendency of the data. The standard deviation is provided as a measure of variation, reflecting the spread or dispersion of the accuracy scores. This helps to understand the consistency or variability in the performance of the model for arousal score.
Similarly, for valence score, the average accuracy is presented as the mean, representing the central tendency. The standard deviation serves as a measure of variation, illustrating the extent of variability in the accuracy scores for valence score.
The average loss per fold provides the central tendency measure for the model's performance in terms of loss. It gives an insight into the average magnitude of errors made by the model across different folds. The standard deviation of the loss per fold helps to understand the variability or dispersion of the loss values, indicating how consistent or varied the model's performance is across different folds.
$\checkmark$  The average runtime for each result, or estimated energy cost. The training of the fNIRS CNN model required 1.5 seconds for each fold of cross-validation, utilizing 24 GB of RAM and 2.5 GB of GPU memory. In contrast, the EEG model took 9 seconds per fold, consuming 35 GB of RAM and 7 GB of GPU memory.
$\checkmark$  A description of the computing infrastructure used. The models were trained on high performance computing cluster with dual AMD EPYC 7542 32-Core processors (3.3 GHz), 1 TB of RAM and two Nvidia A100 GPUs.
F.2 Correlation experiments
$\checkmark$  Relevant statistics. Tables 6 and 8 present the number of (group session, task) combinations considered and accepted for computing fNIRS correlations among participants, and tables 7 and 9 present the experiments rejected and the reasons for their rejections. Some experiments were not included in the final results because they do not have physio data recorded for one of the tasks. Among all experiments that were considered for the Ping Pong Cooperative task, one experiment was rejected because of its abnormally high task score.
Furthermore, unlike the physio task data synchronized dataset we discuss in § J.1 that provides  $1000 \mathrm{~Hz}$  synchronized EEG-EKG-GSR data, our physio correlation analysis
Table 6: Experiments considered and accepted into the computation of results for fNIRS data.
Task	Experiments accepted	Total # Experiments considered
Ping Pong Cooperative	38	40
Minecraft Saturn A	36	36
Minecraft Saturn B	27	27
Table 7: Group sessions for which the data from the Ping Pong Cooperative task were rejected from the computation of results for fNIRS channels.
Group session	Reason for Rejection
exp_2023_01_30_13	Missing fNIRS data
exp_2023_01_31_14	Task score greater than 20
experiment uses synchronized EEG-EKG-GSR data synchronized at  $500\mathrm{Hz}$  and fNIRS synchronized at  $10\mathrm{Hz}$ .
$\checkmark$  Details of train/validation/test splits. NA
$\checkmark$  An explanation of any data that were excluded, and all pre-processing steps. Notably, there were three experiments that were not processed by both the physio synchronization program and the physio correlation analysis program. Two of these experiments contain Minecraft mission data that require additional cleaning to be useful. The last experiment contains file structure that also nseeds to be cleaned.
During the physio correlation analysis process, three more experiments were chosen to be removed from the analysis. These experiments contain very high physio channel correlation values but their Minecraft scores were extremely low. These experiments were considered noise and were removed from the physio correlation analysis.
$\checkmark$  A link to a downloadable version of the dataset or simulation environment. The intermediate dataset (chunked data) used for these experiments is available at https://tomcat.ivilab.org/derived-data-products.
$\checkmark$  For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control. The data collection process is detailed in the main paper and in Appendix H.
$\checkmark$  Specification of dependencies. The dependencies are listed in the requirements.txt file of the code repository.
$\checkmark$  Training code. N/A
$\checkmark$  Evaluation code. The evaluation code is included in the supplementary material. (code/tomcat-signal-correlation-analysis-main, code/synchronize_signal_task, and code/signal Filtering). Specifically, the evaluation metric is  $R^2$ , which evaluates how well linear regression predicts the team score from physio correlation.
$\checkmark$  (Pre-)trained model(s). N/A
$\checkmark$  README file includes table of results accompanied by precise command to run to produce those results. A README file is included in the code repository, but it does not include a table of results.
$\checkmark$  The range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper-parameters used to generate results. NA
$\checkmark$  The exact number of training and evaluation runs. The evaluation is ran once.
Table 8: Experiments considered and accepted into the computation of results for EEG data.
Task	Experiments accepted	Total # of Experiments considered
Ping Pong Cooperative	36	40
Minecraft Saturn A	33	35
Minecraft Saturn B	26	27
Table 9: Experiments rejected from the computation of results for EEG data.
EEG with Task	Experiment Rejected	Reason for Rejection
Ping Pong Cooperative	exp_2022_10_24_12	Missing EEG data
exp_2023_01_30_13	Missing EEG data
exp_2023_01_31_14	Task score greater than 20
exp_2023_05_01_13	Missing EEG data
Minecraft Saturn A	exp_2023_05_01_13	Missing EEG data
exp_2023_05_02_14	Missing EEG data
Minecraft Saturn B	exp_2023_05_02_14	Missing EEG data
$\checkmark$  A clear definition of the specific measure or statistics used to report results. We evaluate the linear regression model using the R-square  $(R^2)$  metric, which indicates how well the the model can predict team score given physio correlation among participants.
$\checkmark$  Results with central tendency The linear regression has low  $R^2$  score overall.