dataesr/scientific-paragraphs-categorization huggingface dataset:
In this dimension, there were more over-assessments for parents who worked part-time or full time than for parents who did not work, perhaps because parents who did not work had more time to pay attention to their children's psychological well-being. For the "autonomy and parents" dimension, there were more under estimations if the responding parent was single or if it was the original couple than if the parents had a new partner. It might be that parents with a new partner were more concerned about their child's relationship with them. This study has some limitations, including the limitations of PIMS2-CoV19 study. First, an online survey method was used: the data were collected using electronic self-report questionnaires. This could lead to selection bias, excluding persons without internet access but this is a minor bias in our time.
A total of 3997 students were randomly selected among all sixth and seventh grade's students to complete the Academic Diligence Task and a student-reported questionnaire during normal school hours. One teacher per class completed the Character Growth Card questionnaire for each student. In addition to these measures of non-cognitive skills, we also collected data from school records. For each student, we recorded the number of late arrivals, the number of absences, the number of sanctions, and the number of disciplinary actions during the school year (sanctions are often hours of detention, while disciplinary actions are more severe than sanctions and are decided collectively by the school administration). We also recorded their math and French GPA. Students were asked how much time they had spent on their homework in the last two days.
The data collection process took place over several years, allowing for a sub-sample of students to be randomly drawn in both their sixth and seventh grade. Students were 13 years old on average (SD = 0. 78), 88% were of French nationality, and 52% benefited from financial aid, which is about 14 points above the national rate. On average, students arrived 4. 8 times late at school during the year (SD = 8. 3), has a total of 3. 0 days of unjustified absences (SD = 6. 8), received 3. 5 sanctions (SD = 7. 5) and 0. 3 disciplinary actions (SD = 1. 2). We standardized, inverted and summed these four measures to create a disciplinary index, higher values indicating more disciplined behavior. French and math average GPA were respectively 12. 1 and 11. 9 (SD = 3. 4 and 3. 9). From the student-reported questionnaire, we obtained a measure of conscientiousness, self-control, and grit.
From the teacher questionnaire, we obtained a measure of self-control and grit. From the behavioral task, we retrieved the number of questions attempted, the number of questions correctly solved (productivity), and the percentage of time spent solving questions versus watching videos. These variables measure the ability of students to work diligently while resisting distractions, and have been shown to be correlated to grit, self-control, and conscientiousness. All the data analysis was conducted using the statistical software STATA. To compare these measures, we first assessed their reliability, i. e. whether each measure is consistent across time, raters, and items, using long-term stability and Cronbach's Alpha. Second, we assessed the validity of each method by testing whether the measures of socio-emotional skills correlate with behaviors related to the same psychological constructs.
Method The experimental protocol was approved by the Ethical Research Committee of the JPAL (Abdul Latif Jameel Poverty Action Lab) in Paris. All methods were carried out in accordance with relevant guidelines and regulations. Participants. We collected data in a sample of 97 French REP middle schools located across the country. Written informed consent was obtained from all subjects or, for subjects under 18, from a parent and/or legal guardian. Seven students per class were randomly selected among all sixth and seventh grade's students to par-Table  2. Breakdown of experimental costs. This table shows the breakdown of costs for each method of measure. Information includes costs pertaining to phoning schools and printing information for consent. Support includes the cost of paper and mailing for teacher questionnaires and digital tablets for student questionnaires and for the behavioral task.
The total cost per sd per student is the cost that is needed to observe one standard deviation in the change of behavioral index for one student. www. nature. com/scientificreports/ ticipate in our study. Students were 13 years old on average (SD = 0. 78), 88% were of French nationality, and 52% benefited from financial aid, which is about 14 points above the national rate. Data collection was embedded in a larger study aimed at measuring the impact of a low intensity intervention in middle school students to improve academic achievement  25. To ensure no confounding with the larger study intervention, we sampled students only from the control group. All students in the experiment completed the questionnaire and the behavioral task, therefore missing values came from random issues with extractions from school records.
We excluded a total of 2,006 students from our sample for whom data was missing. However, to ensure that our results were not affected by attrition, we conducted the same analysis on the total sample using imputations and found no difference (see Appendix Figures C8 to C11). The data collection took place over three years. The first cohort consisted of 784 students in the sixth grade during the spring of 2015. The second cohort consisted of 1,166 students in sixth grade and 1,117 students in seventh grade in the spring of 2016. The third cohort consisted of 930 students in seventh grade in the spring of 2017. A total of 3,997 separate measures were included in our study. Among those, a subset of these measures come from the same students who were randomly selected in both the first and second cohort, or in both the second and the third cohort.
Experimental design. Research assistants were dispatched across the French territory to collect data in each middle school. After training, research assistants collected administrative data from the school and administered both the behavioral task and the student questionnaire during normal school hours. Students each received earbuds and a digital tablet to complete both the questionnaire and the behavioral task. Research assistants also distributed the teacher questionnaire in paper format to one teacher per class and collected the answers a few days later. Research assistants and the teachers involved in the study were blind to the purpose of the experiment and the hypothesis being tested.
Measures of behavioral outcomes and academic achievement. For each student, we extracted data on disciplinary outcomes from school records: number of late arrivals, number of absences, number of sanctions, and number of disciplinary actions. On average, students arrived 4. 8 times late at school during the year (SD = 8. 3), has a total of 3. 0 days of unjustified absences (SD = 6. 8), received 3. 5 sanctions (SD = 7. 5) and 0. 3 disciplinary actions (SD = 1. 2). We standardized, inverted and summed these four measures to create a disciplinary index, higher values indicating more disciplined behavior. Each student was also asked to report the time she spent on homework in the last two days in the student-reported questionnaire. Given that time spent doing homework may be sensitive to the day of the week, we recorded the day of the week when the student completed the survey and added it as a control.
Procedure and Task Participants were seated in a quiet room at an approximate distance of 60 cm from the computer screen (15. 6") that produced and recorded experimental events via Psycophy Software  (Peirce, 2008). Explicit and implicit timing tasks comprised the same stimulus material and general procedure but differed in the specific task instructions given to participants (Figure  1 , A and Figure  2, A ). For both tasks, stimuli consisted of a grey circle and a grey cross presented at the centre of a lighter grey background screen. A thin circle was initially displayed for 500 ms (Inter-Trial-Interval, ITI), followed by a thicker circle that could assume one of the following interval durations: 480, 720, 960, 1200, 1440, 1680, or 1920 ms. After the duration had elapsed, a cross appeared in the centre of the circle for 500 ms.
Implicit timing task. As for the explicit timing task, model comparison (Table  2 ) showed that the best fitting model was again the full model including Interval duration, MMSE and Age variables. Visual inspection of the residuals showed that the model was a bit stressed. Thus, trials with absolute standardized residuals higher than 2. 5 standard deviations were considered outliers  (Baayen & Milin, 2010)  and removed (3% of the trials). The full model was, then, refitted on the resulting dataset (a summary of the model output is presented in Table  4 ). Figure  4.
Acknowledgements: This work was carried out within the scope of the project " Use-inspired basic research ", for which the  Department of General Psychology of the University of Padova  has been recognized as " Dipartimento di Eccellenza " by the  Ministry of University and Research. The authors gratefully thank  Diletta Piazzesi ,  Martina Corda ,  Davide Tempesta  and  Alessandro Vicenzotti  for helping with data collection. Conceptualization: MC, GM; Data curation: AV, GM; Formal analysis: AV; Funding acquisition: GM; Investigation: GM, AF, LM; Methodology: MC, AV, GM; Project administration: GM; Resources: GM; Software: AV; Supervision: MC, GM; Visualization: AV; Roles/Writing -original draft: MC, AV; Writing -review & editing: MC, AV, AF, LM, GM.
Authors in  [41]  use stochastic gradient descent and adopt backpropagation to update the weights matrices. By using the SEED-V dataset,  2  which provides not only electroencephalography (EEG) signals but also eye movement features recorded by SensoMotoric Instrument (eye-tracking glasses  3  ), authors obtained an accuracy of 82. 11% and standard deviation of 2. 76%. In another work  [42] , authors affirm that ''featurelevel fusion methods cannot deal with missing or corrupted data, while decision-level fusion methods may lose the correlation information between different modalities''. For that, a Hierarchy Modular Neural Network (HMNN) is proposed. HMNN recognizes the result according to the activity level of each module (each class is a module), and the one with the maximal activity wins the competition, which is called the winner-take-all strategy.
Experiments using DECAF dataset,  4  showed an improvement with respect to the state-of-the-art and its effectiveness in dealing with missing or corrupted data. In the literature, there are fusion methods that are applied in different domains and use other types of sources (Table  2 ). For instance, the document classification, based on text and visual data  [47] ; in a medical context, for early detection of Alzheimer's disease stage using magnetic resonance imaging, genetic data, and clinical test data  [48] ; in music videos  [49] ,  [50] ; or for multimodal languages  [51] -  [53]. Even though the focus of these works have not been the emotion recognition, they propose interesting network architectures with additional layers to combine multi modalities and outperforms the single modalities.
Afterward, the MMTM and SE (squeeze and excitation) networks are used for information sharing and boosting during training. Earlier the same authors built their own music video dataset and demonstrated that the multimodal results show improvement in various evaluation matrices over unimodal performance  [50]. The proposal presented in  [51]  is focused on the crossmodal attention mechanism, which provides a latent crossmodal adaptation that fuses multimodal information by directly attending to low-level features in other modalities. In  [52] , authors model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. Also, they propose a Recurrent Attended Variation Embedding Network to model the fine-grained structure of nonverbal subword sequences and dynamically shifts word representations based on nonverbal cues.
An interesting result is the difference of performance when the speech network is trained on male and female speakers' samples separately, the obtained accuracy is better compared to the combined dataset. Results in that work show that females are more expressive than male and combining image and speech modalities improves the recognition rate. In  [59] , a robot Pepper is used to improve HRI with users with physical disabilities, based on the recognition of emotions from videos and speech. In  [60] , it is proposed a model able to recognize emotions from text and store this information in a semantic repository, based on an ontology to represent emotions. It is considered as a multimodal approach since text can be obtained directly as text or by converting speech in text. Without pretending to be an exhaustive review, these studies reveal some limitations and some challenges are still open in the area of HRI.
The original version of XLNet is a model that has reached state-of-the-art results in many applications. It consist of an auto-regressive language model based on the transformer architecture which, given a sequence, outputs the probability of the sequence of words to follow. 7  DialogXL improves on XLNet by using an enhanced memory to store longer historical context during dialog and Dialog-Aware Self-Attention to keep track of the different speakers in a conversation. Each sentence by a speaker (utterance) is passed through an embedding layer which tokenizes the sentence into a sequence of vectors as shown in Figure  4. This representation is then passed through a stack of neural networks where each layer contains a Dialog-Aware Self-Attention component and an Utterance Recurrence Component; each of these layers output a vector that is passed to the following layer.
V. EXPERIMENTAL EVALUATION This section shows the performance of the proposed system compared with two state-of-the-art methods proposed by Poria et al. [36]  and by Mittal et al. [34]. These methods report results with the same dataset we use, making them suitable for comparison with the proposed system, although they are not focused on or applied to social robotics. First, the dataset and its pre-processing are explained; then, the training process of the individual models and the fusion methods are described; afterwards, the applied experiments to validate and evaluate the models and the obtained results are presented. All the aforementioned procedures were developed in a Google Colab environment.
A. DATASET PRE-PROCESSING After literature review, the IEMOCAP  [65]  dataset was chosen to train and test the individual and fusion methods. This dataset is one of the main benchmarks for multimodal emotion recognition that is available with no trouble. IEMO-CAP dataset has annotated several data types, such as motion capture face information, speech, videos, head movement and head angle information, dialogue transcriptions, and the word-level, syllable-level and phoneme level alignment. For carrying out the whole pre-processing task (as well as the training and validation procedures), every frame was cut to contain only the annotated person and discard noise data. Regarding the text modality in IEMOCAP, it was obtained by translating speeches into text, that was performed by human professionals.
Although this is not ideal for robotics scenarios, in which such translation is automatically performed by an specialized software application, we used these available data, since IEMOCAP is highly used to train and test emotion recognition models, from which we can compare our approach. For detection of faces, the RetinaFace model  [66]  was used because of its accuracy and ease of use. This model is a robust single-stage face detector that performs pixelbased face localization by taking advantage of joint extrasupervised and self-supervised multi-task learning and using a lightweight backbone (ResNet or MobileNet). When cutting the videos in the annotated sub-videos, several segments were lost because the tools used did not work correctly, causing that the face modality has a much fewer number of samples for training and testing.
B. TRAINING OF MODELS IEMOCAP has its data distributed into five sessions, such that, data from the first four sessions were used for models  VOLUME 10, 2022  training, remaining the fifth session just for the test. For facial modality there are 19384 images, obtained after preprocessing (2423 samples × 8 frames each), from which the ∼ 78. 79% were for training, and the rest (∼ 21. 21%) for the test. For audio modality, there are 5531 samples, where the ∼ 77. 56% were used for training and the ∼ 22. 44% for the test. The DialogXL text model, used to analyze text, is pretrained on the IEMOCAP training dataset, just like the other models in this work and uses six emotions (happiness, neutral, sadness, anger, excitement, and fear). During the original training the AdamW optimizer was used and the tunable hyperparameters were the learning rate, the number of heads for the four types of attentions used in the dialog-aware selfattention component, the max length of memory, and the dropout rate.
However, since we decide to work with only four emotions, it was necessary to adapt the text model to this. Thus, we merge happiness with excitement to balance the number of samples in the dataset and the fear-related samples and predictions were removed for the training and for further fusion. Thus, the text modality has 5188 samples, from which ∼ 79. 76% were used for training and the ∼ 20. 24% were used for test. Face and audio models were trained from scratch with the four emotions considered. For training, both models use cross-entropy loss function and the Adam optimizer with a base learning rate set at 0. 001, but for the audio model, a scheduler was also used to reduce the learning rate up to 0. 000001. The face model presented overfitting, exposed by the increase of validation loss values; thus, the training was stopped in epoch 22 to avoid propagating the errors.
The audio model was trained in 60 epochs. The big advantage in this case was the low level of processing when using a neural network with only 3 convolution layers and the data was converted from audio to images with pre-processing. Finally, the fusion method EmbraceNet+ was trained also with the Adam optimizer and using the cross-entropy loss function. This component takes every available sample from three modalities and the mismatched samples were taken as incomplete, by putting the missing modality as noisy or just absent. Following the number of emotions and the parameters required, the inner EmbraceNet that outputs the final prediction has been configured as  [16, 4, 12]  for first modified EmbraceNet, weighted sum, and concatenation, respectively, and a linear layer with four neurons.
C. EXPERIMENTS AND RESULTS The three individual models and the entire composite system are evaluated separately. Every evaluation is made with common classification metrics: accuracy and F1 score. The accuracy of individual models is shown in Table  3. For face modality the accuracy obtained is 44. 0%, 58. 3% in audio modality, and for text modality is 83. 5% (after reducing to four categories). These results show a deficiency in facial mode due to the smaller number of samples and a bit of overfitting with the training samples. Nevertheless, the model used is robust enough to be considered a contributor to our multimodal system. The poor face modality results might be also related to the data nature (videos) and the processing way (selected frames) because some noise data is introduced without a filter or discriminator.
Likewise, the improved results for text modality could be influenced by the quality of the data. As mentioned before, the text data in IEMOCAP have been heavily curated and processed by professionals, so these data have less noise than video and audio modalities, which we had to pre-process ourselves. This fact naturally leads to better results in the text modality with respect to the other two. Hence, according to these experiments, due to the higher results of the text model, this modality could be sufficient on its own for practical applications, as it is also demonstrated by other authors  [53] ; however, the integration of other modalities is still necessary to provide flexibility and robustness  [51] ,  [52]. While results may vary depending on the fusion method used, using other modalities to complement the text input allows the model to deal with ambiguities or samples where there is not enough information given in the text and a context is necessary.
Due to the text is a transcription of the audio data, the text modality only could be with troubles if the audio data also have problems in the recording. Having both audio and text modalities missing or with problems is the worst-case scenario for our proposed system. To show the results of the proposed approach, a set of comparisons with the results reported by Poria et al. [36]  TABLE 4. Comparisons between state-of-the-art methods in IEMOCAP dataset  [65]. (accuracy) and Mittal et al. [34]  (F1 and accuracy) are carried out, as presented in Table  4  and Figure  7. These methods do not detail how the samples of the happiness category were increased; Poria et al. [36]  report the use of 1630 samples for happiness, which indicates that they should do some data increase, such as merging happiness and excitement or applying some oversampling technique.
These evaluations may show a little dependency on the text modality, which is understandable as FIGURE 7. Graph comparison of reported accuracies (axis y) by category and average over four evaluations (axis x), between our results and poria et al. [36]. this modality is the most informative in terms of people's feelings and has a complex and well-established method behind it. The method from Mittal et al. (M3ER)  [34]  remains as the better method in the IEMOCAP dataset for the combination F+A+T, with 0. 82 of F1 score and 82. 7% of accuracy (considering the happiness category), which means 0. 03 and 5. 1% over our results in the same metrics (see Table  4 ). Mittal et al. [34]  do not detail the number of samples used in their study. However, this comparison is just referential because it is unclear how many samples of each category this model use.
A. DATASET PRE-PROCESSING One of the greatest challenges encountered during the development of this work, was obtaining a dataset from which our models could learn. Most datasets were missing at least one of the modalities we tested in our model; for example, VAM  [67]  and RECOLA  [68]  consider just audio and visual data, and EmoDB  [69]  contains only audio data. After we reviewed different datasets, we decided to use the IEMOCAP, a well-known and widely used dataset. There exist a few others, such as CMU-MOSEAS  [70]  and MELD  [71]  that we plan to test to better validate our approach in future research. CMU-MOSEAS was released in 2020. This dataset, as well as IEMOCAP, is an emotion repository, but it also includes sentimental data. CMU-MOSEAS covers four languages: English, Spanish, Portuguese, and French.