dataesr/scientific-paragraphs-categorization huggingface dataset:
Finally, another important aspect is that features relevance for engagement classification may differ as a function of the participant's conversational role. Some studies have taken this into account by adding a binary variable indicating whether the participant is speaking or not in their model  (Dermouche and Pelachaud, 2018). But this does not allow contrasting models of speaker's and listener's engagement. Our study is limited to the automatic estimation of engagement level when participants are holding the turn. Technically, turns boundaries were automatically detected using a Python module (PyElan) that connects the ELAN annotation software with Python script especially designed for this study. This module allows parsing, modifying and creating ELAN files. It includes other functionalities such as extraction of speech activity or feedbacks time boundaries.
, 1974). A turn ends when the main speaker is pausing while her interlocutor takes the turn (as defined above). However, the turn does not end if the main speaker is still speaking: in this case of overlap, the turn is allocated to both participants. Turns shorter than one second were discarded because of the difficulty of annotating them as well as the risk of extracting unreliable features on such short intervals. The number of turns shorter than 1 s in the whole dataset amounts to 1,040. Overall, a total of 1,200 turns has been kept.
Annotation procedure The degree of engagement was manually annotated by two experts. The dataset being already existing, it has not be possible to ask the speakers for a self-reporting. However, this type of annotation is interesting if we think important to take into account the goals and values of the speaker, which is less relevant in the type of interaction (narration) we are working on. At this stage, we decided to annotate engagement for the speaker's production only, not the listener's one, which is limited to feedback. Of course, the number and the type of feedback produced by the listener may have a consequence on the speaker's production. We chose to focus on speakers only for two main reasons. First, a preliminary analysis of the corpus have shown a regular feedback production across the different listeners  (Boudin et al.
• Level 2: disengaged. • Level 3: neutral. • Level 4: engaged. • Level 5: strongly engaged. Annotations were made using ELAN software. For each speaker in the corpus, an ELAN file was created with the segments corresponding to the speaking turns, which means that annotators visualize this information in the corresponding Elan layer. For each segment, the annotators had to select an engagement value between 1 and 5 (as detailed above). Even though annotators were experts, we decided to keep generic instructions and the assignment of the engagement value a holistic manner. A final adjudication has been done between them to decide in case of conflict. Note that we decided to chose a classical 5-level scale in order to obtain more information in comparison with a binary classification. This granularity seems to us relevant, in particular because it allows to compare different granularities by merging different classes.
Feature extraction Our model relies on a limited set of features from verbal and non-verbal modalities: acoustic, mimogestural and morpho-syntactic cues. Note that some features have been annotated manually (gesture intensity and feedbacks) and that all annotations have been manually corrected. At this stage, this is of course a limitation when trying to implement the level assessment automatically. However, these restrictions are due in the first case to constraint coming from the quality and the framing of the video and in the second case to the absence of automatic tool for efficiently recognizing a feedback. We are working on these two aspects by acquiring new datasets and developing new tools.
Features selection In order to reduce feature space dimensionality, a subset of features is kept when performing classification. Features were selected based on the strength of their correlation with the degree of engagement, as measured by Pearson correlation coefficient (see Table  1 ). Note that feature selection has been applied to the entire dataset, instead of the training subset because of using nested crossvalidation, which limits the risks of over-fitting (see next section). Only features for which the absolute value of the coefficients was higher than 0. 1 were retained. Since we are interested in the potential relation between engagement and syntactic richness we decided to include modifiers and conjunctions frequency in our model even though the Pearson coefficient was small for both.
Regarding the Multilayer perceptron, the small size of our dataset encourages us to limit in practice the number of parameters. The architecture is therefore reduced to two hidden layers, the first with 64 nodes and the second with 32 nodes. Only the learning rate and the batch size were optimized using the nested cross-validation procedure. The number of epochs was set to 50. Taking into account the fact that this study does not have direct equivalent in the literature (measuring engagement in naturalistic human-human conversation at the turn level in a 5-level scale), we have decided to experiment the classic models, in order to compare as directly as possible our results with the state of the art. We also designed two different baselines: • Baseline 1: The classifier always predicts the majority class.
classes As aforementioned subsection 4. 1. 2 strongly disengaged turns rarely occur in our dataset. This incites merging level 1 and 2 when classifying engagement degree. Other studies have adopted this strategy for similar reasons  (Huang et al. , 2016). Annotators also sometimes reported difficulty when deciding between attributing level 4 or level 5. We decided therefore to perform a new classification task, adopting that time 3 levels of engagement (i. e. , level 1: strongly disengaged or disengaged, level 2: neutral and level 3: engaged or strongly engaged) rather than 5. Results of the 10fold cross validation procedure are displayed in Table  4. The SVM achieves an F-score of 0. 58, almost at the same performance as Logistic Regression and outperforming both the baseline and closest models (Multilayer perceptron and KNN) significantly.
Discussion All classifiers trained on our multimodal dataset significantly outperform the baselines in both the 5-class and the 3-class tasks. The best results were obtained with using logistic regression for the former, and with the two-layer perceptron for the latter. Limiting the number of classes to 3 by merging level 1 and 2 as well as level 4 and 5 leads to a small improvement in classification performance. Results for the 3-class task are comparable to the ones from previous studies. However it is worth noting that most of them were only interested in discriminating engagement from disengagement  (Hsiao et al. , 2012; Liu and Kappas, 2018; Ben-Youssef et al. , 2019).
Looking at engagement at the turn level Annotators' reports confirm the reliability of our turn detection algorithm: the small amount of incorrect turns were so because the participants were either reading the short story or talking with an experimenter. This supports its application to other corpora in order to increase the size of our dataset. In the literature, turn-level engagement has only been investigated in the context of human-agent interactions  (Forbes-Riley et al. , 2012). The researchers achieved a F-score of 0. 69 using a 2-level engagement scale. Our results from the 3-class classification task come close despite the more unconstrained nature of the interactions in our corpus and our more detailed engagement scale. The main limitation of such approach originates in the challenge of annotating heterogeneous turns with respect to engagement.
Mental averaging is likely to have introduced additional noise in the data. Indeed, turns longer than 10 s, for which difficulties of annotation were often reported, represent approximately 40% of our dataset. Another limitation of focusing of turns relates to comparability, since the majority of studies are based on annotatordefined segments as already mentioned. It would be interesting to compare results when annotating engagement this way and when doing so on the basis of turns using the same dataset. Future directions Gaze direction was not included in our dataset. Given its central place in the engagement literature  (Sidner et al. , 2004; Ishii et al. , 2021)  it may be fruitful to incorporating it in future models. Including gaze direction may be far from being a silver bullet however, as some authors have found that it performs worse than facial features  (Dermouche and Pelachaud, 2019).
Head pose data might also be worthy of investigation given the good results obtained by  Ooko et al. (2011) (0. 89 mean F-score)  when classifying engagement on a 3-level scale using a combination of head pose and head rotation. An additional interesting avenue would be to make use of AUs activation and intensity time series. The issue is that the framework used in previous studies using these features  (Liu and Kappas, 2018; Ben-Youssef et al. , 2019; Dermouche and Pelachaud, 2019 ) cannot be straightforwardly applied to our dataset to the extent that turns are of unequal sizes, which prevents from using an LSTM model with a fixed number of frames for example. Another area worth of investigation relates to model building. As aforementioned, annotators have reported intra-turn variability in engagement level for turns longer than 10 s.
This also means to analyze the interplay between participants productions, how one participant behavior influence the other. Compared to other works focusing on human-human interactions the size of our dataset (approximatively 1,200 turns after removing incorrect turns and turns for which feature values were missing), is fairly small. Increasing it might benefit the performance of our model. Another corpus of natural dyadic human-human conversations is currently being annotated. Once IPUs, feedbacks, smiles and head nodes are annotated, it will be possible to extract turns and then proceed to engagement annotation. At a smaller (and technical) scale, another way to alleviate this issue could be to use data augmentation techniques such as Smote as done in prior studies of engagement  (Fedotov et al.
Three-dimensional (3D) real-time object detection and tracking is an important task in the case of autonomous vehicles and road and railway smart mobility, in order to allow them to analyze their environment for navigation and obstacle avoidance purposes. In this paper, we improve the efficiency of 3D monocular object detection by using dataset combination and knowledge distillation, and by creating a lightweight model. Firstly, we combine real and synthetic datasets to increase the diversity and richness of the training data. Then, we use knowledge distillation to transfer the knowledge from a large, pre-trained model to a smaller, lightweight model. Finally, we create a lightweight model by selecting the combinations of width, depth & resolution in order to reach a target complexity and computation time.
• Instance segmentation associates each pixel with a class to uniquely identify it. The segmentation trims the object and is more accurate than the bounding box detection. Among all these approaches, in our case of autonomous cars and trains, 3-dimensional object classification is the most relevant method. In addition to this, additional challenges have been identified. The first one is the availability of databases in the railway and road domains, rich enough (both in terms of the number of images and data as ground truth) to allow the adequate training of deep learning methods. While there are plenty of datasets covering the road domain, datasets covering the railway domain for object detection are much less numerous. As our work used 3D detection, what we needed was datasets with not only ground truth data about the objects' classes and positions on the pictures, but also their real location, dimensions, and orientations.
Our aim was to provide an answer to both challenges by making full use of dataset combination, and by developing and improving a monocular 3D object detection and tracking system that can provide real-time performance, even on embedded systems, with limited processing resources. This paper is organized as follows: Section 1 introduces this paper. In Section 2, we review the related works, YOLO-based object-detection algorithms, existing datasets for smart mobility, methods to extract synthetic data, object tracking, and, finally, knowledge distillation. In Section 3, we introduce the work done by our team in regard to our improvements on YOLOv5-3d  [1]  using tracking visualization, and dataset combination. The implementation of our algorithm on an embedded system for field usage is described in Section 4.
Datasets The volume and quality of data in a learning problem remain a fundamental issue, as well as the structure of the model to be trained. The field of autonomous cars, being in full development, has a multitude of data sets for the road domain, such as KITTI  [20] , which provides images from a stereoscopic camera, with the depth of the scene measured by a Velodyne LIDAR, as well as vehicle and pedestrian annotations for object detection, or NuScenes  [21] , which has six cameras and a LIDAR with a large volume of data that can be used for 2D and 3D detection and tracking. There are also other datasets available, like CityScapes  [22] , Pascal VOC  [23] , ImageNet  [24]  or MS-COCO  [25] , but they were not used during our work because they did not contain the information we needed. We used KITTI and NuScenes, mainly because they are both open-source datasets dedicated to 3D detection for autonomous vehicles and both contain sequences of videos from urban environments.
These datasets were especially useful for this project because they feature most of the situations which might be encountered during real-life use of the algorithms. In addition, there are simulators that allow automatic acquisition and annotation, such as in CARLA  [26]  or SYNTHIA  [27]. However, the graphical rendering of simulators is not always up to par with reality, due to rendering quality, texture issues, dated graphics, and domain shifts, and were not photorealistic enough for our methods once applied in real conditions. An acquisition from Grand Theft Auto V  [28]  (GTA V), a video game with a photorealistic appearance, allowed us to obtain a large virtual dataset for 3D detection capable of being combined with our previous datasets. There are several articles explaining how to acquire data from this specific game, such as Richter's  [28]  works, which include instructions and available code to use, and Philipp Krähenbühl's Gamehook  [29] , a tool allowing extraction of the rendering code of a video game.
In addition to the hybrid datasets used during these works, our team also developed a rail-road real-synthetic hybrid dataset called ESRORAD  [30]. To develop the hybrid road-railway dataset, we used a combination of synthetic data generated using GTA, as well as real image recordings. We used GTA to generate a large number of images of roads and railways, along with their corresponding labels and ground truth data. To augment the synthetic data with real examples, we also collected a set of real images of roads and railways by using a car equipped with captors and driving over an existing railway. We then manually annotated a subset of the images (approximately 2500) for use in our algorithms, resulting in a hybrid dataset that includes both synthetic and real-life data. For the railway domain, the number of available datasets is much lower: we have so far only identified two datasets, RailSem19  [31] , and FRSign  [32].
Object 3D Detection & Classification: Improvements on YOLOv5-3D [1] To create a 3D object detection algorithm from YOLOv5, we modified the network architecture to add additional outputs for 3D information. This involved adding new layers to the network that could predict the 3D center of the object, as well as its dimensions and orientation, based on the 2D bounding box predicted by YOLOv5. Next, we trained the YOLOv5-3d  [1]  network on a dataset of images with corresponding 3D ground-truth labels. The resulting 3D object detection algorithm was able to accurately predict not only the class and 2D bounding box of each object, but also its 3D center, dimensions, and orientation. While this algorithm has already been the subject of its own article  [1] , we made some improvements such as improved hyperparameters (results shown in Table  2 ), as well as the use of new methods, namely the integration and modification of the SORT  [33]  algorithm for tracking, the use of dataset combination, knowledge distillation and the creation of a lightweight model (i.
e. , Table  1 ). One of the advantages of our 3D detection algorithm is that it has a single-stage architecture and a limited number of layers, which makes it highly efficient and allows it to achieve good real-time performance. Our algorithm can be run on a wide range of devices, including embedded systems with low computational capacities, such as the Jetson TX2. In the field of autonomous vehicles, it is critical that the algorithms used for object detection and localization can operate in real-time, because the decisions made by these algorithms can have significant consequences, such as avoiding collisions with other objects. Table  2  shows the quantitative evaluation of YOLOv5 on the KITTI dataset. Additionally, the accuracy of our algorithm is also dependent on the camera's intrinsic parameter differences between the training and inference.
Dataset Combination Combining and expanding the training dataset has a dual purpose. Firstly, increasing the quantity of the training data means increasing the diversity of represented situations, allowing the model to adapt to those newly represented situations. In addition, increased training data avoids over-fitting, and training coming from different datasets has the additional benefit of mitigating dataset-related bias, such as specific lighting conditions or scene over-representation. The differences between the data provided by the KITTI and the GTA V datasets are shown in Table  3. As we can see, GTAV provided us with about 10 times the number of pictures provided by KITTI. As we had a majority of annotated data coming from our custom GTA V dataset, we adjusted KITTI's data to follow GTA V's data formatting, through resizing and cropping, using the OpenCV  [40]  library.
Since the original combined dataset had a large disproportion between both synthetic and real images, and the training dataset and the test dataset, we created another combined dataset with, this time, only 50% of synthetic data, in order to see whether the disproportion negatively affected the precision of the trained model or not. The new dataset combinations are shown in Table  4. All the datasets were used to train the large version of the YOLOv5-3d algorithm  [1]. For additional results, we also comparef the 50% synthetic-real dataset with the KITTI dataset when both of them used a pre-training of 15 epochs on KITTI. Regardless of the dataset, in order to effectively compare them we needed to use the same validation test, which was extracted from KITTI.
Dataset Combination To see the effect of the dataset combination, we trained our model on the different combined and reference models described in Table  4. The results can be seen in Figure  5. Here, we can see in green and blue the performance of the model trained on combined models (50% synthetic-real combination with pre-training in blue and regular combination in green), while in orange and red are the results with the original KITTI database (with pre-training in orange, no pre-training in red). We compared these models on 4 metrics: mAP (mean Average Precision), depth accuracy, center position accuracy, and, finally, orientation accuracy. The combined models allowed for increased performances in mAP, accuracy, and center position, with only the orientation not seeing any improvement.
Lightweight Model We first trained YOLOv5 with the same dimensions as this new model on the COCO dataset for 300 epochs. The pre-trained weights of the first layers were then transferred to our model for 15 epochs of training on GTA. Finally, we trained this model on the KITTI and NuScenes datasets. This new model corresponded to the best compromise between low computational time and accuracy, as shown in Table  1. The results from Table  1  are especially promising, because they allowed us to reach an inference time of 28 ms on the Jetson TX2 (corresponding to a frame-rate of 35 Frame-Per-Second), which is an embedded system with limited resources. In comparison to the light model, we went from an inference time of 70 ms to 28 ms (a 60% relative improvement), while only going from a mAP@0.
Conclusions In conclusion, we were able to improve 3D object detection through the use of knowledge distillation, dataset combination, and the creation of a lightweight model. These techniques can effectively improve the performance of 3D object detection models without changes in the algorithms themselves and can be applied to different models. Combining multiple datasets allows the model to learn from a larger, more diverse set of data, which, in turn, improves its performance by exposing it to more situations it might encounter. The results of this study showed that combining multiple datasets improved the model's performance without any drawbacks. The use of knowledge distillation allowed for the transfer of knowledge from a larger, pre-trained model to a smaller, more efficient model.
In this work, while the use of knowledge distillation did not result in direct real-time performance improvement, it nevertheless resulted in improved accuracy of the small model. Overall, this work demonstrated the effectiveness of these techniques for improving the performance of 3D object detection models. Every approach separately provides some improvements to the model, whether in performance or in accuracy, and combining them and using them together is an essential step towards developing more efficient and effective 3D object detection systems for a variety of applications. In this work, we focused on improving 3D object detection through the use of knowledge distillation, dataset combination, and the creation of a lightweight model. While these techniques were effective in improving the performance of our 3D object detection model, we believe that there are further improvements that can be made to our model.
Quantitative evaluation of YOLOv5-3d on KITTI after training on different databases. One of the main limitations of YOLOv5-3d is that it is extremely dependent on the resolution of the training images. If the images used during inference have a different resolution than the images during training, this negatively affects the accuracy of the different predictions, especially the distance prediction, as seen in Table2. In this table, we can see that the RE (Relative Error) and RMSE (Root Mean Square Error) are both higher after a change in resolution between training and inference.
Acknowledgments: We acknowledge the support of SEGULA technology and the role it played in allowing us to conduct this research. We would like to thank the engineers of the  Autonomous Navigation Laboratory (ANL) of IRSEEM  for their support. We thank  Anthony Deshais ,  Vincent Vauchey , and  Vishnu Pradeep  for their precious help in the development of this work. In addition, this work was performed, in part, on computing resources provided by  CRIANN  ( Centre Régional Informatique et d'Applications Numériques de Normandie, Normandy, France ).