dataesr/scientific-paragraphs-categorization huggingface dataset:
These models are dependent on user attention prediction. For example, Xu et al. [25]  considered content saliency and field of view (FoV) preferences to extend the PSNR and SSIM metrics to 360°content. It is therefore key to understand the attentional and emotional processes of users in an immersive environment, and how these processes are associated with the content. To enable the study of this connection between user attention, user emotions and immersive visual content, this article introduces: • PEM360, a new dataset of user head movements and gaze recordings in 360°videos, along with self-reported emotional ratings of valence and arousal, and continuous physiological measurement of electrodermal activity and heart rate. The stimuli are selected based on high-level and low-level content saliency to enable the spatiotemporal analysis of the connection between content, user motion and emotion.
• a set of software tools to pre-process the data of gaze, electrodermal activity and content, and to visualize jointly instantaneous heat maps of gaze and arousal level superimposed on the frame, which we name Emotional maps. • a preliminary analysis validating the data and verifying known results, and examples of new connections that can be investigated. The entire collection of artifacts is presented as Python tools and notebooks to enable reproducibility of the data processing. The dataset and tools are now available in a public GitLab repository 2.
EXISTING DATASETS While a number of works in the domain of human-computer interaction and cognitive sciences have studied emotions in immersive environments  [2, 9, 19, 24] , it has been only more recently that the multimedia community in particular has considered sensing and recording emotions along with head and gaze motion  [16, 20, 22, 26, 27]. Human emotions are commonly decomposed along two main dimensions: valence, representing the negative or positive nature of an emotion (unpleasant-pleasant), and arousal, representing the intensity of the perceived emotion (calm-excited)  [3]. Li et al. [16]  introduced the first reference database obtained from 95 users freely watching 73 videos in 360°who provided their valence and arousal ratings after every clip using the self-assessment manikin (SAM) tool  [5].
Their head positions were recorded. The material publicly available however only consists of the videos and the average valence and arousal value pair for each video (averaged over all users). Subsequently, Tang et al. [20]  presented an experiment where 19 users watched 36 images in 360°while their self-reported emotions and eye motion were collected. More recently, it has been shown that the capability of a single rating issued after experiencing the 360°content is limited and unable to fully represent the variations of user state  [24]. This is why new tools have been introduced to enable the continuous collection of self-reports inside the immersive environment  [22, 27]. The data collected by Toet et al. [22]  and Xue et al. [27]  also comprise physiological measurements of heart rate and electrodermal activity (EDA, as skin conductance), which has been shown to reliably represent user instantaneous arousal  [4].
Toet et al. [22]  presented a new emotions rating tool, named EmojiGrid, tested on 40 users viewing 62 videos from the reference database of Li et al. [16]. While they provide the per-user per-video valence and arousal ratings, only time averages are made available for EDA, and no gaze or head motion traces. Xue et al. [27]  introduced a continuous grading tool of valence and arousal. They provide a dataset of 11 immersive videos from the same database  [16]  experienced by 32 users. Subjective emotional ratings, physiological measurements (including EDA) and head and gaze movements are continuously collected and made available. This latter work  [27]  is closer to ours and has been made partly concurrently. Our dataset is however complementary and enables other types of studies. We provide EDA streams at a higher rate, acquired at 16Hz, compared to 4Hz in  [27].
In the aforementioned objective of understanding the connection between attended regions and instantaneous emotions, like arousal, it is important to enable the detection of several peaks of the phasic component of EDA per second, requiring hence a higher acquisition rate. Also, we sample seven video stimuli from the same reference database  [16]  for our experiments, so that specific criteria on saliency are met, as detailed in Sec. 3. 1. Out of the seven videos, six differ from the videos selected by Xue et al. [27]. Our dataset therefore enriches the existing datasets and enables extensive analysis to gain new insights on the connection between attention, emotion and content.
High-level saliency maps are composed of high-level semantic features, such as faces, cars, or animals. Inspiring from Chopra et al. [8] , high-level saliency is obtained from YOLOv4 object detector, with object bounding boxes being used as binary saliency maps. We selected 7 videos from the reference database of Li et al. [16]. The selected videos should have a range of valence and arousal as wide as possible of level of valence and arousal, and the LL saliency should be evenly distributed both within and outside object bounding boxes characterizing HL saliency. To select these videos we compare (i) the number of pixels inside and outside objects, and (ii) the perpixel LL saliency (ranging between 0 and 255), computed as the total LL saliency inside and outside objects normalized with the corresponding number of pixels.
Procedure The lab experiment started with a pre-questionnaire assessing the user's background with VR and checking for visual deficiencies. Eye tracking calibration was done using the FOVE software for each user before beginning the experiment to make sure the eye tracking data is properly recorded. The VR experiment systematically started with a low-arousal (relaxing) video (ID 32) to bring EDA and HR levels to a user-relative baseline. The remaining six VR videos were then experienced in a random order by every user. Users were in standing position during the experience and could freely explore in 360°while holding the back of a chair to maintain balance and orientation. The videos were played without audio. After each viewing, the headset was removed and the SAM scale presented for arousal and valence rating.
Dataset structure The resulting dataset PEM360 is provided with the structure shown in Fig. 4. The raw_data folder contains 34 folders, one for each user. User folders contain a Shimmer CSV file containing the EDA and optical pulse data recorded over all the 360°videos experienced by the user, and seven CSV files, one per video, containing the gaze and head motion data recorded during the corresponding video. Entries in the CSV files include system timestamps to synchronize the data modalities for analysis. Valence and arousal ratings of each user for each video are stored in the root folder under graded_valence_arousal. csv. Finally, the root folder PEM360 also contains the Python Jupyter notebook providing the software tools described in Sec. 4, and the entire data processing workflow to reproduce the analysis presented in Sec.
Processing EDA data The EDA signal is the raw measurement of skin electrical conductance in micro-Siemens (µS). Two main components can be distinguished in an EDA signal  [4, 6] : the tonic level, also called skin conductance level (SCL), varies slowly and represents slow autonomic changes that may not be associated with stimulus presentation; and the phasic level, which represents faster changes in EDA, and can better reflect the impact of successive stimuli. Raw EDA, phasic and tonic components are shown in Fig. 5 -top and 5-center. We use the Python toolbox Neurokit  [17]  to process EDA data, which uses the cvxEDA method to extract the phasic component. Finally, the physiological arousal to be analyzed in connection with experimental stimuli can be assessed from several metrics on the phasic level, such as peak frequency, duration and amplitude.
Processing video content As introduced in Sec. 3. 1, we use LL and HL saliency models designed for regular flat images. We therefore apply them on FoV projections of the entire frame. We first uniformly sample 100 points on the unit sphere and project them on the equirectangular frame using the equirectangular-toolbox  [18]. Each "patch" is made of a projection centered on one of these points, it is a 512x512 image corresponding to a 108°x108°FoV. These patches can overlap each other and are separately given to the appropriate models for both LL and HL saliency. For LL saliency, we use a Python implementation of Itti's saliency map  [13] , which also allows the combination of Itti saliency with the optical flow between consecutive frames, which we do by using separate extractors for each patch.
The LL saliency is downscaled using average pooling over blocks of 5x5 pixels, whereas HL saliency is downscaled using max pooling over blocks of the same size. The files are stored in HDF5 format and can be accessed from a link given in the article's repository mentioned in Sec. 1, but can also be re-computed from the provided code. As previously discussed, the stimuli choice and experimental procedure are designed to collect data enabling a time-dependent analysis of the connection between attention, emotion and content. That is why we provide a tool for the experimenter to play the 360°v ideo and visualize the instantaneous gaze locations and arousal (SCR) of a given user from the recorded data. This tool implements a new way of visualizing arousal in connection with gaze, which we name emotional maps.
Instantaneous visualization of gaze and emotions: Emotional maps • pixel luminance reflects the time the user spent attending the area over a past window of T seconds. A Gaussian kernel of parameter σ is convolved with every gaze location, and accumulated over the sliding window of T seconds. A bright (resp. dark) area can therefore reflect a fixation (resp. a saccade). • pixel color represents the user's SCR, from blue (low arousal) to red (high arousal). Emotional maps generated from a record with our tool are accumulated into videos. Each point persists on the video for P seconds, creating a trail to easily visualize the gaze path and arousal changes. An example of such a video frame is shown in Fig. 6  3. The script compute_emotional_map. py creates the emotional maps and blends them with the frames to produce the resulting video visualization from records of gaze and EDA data.
Data validation Reliability of the collected ratings. We verify the reliability of the collected arousal and valence by assessing the similarity of the user ratings for each video. This is achieved with the intra-class correlation coefficient (ICC), with classes corresponding to the 360°v ideos. ICC estimates based on mean ratings with a two-way mixed effects model are 0. 96 (95% CI 0. 87-0. 99) for arousal and 0. 88 (95% CI 0. 72-0. 98) for valence. According to Koo and Li's guidelines  [14] , this is excellent and good inter-rater agreement, respectively. Agreement between collected ratings and original dataset. Fig. 7  shows the valence and arousal ratings of our users as a boxplot for each video, along with a red dot representing the corresponding average values available in the original dataset  [16].
CONCLUSION In this article, we have presented the new PEM360 dataset of 360°v ideos with continuous physiological measurements, subjective emotional ratings and user motion traces. The stimuli are selected to enable investigating the spatiotemporal connection between user attention, user emotions and visual content. We have described the data collection process, the pre-processing workflow of the different data modalities, and exemplified some possible novel types of analyses to demonstrate the potential insights that can be drawn from PEM360. The artifacts are made available in a reproducible framework based on notebooks.
A heuristic is used to recognise blinking states. We consider FRSign dataset, a large collection of over 100,000 images of traffic signals from some of the trains in French Railways. A distilled and cleaned version of the dataset curated by us is used for training. The trained network has low computational overhead and can recognise traffic signals in real time and under diverse field conditions. It has robust performance even for complex scenes having multiple signals and light sources, and in adverse circumstances such as rain and night environments. The refined version of the dataset is published as open for validation and further research and development.
Note that, the recognition time in addition to driver response time constitute the real-time window in GoA1 and GoA2 grades of automation. To train the system, we leverage a large railway signalling dataset collected from trains in field operation  (Harb et al. 2020) , with video sequences acquired from cameras filming the railway from the train drivers point of view. The original dataset is big and it needs some pre-processing operations to make it more suitable for machine learning applications. It has been distilled into a smaller dataset after removing duplicates and cleaning erroneous annotations. Also, the dataset contains various types of signals with multiple combinations of lights/signal states, including blinking patterns. Despite being a key feature for practical use of vision-based driving assistance tools, automated detection and recognition of blinking lights has not received much attention in the literature.
The paper is organised as follows: next section describes outlines the related works in the literature. The methodology followed to address the fast traffic signals detection and recognition issue is discussed in Section "Methodology". In Section "Distilled French Railway Signal Dataset", we describe the dataset and how it has been distilled in order to train a high performance deep learning system. The fifth section presents the results and the performance of the system, with a focus on illustrative scenarios that highlight the robustness of the proposed method. The conclusions of the study are provided in the last section.
Related Work Automated signal detection and recognition has been the focus of study for road traffic in the context of self-driving cars  (Yuan et al. 2019). For instance, recognition of road signals in poor environmental conditions has been studied by  Temel et al. (2020). A comparative analysis of several stateof-the-art object detection and tracking algorithms to detect and track different classes of road vehicles is presented in  Mandal and Adu-Gyamfi (2020). Further, many open datasets are available for researchers working on autonomous vehicles  (Harb et al. 2020). However, only a scant amount of literature has been reported on application of these methods on railway transportation. Early attempts were often based on traditional image processing techniques and on handcrafted features used in computer vision for object detection  (Melander and Halme 2016).
Methodology We follow a two-step procedure for detecting the static and the blinking states, respectively. First, a YOLOv5 deep learning architecture is trained using image frames from a railway signal image frames dataset captured from specific trains of the French Railway. The trained network is then used to detect traffic signals in individual frames. The frames containing traffic signals are passed through the classification modules of the deep learning architecture to recognise the signal states. A bounding box around the signal is also generated. Since some of the signals involve blinking lights we need to further postprocess these information. For this, in the second step, we consider a sequence of m bounding boxes in successive frames. A simple heuristic is used to detect signal blinks in these sequences.
The probability is calculated using the logit function. Bounding box regression is used for the purpose of signal localization. The loss metric used for bounding box regression is based on the popular intersection over union (IOU) measure computed between the true bounding box and the predicted bounding box. Three geometric factors in bounding box regression, i. e. overlap area, central point distance and aspect ratio, have been used to define the Complete IoU (CIoU) loss in  Zheng et al. (2020). This is used as the loss function during training of YOLOv5. A batch size of 32 images is used during the training phase. We use a pre-trained YOLOv5 network  (Jocher 2020). It is pre-trained on the MS COCO dataset. COCO stands for Common Objects in Context. It is a large-scale object detection, segmentation, and captioning dataset benchmark  (Lin et al.
Distilled French Railway Signal Dataset The French Railway Signalling (FRSign) dataset 1  is a large collection of railway traffic signal images and videos captured from selected running trains in France over a time period spanning several months since September 2017. Sequences are hand labelled by domain experts. The dataset contains more than 100,000 images illustrating six types of French railway traffic lights and their possible color combinations, together with the relevant information regarding their acquisition such as date, time, sensor parameters, and bounding boxes  (Harb et al. 2020). Most of the existing French railway panels with the corresponding state combinations are found in the FRSign dataset. Each image is of high resolution having 1024 × 1024 pixels. The images were captured under a large variety of environmental conditions in open lines as well as in rail yards.
While the presence of background elements, obstacles, and multiple objects in the scene make the automated detection and recognition tasks particularly challenging, the successful application of the proposed method on images collected in field operation proves its effectiveness and its suitability for real-world sequences. Like most other object detection tasks in the transportation domain, the FRSign dataset has significant class imbalance. Only about 5% of the frames have a signal present in them. Instances of some of the signal classes, particularly those involving blinking lights, are relatively scarce in the dataset.
Railway Traffic Lights The shapes of various traffic signals are shown in Fig. 5. The states of the traffic lights (and the associated traffic management information) for panels A, C, F, and H are shown in Table  1. Two example images from the dataset along with their annotations are shown in Fig. 6. A comprehensive description of the traffic lights states with the list of possible combinations for the different panel shapes considered in this study can be found in  Harb et al. (2020). The original dataset is large in size (approximately of 289GB). However, a large number of images are almost duplicates as shown in Fig. 7. We have performed semiautomatic image screening to obtain a distilled dataset containing only distinct images. Also, the dataset contains a number of wrongly annotated images as shown in Fig.
8. We have also cleaned such annotations by semi-automatic processes. The perceptual hashing algorithm is used for detecting near duplicate images in the collection  (Hamon et al. 2006 ). Perceptual hashing is a special type of locally sensitive hashing and is effective in detecting image frames with slow changing differences among them. We use the open source library pHash  (Zauner 2010)  to implement the near duplicate detection scheme. We identify all images whose classification differs from those of adjoining frames. Class annotations of these images were inspected and re-annotated to remove the anomalies. A distilled and cleaned dataset obtained from the FRSign the original dataset is available online. 2 There are a total of 2800 distinct images in the reduced and cleaned dataset. A higher accuracy is obtained by training on this reduced dataset instead of the original dataset containing duplicate images and wrong annotations.
Results We compare three algorithms with respect to the evaluation metrics described above, namely: YOLOv3 (Bochkovskiy et al. 2020), YOLOv5  (Jocher 2020)  and Efficient-Det  (Tan et al. 2020). The YOLOv3 is a faster version of YOLO but has a lower detection accuracy, whereas YOLOv5 has a higher accuracy but takes relatively more processing time. EfficientDet is bi-directional feature pyramid based networks with a scaling layer. It is known to provide good performance in resource constrained situations. For the FRSign dataset, the EfficientDet algorithm provides weaker O F F & A A & C C v & C V L & C O F F & C C & F V L & F S & F O F F & F V L & H R & H R R & H O F F & H M & ID 3 A & R A + R & H YoloV5 YoloV3 EfficientDet YoloV5, YoloV3 and EfficientDet We also compare the computation time in milliseconds required by the algorithms to infer the class label for a single image frame.
13. This is by far the most difficult error to correct. Manual overrides may be employed in such situation. We also find that the system classifies the signals correctly in many difficult scenes. Notable among them follows: 1. The YOLOv5 works even in rainfall and cloudy conditions. This can be seen from the example images in   5. Multiple signal lights present in a single frame have also been found to be correctly detected by the system. This is illustrated in Figure  18. Further analysis of the errors and success of the system is being carried out on a larger set of images from the dataset. It may be noted that the dataset is unbalanced in terms of the signal classes. This will be balanced in future with more images being processed. Although the number frames having blinking signal classes are relatively less, the blink processing heuristic is successful in recognising a majority of them.
Conclusions We present a deep learning-based methodology for detection and classification of railway signals. The YOLOv5 architecture is used for this purpose. A dataset from French Railways has been studied. We obtain a high precision and recall for the detection task. The computational efficiency of the YOLOv5 architecture makes the method suitable for real time application. We also propose a sequence based blink detection heuristic to detect blinking signal states. Deep neural architecture combining convolution, recurrent, and attention structures may be used to handle such dynamic signal states. The system has been found to be robust in night, rain and other adverse environmental conditions. It has also been found to perform satisfactorily in complex scenes containing several other light sources and various similar objects.
The instance was first introduced in Saraç and Sipahioglu (2014) and faithfully regenerated by the authors of  Chen and Hao (2016). This instance has the following features: n = 500 items, m = 40 knapsacks, h = 300 classes, and density d = 1. 00. These instances have unknown optimal solutions. All the input data and the certificates of the results obtained by the proposed HESA algorithm are available at http://www. info. univ-angers. fr/ pub/hao/HESA-GQMKP. html. Table  1  shows for each instance of Set I and Set II, the following information: instance name (Ins. ), number of items ( n ), number of knapsacks ( m ), number of classes ( h ), and density ( d). The HESA algorithm was implemented in C++ and compiled using GNU g++ 8. 3. 0 with the "-O3" option (We will make our code publicly available at the link above).