dataesr/scientific-paragraphs-categorization huggingface dataset:
Issues with the existing databases Due to the constraints on databases presented in Section 2. 1, almost all investigations dealing with affective computational models use their own database designed according to their goals and needs (except the DEAP database which has recently been used in  [28] ). For example, some work represents emotions in the 2D valence-arousal space or in the 3D valencearousal-dominance space, while other work represents emotions using discrete categories. Furthermore, the models are sometimes dedicated to specific video categories, i. e. music videos or a particular movie genre. LIRIS-ACCEDE uses the widely employed 2D valence-arousal space. However, as the database is freely shared, everyone is free to add new modalities, thus enhancing the range of possible applications.
Furthermore, this database composed of 9,800 excerpts is very large and diversified, unlike most of the databases presented in the previous sections. As a consequence, we think it could be general enough to be used as a reference in the future. Many studies such as  [4]  or  [29]  deplore the lack of a standard affective video database which, combined with the lack of standard evaluation protocols, decreases the efficiency of the affective research community  [5]. Indeed, benchmarking and reproducibility both make it easier to know how computational models perform with respect to the state of the art, and to focus on promising research avenues. This is why we introduce LIRIS-ACCEDE and define reproducible protocols in the following sections.
Movies used in LIRIS-ACCEDE One of the main requirements of LIRIS-ACCEDE was that it should be freely available to the research community. That is why the 160 movies used for creating the database are shared under Creative Commons licenses. Creative Commons is a non-profit corporation providing standardized free copyright licenses to mark a creative work with the freedom the creator wants it to convey. The CC BY license known as "Attribution" is the most accommodating license since users can reuse the original creation as long as they credit the creator. Three modules adding more restrictive conditions can be combined. The SA module (ShareAlike) requires that works based on other works shared using this module, have to be licensed under identical terms. The NC module (NonCommercial) prevents original works from being reused for commercial purposes.
Last but not least, the ND module (No Derivative Works) prohibits altering, transforming, or building upon original works. To create the database, we have used only movies shared under a Creative Commons license that do not contain the ND module, because our goal was to modify the selected movies by extracting several excerpts from them. Thus, using videos shared under Creative Commons licenses makes it possible to share the database publicly without copyright issues. Most of the 160 movies used for creating LIRIS-ACCEDE come from the video platform VODO. This references best free-to-share feature films and short films that have been submitted on the website and makes them easily available to millions of people. It is important to notice that free-to-share films do not mean User Generated Contents with low expertise levels.
Movies referenced on VODO have been created by filmmakers with excellent technical expertise. Many films in the database have been screened during film festivals including, but not limited to, "RIP! A remix manifesto" directed by Brett Gaylor (Special Jury Prize at the "Festival du Nouveau Cinéma in Montreal"), "Emperor" directed by Juliane Blockand (winner of the Feature Category at the Portable Film Festival) and "Pioneer One" produced by Josh Bernhard and Bracey Smith (winner of the Best Drama Pilot at the New York Television Festival). The "Home" documentary directed by Yann Arthus-Bertrand included in the database is a special case since it is a big budget movie distributed by 20th Century Fox that has no copyright. In brief, 40 high quality feature films and 120 short films shared under Creative Commons licenses have been collected to create the 9,800 excerpts making up LIRIS-ACCEDE.
The total time of all 160 films is 73 hours, 41 minutes and 7 seconds. A list of 9 representative movie genres describes the movies: Comedy, Animation, Action, Adventure, Thriller, Documentary, Romance, Drama and Horror. By displaying the normalized distribution of movies by genre in LIRIS-ACCEDE compared to the normalized distribution of movies by genre referenced on IMDB and on ScreenRush, it can be observed that distributions appear to be similar, as shown in Figure  1. Thus, movies used in LIRIS-ACCEDE are representative of today's movies. Languages are mainly English with a small set of French, German, Icelandic, Hindi, Italian, Norwegian, Spanish, Swedish and Turkish films, subtitled in English. Note that 14 movies are silent movies.
Characteristics of LIRIS-ACCEDE The database is made up of 9,800 excerpts extracted from the 160 selected movies presented in Section 3. 1. 1,000 excerpts have been manually segmented because they were part of the pilot test to ensure the reliability of the annotations. Subsequently, the other excerpts have been automatically segmented using a robust cut and fade in/out detection, implemented based on the algorithms described in  [30]. Because all the segmented excerpts start or end with a cut or a fade, it is very likely that each segment be perceived by users as semantically coherent. The 9,800 segmented video clips last between 8 and 12 seconds, and the total time of all 9,800 excerpts is 26 hours, 57 minutes and 8 seconds. Even if the temporal resolution, or granularity, of emotions is still under debate, most of psychologists agree that they are part of a complex but very rapid process  [31].
This is essential since many previous studies, including  [18]  and  [33] , have shown that the arousal dimension was correlated to editing features such as the shot cut rate or the presence of dissolves. Only 1760 excerpts do not include any scene cut or fade in/out. On average excerpts are composed of 2. 8 video-editing features (this statistic does not count the editing features on the boundaries). More generally, we achieved a great variety of excerpts reflecting the variety of selected movies. The excerpts contain scenes of violence, sexuality, murders, but also more common scenes such as landscapes, interviews and many positive scenes of daily life. LIRIS-ACCEDE is currently the only video database annotated along induced emotions that includes such a large range of contexts.
This is one of the most efficient sorting algorithms. Indeed, the average computational complexity of this algorithm is O(n log n), where n is the number of data to sort. In the worst case, complexity is O(n 2 ), but this performance is extremely rare and in practice the quicksort is often faster than other O(n log n) algorithms  [43]. As the cost of the sorting operation is proportional to the number of comparisons, the quicksort seems the best choice for reducing costs to sort the whole database compared with other sorting algorithms. In practice, the quicksort algorithm allows costs to be reduced to approximately US$10,000 for ranking of the whole dataset along one axis. The principle of the quicksort algorithm is to choose an element, called a pivot, to which all other elements are compared.
Experimental setup The annotation of the database along the arousal axis was performed three months after the annotation along the valence axis. Meanwhile, a new interface for displaying tasks to workers had been released on CrowdFlower. This explains why there are few changes in both protocols to adapt the experimental setup to the new interface. Given a pair of video excerpts, annotators had to select the one that conveys "the most positive emotion" (for valence) or "the calmest emotion" (for arousal). The words "valence" and "arousal" were not used since they might be misunderstood by the annotators. They were asked to focus on the emotion they felt when watching the video clips, i. e. the induced emotion, and not on that of the characters. As the arousal axis was more challenging to annotate, the arousal axis of the Self-Assessment Manikin and an example were displayed at the beginning of the task to make sure that annotators had understood the task properly.
Test questions were also randomly inserted to test annotators that passed the quiz on an on-going basis as they worked through the job. In concrete terms, the quicksort algorithm was used in both annotation experiments to generate the comparisons. First, an initial video excerpt was randomly chosen to be the first pivot. All the other clips from the database were compared to this excerpt meaning that 9799 pairwise comparisons were generated for the first iteration. Each pair was displayed to workers until three annotations were gathered. We found this was a good compromise between the cost and the accuracy of the experiment. Once three annotations per comparison were made for all the comparisons, all the annotations were collected. In each comparison, the pivot was considered as inducing the most positive emotion for valence or the calmest emotion for arousal if at least two annotators selected the pivot during the annotation process.
The final rank of the pivot was thus computed. Assuming that the pivot does not induce the lowest or the highest valence or arousal, this process splits the database into two subgroups. For the second iteration, one pivot was selected in each subgroup and the two pivots were compared to the other video clips inside their subgroup, generating 9,797 new comparisons. For the next iteration, four pivots were selected and so on. The process was repeated until a rank was assigned to all the 9,800 video excerpts. Finally, each video excerpt is accompanied by two discrete values ranging from 0 to 9,799 representing its arousal and valence ranks.
Annotation results For annotation of the valence axis, more than 582,000 annotations for about 187,000 comparisons were gathered from 1,517 trusted annotators from various countries. Annotators from 89 countries participated in the experiment, reflecting a huge diversity in cultural background. The majority of workers originated from India (18%), USA (16%), Romania (4%) and Vietnam (4%). A more detailed distribution of countries is displayed in Figure  3. Over 90% of data come from 530 of these annotators. The 1,517 trusted annotators showed an accuracy of 94. 2% on test questions, whereas this accuracy was about 42. 3% for untrusted annotators. More iterations were needed to fully rank the database along the arousal axis. More than 665,000 annotations for around 221,000 unique comparisons were gathered from 2,442 trusted annotators also from 89 countries.
As displayed in Figure  3 , the countries of annotators is also diversified but different since most of the workers are American (33%), Vietnamese (16%), Indian (10%) and British (5%). As a point of comparison, this time over 90% of data come from 830 annotators. The accuracies on test questions for trusted and untrusted annotators were approximately the same as for those annotating the database along the valence axis. However, the number of untrusted annotators was slightly lower than for the first experiment thanks to the Quiz Mode. When creating crowdsourcing tasks, ethical concerns have to be considered and the anonymity of the crowdworkers must be preserved. It is worth mentioning that, in our experiments, crowdworker privacy has been protected since the annotations that led to the ranking of the excerpts are not published.
Regression SVR has demonstrated good performance in many machine learning problems and, more specifically, in affective content analysis work such as  [50] ,  [27]  or, more recently,  [51]. SVR models construct a hyperplane by mapping vectors from an input space into a high dimensional feature space such that they fall within a specified distance of the hyperplane. Even if we have shown in Figure  4  that arousal and valence are correlated, two independent -SVRs are used in this work to model arousal and valence separately. The Radial Basis Function (RBF) is selected as the kernel function and a grid search is run to find the C, γ and p parameters. Since the database is ranked along the induced arousal and valence axis, the ground truth is made up of these raw ranks, initially ranging from 0 to 9,799, which are uniformly rescaled to a more common [-1, 1] range.
Protocols The purpose of this section is to introduce standard protocols using the database in different ways to make possible comparisons within the field. The MSE and Pearson's r are computed to quantify the performance of each protocol. The MSE for regression models is widely used to quantify the difference between estimated values and the true values estimated. It measures the amount by which the estimated values differ from the ground truth and assesses the quality of the regression in terms of its variation and degree of bias, while the Pearson product-moment correlation coefficient (or Pearson's r) is a measure of the linear correlation between estimated and true values.
Protocol A: Predefined subgroups In this protocol, the training and test sets have been manually defined to make sure that they each include 4,900 excerpts from 80 films. In this way, the excerpts extracted from the same film are only in one of the sets, either the training set or the test set. Insofar as possible, we tried to distribute movies equally in the sets according to their genres. We also defined a validation set, should it be needed in future studies, by dividing the training set into two subgroups, each made up of 2,450 excerpts extracted from 40 films. The list of excerpts in each set is available alongside the database and the annotations. The MSE and Pearson's r for this protocol using our baseline model are shown in Table  4. 4.
Protocol D: Same movie The purpose of this last protocol is to gain insight into the regularity of the movie in terms of affective impact. Indeed, by learning on samples from the first half of a movie and testing on the remaining excerpts, the results can provide information on how well the first part of a movie is able to model and to be generalized to the induced valence and arousal of the whole movie. The results of this protocol, applied to some movies of the database, are displayed in Table  6.
Regression results The results are promising given the huge variety of movies in the database. They indicate that regression On the other hand, researchers using one of the protocols defined in Section 5. 3 will be able to know how their model performs not only with respect to this baseline but also to all future work using one of these protocols. There are several unexplored leads in this work that we would like to share with the reader. One way to improve performance is to check inter-annotator reliabilities to detect outliers and thus remove them from the process. Furthermore, protocol C shows different levels of performance depending on the movie genre, indicating that our model misses some information that could potentially be added by higher level features. Last but not least, as arousal and valence are correlated, it seems legitimate to model them jointly.
DISCUSSION One of the main limitations of the proposed database lies in the fact that the video clips have been ranked relatively to each other. Thus the rankings provide no information on distribution of the database in the 2D valence-arousal space. In other words, it is uncertain whether the extreme cases with the lowest or highest ranks elicit extreme emotions. Furthermore, these ranks are relative to this particular database, which prevents comparison with other video clips annotated with absolute valence and arousal scores. To address this limitation, we carried out an experiment  [59]  in which annotators were asked in a controlled environment to rate on a 5-point discrete Self-Assessment Manikin scale some excerpts from the database for arousal and valence. The results have shown that the Spearman's rank correlation coefficient (SRCC) between affective ratings and crowdsourced rankings is significantly high for both arousal (SRCC = 0.
751, t(44) = 7. 635, p < 1 × 10 -8 ) and valence (SRCC = 0. 795, t(44) = 8. 801, p < 1 × 10 -10 ), thus cross-validating the overall dataset. The controlled affective ratings also make it possible to estimate the distribution of the database and understand the range of emotions elicited by the database. Several other unknown factors can potentially affect the ratings and would require further research. First, crowdworkers were asked to focus on what they felt in response to the video excerpts. Contact with the crowdworkers was quite limited. As such, it was not possible to ascertain that annotators were annotating the induced emotion and not the perceived emotion or even the emotion they thought they should feel, since it is possible to make judgments on the basis of conventional characteristics without experiencing any emotion  [60].
As a consequence, we hypothesize that most crowdworkers turned the volume on to rate the pairwise comparisons. Third, the crowdworkers made the annotations in various uncontrolled environments under different conditions. However, elicitation of an emotion is a subtle process depending on a large number of factors (e. g. listener, performance or contextual features)  [62]. Despite this, inter-annotator reliability indicates that an overall agreement was achieved among crowdworkers and that annotations tend to be stable. Moreover, these results have been compared to ratings gathered in controlled conditions in order to validate the annotations made in uncontrolled conditions and to detect potential outliers  [63]. The correlation between affective ratings and crowdsourced rankings is significantly high, thus cross-validating the overall database for future uses in research work.
CONCLUSION This paper has addressed the lack of large video databases for affective video analysis, as current existing databases are limited in size and not representative of today's movies. Following the work began in  [6] , we proposed LIRIS-ACCEDE, a large video database freely shared to be used by the research community. The database is made up of 9,800 excerpts lasting from 8 to 12 seconds, extracted from 160 diversified movies. All the 160 movies are shared under Creative Commons licenses, thus allowing the database to be shared publicly without copyright issues. It is available at: http://liris-accede. ec-lyon. fr/. All the excerpts have been ranked along the induced valence and arousal axes by means of two experiments conducted on a crowdsourcing platform. Both experiments were highly attractive.
A large number of annotators performed each experiment, making it possible to collect large volumes of affective responses from a wide diversity of annotators and from a large spectrum of contexts. With this experimental design, high inter-annotator reliabilities were achieved considering the subjectivity of the experiments. We also introduced standard protocols using the database in an attempt to perform standardized and reproducible evaluations to fairly compare future work within the field of affective computing. Four protocols were proposed corresponding to different goals and needs. Moreover, we implemented a baseline and used these protocols to assess its performance, showing promising results. Note that all the audio and visual features used for the baseline are also released alongside the database.
Nowadays molecular species delimitation methods promote the identification of species boundaries within complex taxonomic groups by adopting innovative species concepts and theories (e. g. branching patterns, coalescence). As some of them can efficiently deal with large single-locus datasets, they could speed up the process of species discovery compared to more time consuming molecular methods, and benefit from the existence of large public datasets; these methods can also particularly favour scientific research and actions dealing with threatened or economically important taxa. In this study we aim to investigate and clarify the status of economically important moths species belonging to the genus Spodoptera (Lepidoptera, Noctuidae), a complex group in which previous phylogenetic analyses and integrative approaches already suggested the possible occurrence of cryptic species and taxonomic ambiguities.
All these processes usually generate discordance between gene trees and species trees  [67] , which can biases the inference of putative species clusters. Nonetheless, single locus data have been proved useful to provide robust inferences of evolutionary histories  [68]  and quick assessments of species boundaries and diversity  [26, [69] [70] [71] [72]. More recently they have been also associated with concise morphological descriptions to produce rapid descriptions of new species, an approach referred to as turbo-taxonomy  [73] [74] [75]. Finally the use of single locus data also benefits from the fact that several genes (e. g. 16S ribosomal RNA) are well represented in large public databases such as GenBank (http://www. ncbi. nlm. nih. gov/genbank/) or the Barcode of Life Database (BOLD; http://www.
The model assumes that each substitution has a small probability of generating a speciation event; hence the number of substitutions between species is expected to be significantly higher than those within species  [60]. The model then implements two independent classes of Poisson processes (one describing speciation and the other describing within species branching events) and searches for transition points between interspecific and intraspecific branching patterns. Potential species clusters are then determined by identifying the clades (or single lineages) that originate after these transition points. Because both methods efficiently deal with large single-locus datasets  [110] , they can readily benefit from the mass of sequence data generated through the Barcode of Life (BoL) initiative or deposited on public database such as GenBank.
Sampling design The sampling for this study covers 27 of the 30 known Spodoptera species  [76]. We were unable to acquire sequence data for three species: S. compta (Walker, 1869) (from Peru), S. malagasy Viette, 1967 (from Madagascar) and S. roseae (Schaus, 1923) (from the Galapagos Islands). Both S. compta and S. roseae are extremely rare in collections (especially S. compta which is only known from three specimens collected in the 19 th century). As for S. malagasy, previous sequencing attempts on several old museum specimens have been unsuccessful  [85]. For this study we generated 86 new sequences for specimens obtained from the Muséum National d'Histoire Naturelle in Paris (MNHN). In addition we used 118 sequences from the study of Kergoat et al. ,  [85] , 463 additional sequences from GenBank and five sequences from BOLD.
The resulting dataset encompasses 672 Spodoptera specimens, with a mean number of 24 individuals per species (from two individuals for S. apertura (Walker, 1865), S. pecten Guenée, 1852, S. pulchella (Herrich-Schäffer, 1868) and S. triturata (Walker, 1857) to 101 in S. frugiperda; S1 Table ). All specimens, their geographic origin and the corresponding sequence accession number are listed in a supplementary table (S1 Table ). For outgroup choice we used the results of the study of Mitchell et al. ,  [111]  to select the following seven noctuid taxa: Agrotis ipsilon (Hufnagel, 1766) (Noctuinae), Helicoverpa armigera (Hübner, 1805) (Heliothinae), Heliothis virescens (Fabricius, 1777) (Heliothinae), Mythimna unipuncta (Haworth, 1809) (Hadeninae), Noctua atlantica (Warren, 1905) (Noctuinae), Psychomorpha epimenis (Drury, 1782) (Agaristinae), Sesamia inferens (Walker, 1856) (Xyleninae).