dataesr/scientific-paragraphs-categorization huggingface dataset:
Statistics on the WASABI Song Corpus This section summarizes key statistics on the corpus, such as the language and genre distributions, the songs coverage in terms of publication years, and then gives the technical details on its accessibility. Selection Bias. The amount of lyrics published per decade, per genre, per language etc. is biased by the availability of lyrics for that decade on the crawled web sites. More specifically, the WASABI project initiated the construction of its knowledge base in 2016, at the beginning of the project, taking as its "initial seed" the songs published on the now closed LyricsWikia site, a crowd sourcing wiki manually created by humans. The fact that such source contains more English songs than others, more popular songs, etc. is reflected in the following distributions of our dataset.
Language Distribution. Figure  3a  shows the distribution of the ten most frequent languages in our corpus. 7 In total, the corpus contains songs of 36 different languages. The vast majority (76. 1%) is English, followed by Spanish (6. 3%) and by four languages in the 2-3% range (German, French, Italian, Portuguese). On the bottom end, Swahili and Latin amount to 0. 1% (around 2k songs) each. Genre Distribution. In Figure  3b  we depict the distribution of the ten most frequent genres in the corpus. 8 In total, 1. 06M of the titles are tagged with a genre. It should be noted that the genres are very sparse with a total of 528 different ones. This high number partially stems from the fact that many "subgenres" such as Alternative Rock, Indie Rock, Pop Rock all constitute own genres in the dataset and we take them as is.
Specifically, we do not perform any clustering to -for instance -unite Alternative Rock with Rock. We omit displaying "subgenres" in Figure  3b  for clarity. The most common genres are Rock (9. 7%), Pop (8. 6%), Country (5. 2%), Hip Hop (4. 5%) and Folk (2. 7%). Publication Year. Figure  3c  shows the number of songs published in our corpus, by decade. 9 Over 50% of all songs in the WASABI Song Corpus are from the 2000s or later and only around 10% are from the seventies or earlier. Note one peculiarity, the number of songs in the 2010-2020 decade is lower than in the previous decade. The main reason for this is the initial creation of the dataset in 2016, i. e. our dataset can contain only songs for 6 years of the decade concerned. Assessing and ensuring the quality of a large dataset formed by the aggregation of multiple data sources is a difficult and ongoing process.
Metadata from various sources can often be inaccurate or contradictory. Multiple hackathons were held between 2017 and 2021, during which a variety of people from different backgrounds and with varying levels of expertise explored the dataset in an open-ended manner. These hackathons have helped identify many recurring errors, conflicts, and problems, and we have written a series of scripts (available in the dataset's Github repository) to fix some of them. The dataset is intended to be maintained for at least the next three years, as we have new projects and ongoing PhD theses around this work, that will leverage and extend it. More metadata will be added (from the MIR audio analysis of the songs, or obtained by linking the songs to existing midi transcriptions or to their online music scores), and we are developing new metadata quality assessment tools (based on visualizations and inference rules) that will allow the community to better detect and report erroneous, contradictory or missing metadata  [42].
Availability and Accessibility of the WASABI Song Corpus The WASABI Song Corpus is available and accessible in multiple ways and formats. First of all, it can be searched by end-users through the WASABI Interactive Navigator 10  , a rich Web application that relies on a MongoDB back-end server and an Elasticsearch index. It can also be queried by applications in two ways: either using a REST API 11  backed by the MongoDB database, or using our public SPARQL endpoint hosted on a Virtuoso OS triple store  12. Furthermore, the dataset is also provided as a dump that can be downloaded from Zenodo  13. The dump contains the dataset in two different formats: the JSON format extracted from our MongoDB back-end, and the RDF knowledge graph that is described in details in section 9. In all cases (web application, API/SPARQL programmatic access, dumps) all the metadata is publicly available under a Creative Commons Attribution-Non Commercial license.
Getting Access to Copyrighted Content It is important to note that during the project we had access to copyrighted content -song lyrics and audio files -that we are not allowed to include in the publicly released dataset. Nevertheless, significant work has been done on the analysis of the lyrics and the results are available on the project's GitHub (metadata, ML models, Python scripts and Jupyter notebooks). These results open the way to many scientific uses by the NLP community. Notice that an online data source we used for song lyrics at the beginning of the WASABI project (2017-2021), LyricsWikia, is now offline  14. However, it is still possible for researchers to obtain the full lyrics we used using commercial APIs such as MusixMatch  15. Moreover, if you contact the authors, it is possible to obtain our unrestricted lyrics files under conditions of confidentiality and scientific use only.
The WASABI RDF Knowledge Graph The WASABI RDF Knowledge Graph relies upon the Semantic Web standards: RDF provides the conceptual data model and syntaxes, RDFS and OWL provide support for knowledge formalization and reasoning, while SPAR-QL supports querying RDF graphs. All the tools, scripts, configuration and mapping files required to generate the knowledge graph, as well as the WASABI ontology, are available from the project's Github repository  28.
RDF Knowledge Graph Model, Vocabularies and Formalization The first version of the WASABI RDF Knowledge Graph, published in late 2020, contains the metadata of the songs, artists and albums described in Section 2 (date and place of recording, producer, genre, record label, composers, instruments, etc. ), as well as the lyrics summary (Section 4) and explicitness labels (Section 5). The RDF model relies primarily on two vocabularies: the Music Ontology  [51] , a rich vocabulary to describe musical metadata, and the WASABI ontology  [11]  that we designed as an extension of the Music Ontology with respect to specific entities and attributes pertaining to the analysis of song lyrics, such as the detected language and the existence of explicit lyrics in a song. Additionally, the WASABI Knowledge Graph reuses classes and properties from several common metadata vocabularies: Dublin Core Metadata, FOAF, SCOT 29 , Schema.
org and the DBpedia ontology. Specialized terms were also imported from the Audio Features Ontology 30 and the OMRAS2 Chord Ontology 31. We recently published version 2. 0 of the dataset that relies on an extension of the WASABI ontology to account for the identified emotions and topics of a song (Sections 6 and 7 respectively). A subset of the RDF representation of a song is depicted in Figure  18 , with a specific focus on lyrics-related properties. To represent the emotions, we updated the WASABI ontology to model the valence and arousal dimensions of Russel's model as illustrated in Figure  7. Initially, we considered using the EmOCA ontology  [6] , however its use requires to represent emotions with emoca:hasMinimum and emoca:hasMaximum properties that are not relevant in WASABI where the valence and arousal exist as single values.
Interlinking, Querying and Reasoning on the RDF Knowledge Graph The 55 million triples RDF dataset is available as a DOI-identified dump that can be downloaded from Zenodo  34  , and a public SPARQL endpoint  35  set up to enable the community to query the knowledge graph. Complying with Linked Data good practices, all resources in the knowledge graph, may they denote vocabulary terms, concepts or physical objects, are given URIs that i) can be dereferenced to documents describing the resources (selecting RDF or HTML based on content negotiation), and ii) can be shared across datasets, making it possible to interlink resources with rich semantic relationships. In the WASABI RDF Knowledge Graph, all artists, songs and albums are linked to their corresponding web pages in multiple websites. More interestingly, songs and albums are linked to genres named after their DBpedia URIs.
Therefore, beyond looking within the WASABI dataset for songs of a given genre, SPARQL federated queries make it possible to query WASABI and DBpedia simultaneously, so as to aggregate songs pertaining to a given set of genres from both sources. We are currently working on broadening these possibilities of simultaneously querying multiple datasets, relying not only on linked resources but also on external knowledge to improve different reasoning tasks. Following the same direction as the work shown in Application Scenario B (Section 8. 2), we are currently investigating how DBpedia abstracts (text in natural language providing rich information about artists biography, influences, historical and political contexts, interpretations of a song's meaning and so on) can be used to extract Named Entities and the relationships among them.
As shown in Application Scenario C (Section 8. 3), being able to link such Named Entities to structured knowledge bases (such as DBpedia and Wikidata) would allow to push forward with the lyrics understanding through advanced reasoning capabilities. To answer the query about the songs pertaining to protests and rebellions, we could retrieve all songs mentioning Wikidata entities that are instances or sub-classes of the massacre class, or DBpedia entities that are instances or sub-classes of the MilitaryConflict class. Results about all sorts of conflicts would be returned, including e. g. the song Sunday Bloody Sunday. Such query can then be specialized to filter undesired entities (and songs). Furthermore, reasoning can be conveyed by the application of rules implementing specific domain knowledge, meant to infer new facts.
Databases of Songs and Lyrics The Million Song Dataset (MSD) project 36   [7]  is a collection of audio features and metadata for a million contemporary popular music tracks. Such dataset shares some similarities with WASABI with respect to metadata extracted from Web resources (as artist names, tags, years) and audio features, even if at a smaller scale. Given that it mainly focuses on audio data, a complementary dataset providing lyrics of the Million Song dataset was released, called musiXmatch dataset 37. It consists in a collection of song lyrics in bag-of-words (plus stemmed words), associated with MSD tracks. However, no other processing of the lyrics is done, as is the case in our work. MusicWeb and its successor MusicLynx  [2]  link music artists within a Webbased application for discovering connections between them and provides a browsing experience using extra-musical relations.
The project shares some ideas with WASABI, but works on the artist level, and does not perform analyses on the audio and lyrics content itself. It reuses, for example, MIR metadata from AcousticBrainz. The WASABI project was built on a broader scope than these projects and mixes a wider set of metadata, including ones from audio and natural language processing of lyrics. In addition, as presented in this paper, it comes with a large set of Web Audio enhanced applications (multitrack player, online virtual instruments and effect, on-demand audio processing, audio player based on extracted, synchronized chords, etc. ) Some of the goals of the DOREMUS project  [37]  overlap with WASABI, yet in a rather different context. DOREMUS integrates musical metadata from the Bibliothèque nationale de France, Radio France and Philharmonie de Paris.
It focuses specifically on classical and traditional music, therefore it is likely that there is little overlap with WASABI that focuses on popular songs. Furthermore, DOREMUS does not deal with audio signal analysis nor lyrics processing as in WASABI, but integrates MIDI resources to achieve music recommendation and automatic playlists generation. The Listening Experience Database (LED) collects people's music listening experiences as they are reported in documents like diaries, memoirs, letters or oral history recording  [1]. It mostly relates to legacy music that has little overlap with WASABI. The MELD framework  [46]  supports the publication of musicology articles with multi-modal user interfaces that connect different forms of digital resources such as text, audio, video, images and musical scores.
Some development could be undertaken to allow musicologists publish articles that would leverage musical data from the WASABI RDF Knowledge Graph. The MIDI Linked Data project  [40]  publishes a large knowledge graph representing in RDF over 300,000 MIDI files. Each one is linked to its DBpedia counterpart, and the RDF model relies on the MIDI ontology and Music Ontology. As such, MIDI Linked Data can be used as a complement of WASABI to allow working on MIDI files together with the audio and text analyses provided by WASABI. It is worth noticing that some MIDI content was used during the evaluation of the chords extraction in WASABI  [48, 50, 49]  yet not from the MIDI Linked Data project. The works mentioned above are all about public databases. Let us however notice that companies such as Spotify, GraceNote, Pandora, or Apple Music have sophisticated private knowledge bases of songs and lyrics to feed their search and recommendation algorithms, but there are not publicly available (and mainly rely on audio features).
Their goal is not to label a set of lines in a given way (e. g. verse, chorus), but rather identifying recurrent as well as non-recurrent groups of lines. They propose a rule-based method to estimate such structure labels of segmented lyrics. Explicit Content Detection. Bergelid  [5]  consider a dataset of English lyrics to which they apply classical machine learning algorithms. The explicit labels are obtained from Soundtrack Your Brand 38. They also experiment with adding lyrics metadata to the feature set, such as the artist name, the release year, the music energy level, and the valence/positiveness of a song. Chin et al. [19]  apply explicit lyrics detection to Korean song texts. They also use tf-idf weighted BOW as lyrics representation and aggregate multiple decision trees via boosting and bagging to classify the lyrics for explicit content.
Conclusion In this paper we have described the WASABI Song Corpus, focusing in particular on the lyrics annotations resulting from the applications of the methods we proposed to extract relevant information from the lyrics. So far, lyrics annotations concern their structure segmentation, their topic, the explicitness of the lyrics content, the summary of a song and the emotions conveyed. We motivated using our corpus by presenting three different application scenarios. While parts of them are not yet fully implemented, we described our progress in transforming our dataset into a knowledge graph, the WASABI RDF Knowledge Graph, which will be vital both in these implementations and in enabling further interesting application scenarios in future work.
ABSTRACT The monitoring of biological indicators is required to assess the impacts of environmental policies, compare ecosystems and guide management and conservation actions. However, the growing availability of ecological data has not been accompanied by concomitant processing tools able to facilitate data handling and analysis. Multiple common challenges limit the usefulness of biomonitoring information across ecosystems and biological groups. Biomonitoring data analysis is currently constrained by time-consuming steps for data preparation and a data processing environment with limited integration in terms of software, biological groups, and protocols. We introduce biomonitoR, a package for the R programming language that addresses technical challenges for the management of ecological data and metrics calculation.
biomonitoR implements most of the biological indices currently used or proposed in different fields of ecology and water resource management. Its combination of customizable functions aims to support a transferable and comprehensive biomonitoring workflow in a user-friendly environment. biomonitoR represents a versatile toolbox with five main assets: (i) it checks taxonomic information against reference datasets allowing for customization of trait and sensitivity scores; (ii) it supports heterogeneous taxonomic resolution allowing computations at multiple taxonomic levels; (iii) it calculates multiple biological indices, including metrics for both broad and stressorspecific ecological assessments; (iv) it enables user-friendly data visualization, helping both decision-making processes and data interpretation; and (v) it allows working with an interactive web application straight from R.
g. ,  Grenié et al. , 2022). Such heterogeneous sources and data availability brought new challenges and have not been adequately coupled with corresponding processing tools able to fully explore the great potential of biological databases, especially from a monitoring perspective. Due to their long-standing exposure to human exploitation  (Feio et al. , 2021; Friberg, 2014) , the case of freshwater ecosystems is emblematic. A multitude of information has been produced and indices have been developed for freshwater systems, especially following the implementation of environmental legislation such as the Water Framework Directive in Europe, the Clean Water Act in USA or the Canadian Protection Act  (Birk et al. , 2012; Vitecek, Johnson & Poikane, 2021). In this case, biomonitoring relies on a set of biological elements such as phytoplankton, phytobenthos, macrophytes, macroinvertebrates, and fish for assessing the status and integrity of communities and ecosystems  (Birk et al.
In this context, an integrative data processing environment for biomonitoring is still lacking and would facilitate data-processing and index calculation, which are currently poorly integrated in terms of specific national protocols and informatic programs. One of the most important data-processing challenges is addressing the inconsistency among taxonomic, trait, and sensitivity scores because mismatches can lead to excluding taxa from analyses  (Guareschi & Wood, 2019). Likewise, for highly diverse taxa (e. g. , invertebrates), data frequently relies on mixed taxonomic resolution (e. g. , species, genus, and family) due to inherent difficulties in identifying taxa at species level (e. g. , in early life stages)  (Jones, 2008). Heterogeneous taxonomic resolution and nomenclature discrepancies complicate assigning sensitivity scores or trait values to taxa and requires time-consuming data manipulations to properly organize the dataset.
A thorough exploitation of these data, which are often only partially explored, will optimize the use of ecological data and reduce inefficiencies  (Patterson et al. , 2010; Grenié et al. , 2022). Similarly, functional indices calculated from trait-based information require complex calculations that may prevent their wide use in biomonitoring  (Maire et al. , 2015). Here, we introduce biomonitoR, a new R package (R Core Team, 2021) supporting the calculation of a wide range of biomonitoring indices and the effortless management of biological information from different sources (Fig. 1 ). The package favours a smooth and reproducible workflow in biomonitoring science and allows both new operations (e. g. , metrics calculation and traits manipulation) as well as existing ones currently requiring the use of several packages.
Import data The first step consists in comparing user's data to a reference taxonomic dataset with the function as_biomonitor. User's data consists in a data. frame with taxa on the first column and occurrences/abundances on the following columns, while the taxonomic reference dataset includes the relationships of taxa among levels. biomonitoR comes with four built-in reference datasets for diatoms, macrophytes, macroinvertebrates and fish derived from Diat. barcode  (Rimet et al. , 2019) , AlgaeBase  (Guiry et al. , 2014 ), freshwatercology. info (Schmidt-Kloiber & Hering, 2015) , and FishBase  (Froese & Pauly, 2019) , respectively. To ensure an up-to date and global taxonomy information, several functions (e. g. , get_gbif_taxa_tree) are already available, permitting the user to build a biomonitoR reference dataset from online resources such as the Global Biodiversity Information Facility (GBIF), World Register of Marine Species (WoRMS), National Biodiversity Network Atlas (NBN Atlas) and International Union for Conservation of Nature (IUCN) (SM1).
Moreover, users can also provide their own reference dataset or build it from a taxonomic tree with the function ref_from_tree. This is a crucial step in data handling to organize and clean the dataset as well as unify writing styles (e. g. , lexical variants). Taxa names that do not match with any taxon of the reference taxonomy are excluded to ensure consistency of further steps. Nonetheless, suggestions are proposed either silently or interactively by setting the correct_names argument to TRUE. To explore the general data structure, the object generated by as_biomonitor can be subsetted or plotted by leveraging on the interactive plotly package  (Sievert, 2020). The second step consists in using the function aggregate_taxa to aggregate taxa at.
Multiple taxonomic levels and back trace biomonitoR calculates ecological indices at different taxonomic levels. The desired taxonomic level (e. g. , species, family) can be specified with the argument tax_lev. This is a key feature to develop robust methods based on taxonomic sufficiency, the level of taxonomic detail to which organisms must be identified to recognize ecological patterns  (Jones, 2008). To keep track of the computations performed by biomonitoR, the original information can be obtained by setting the traceB argument to TRUE. This is especially useful when mismatches between taxonomic and sensitivity/traits datasets lead to the exclusion of some taxa. To further assure the transparency of the entire process, default scores are made available by using the show_scores function.
Tools for managing trait-based information and indices availability biomonitoR allows users to overcome problems frequently arising during the computation of trait-based indices. The package includes several functions to manage functional traits information allowing the calculation of a range of functional metrics most of which are based on the FD  (Laliberté & Legendre, 2010)  and ade4 packages  (Dray & Dufour, 2007). For example, the function assign_traits matches the taxa to those of the traits dataset. This is particularly useful when traits of several taxa in the trait dataset (e. g. , species-and genus-level) can be assigned to a higher taxonomic level (e. g. , family level). Currently, biomonitoR comes with a built-in macroinvertebrate trait dataset (European fauna: Schmidt-Kloiber & Hering, 2015;  Tachet et al.
, 2010)  but users can also add their own (see an example of loading a dataset including North American insect traits in SM3). A finer control over trait information can be achieved with the manage_traits function, that allows users to select the traits belonging to the nearest taxon in the taxonomic tree (e. g. , SM3). Finally, traits at fine levels (e. g. , species or genus) can be averaged at coarser levels (family) (average_traits) or a random taxon (e. g. , genus) can be selected as representative (sample_traits). The calculation of some functional indices (e. g. , functional richness, dispersion and evenness) requires summarizing trait variation into a low dimensional trait space. This can be achieved through a principal coordinate analysis based on a matrix of Euclidean distances or Gower dissimilarities.