dataesr/scientific-paragraphs-categorization huggingface dataset:
Dataset and Evaluation Protocol The new data includes about 350 hours of manually annotated soundscapes from past editions and soundscapes that were recorded using 30 field recorders between January and June of 2017 in Ithaca, NY, USA. This dataset was split into a validation set with labels provided to the participants (about 10%) and a test set to be processed by the evaluated systems. As for training data, we provided an newly composed Xeno-Canto subset covering 659 species from South and North America. Additionally, eBird. org frequency lists were provided to enable participants to decide which species are plausible for a given time, date and location. The goal of the task was to localize and identify all audible birds within the provided soundscape test set. Each soundscape was divided into segments of 5 seconds, and a list of species associated to probability scores had to be returned for each segment.
Participants and Results 103 participants registered for the BirdCLEF 2019 challenge and downloaded the dataset. Five of them succeeded in submitting runs. Details of the methods and systems used in the runs are synthesized in the overview working notes paper of the task  [23]  and further developed in the individual working notes of the participants (MfN  [26] , ASAS  [6] , NWPU  [21] , MIHAI  [8] ). In Figure  4  we report the performance achieved by the 25 collected runs. In this edition, participants built on established systems from previous years, all submitted runs featured a CNN classifier trained on spectrograms-very deep networks once again performed best. Participants were able to significantly improve the detection performance. In fact, we saw an increase of more than 180% for the best performing runs (2018: 0.
Instead of learning a model from environmental feature vectors, the goal of the task will be to learn a model from k-dimensional image patches, each patch representing the value of an environmental variable in the neighborhood of the occurrence. As last year, the task consists of predicting plant species from location, but we added a very large and newly published dataset of plant occurrences from a citizen science project. We also proposed to participants to use an even bigger dataset of non-plant species that might interact with plants.
Data Set and Evaluation Protocol Training set -The training data provided for the task included three distinct occurrences data sets: -Pl@ntNetFranceRaw: 2,367,145 occurrences of plants that were collected via the Pl@ntNet application and automatically identified (using a convolutional neural network). These original data is described and permanently hosted in  [3]. -Pl@ntNetFranceTrusted: a subset of Pl@ntNetFranceRaw including only the occurrences for which the prediction score (softmax output of the CNN) was higher than a threshold equal to 0. 98. -GBIFPlantFrance: 291,392 occurrences of 3,336 plant species collected by experts on the French territory between 1835 and 2017 (coming from the GBIF database 11. -GBIFAllFrance: 10,618,839 occurrences of species from other kingdoms than plants including mammals, birds, amphibians, insects and fungus (also coming from the GBIF database).
Environmental data -We provided 33 geographic rasters of various spatial resolutions containing containing bioclimatic, pedologic, topologic, hydrographic and land cover variables suited for modeling plant species distributions. The original data compilation is freely downloadable and described in details at  [2]. We also provided a python tool allowing to extract the automatically environmental patches: A 3 dimensions array where each layer is the is a window matrix cropped into one raster, and centered at the specified location. Test set -We used 25,000 plant occurrences of high location accuracy (inferior to 50 meters) and identification certainty collected by the Mediterranean National Botanical Conservatory (CBNmed) and their partners over the French Mediterranean region. They have been selected to insure that spatial coverage is uniform and that locally each present species have an equivalent number of occurrences.
Participants and Results 61 participants registered for the GeoLifeCLEF 2019 challenge and downloaded the dataset. Five of them succeeded in submitting 44 runs in total. Details of the methods and systems used in the runs are synthesized in the overview working note paper of the task  [4]  and further developed in the individual working notes of the participants (LIRMM  [30] , SaraSi  [33] , SSN CSE  [25] , sergiu atodiresei  [1]  and Lot of Lof  [29] ). In Figure  5  we report the performance achieved by the 44 collected runs. The 5 best runs of this challenge all used Convolutional Neural Network models applied to environmental patches, which confirms results of last year edition. This performance gap might be also due to the fact that those models training included both Pl@ntNetFranceRaw and GBIFPlantFrance plant occurrences, whereas non-CNN methods only used Pl@ntNet occurrences.
Acknowledgements We would like to thank very warmly  Julien Engel ,  Rémi Girault ,  Jean-François Molino  and the two other expert botanists who agreed to participate in the task on plant identification. We also we would like to thank the  University of Montpellier  and the  Floris ' Tic  project ( ANRU ) who contributed to the funding of the 2019-th edition of LifeCLEF.
As the Web of Linked Open Data is growing the problem of crawling that cloud becomes increasingly important. Unlike normal Web crawlers, a Linked Data crawler performs a selection to focus on collecting linked RDF (including RDFa) data on the Web. From the perspectives of throughput and coverage, given a newly discovered and targeted URI, the key issue of Linked Data crawlers is to decide whether this URI is likely to dereference into an RDF data source and therefore it is worth downloading the representation it points to. Current solutions adopt heuristic rules to filter irrelevant URIs. Unfortunately, when the heuristics are too restrictive this hampers the coverage of crawling. In this paper, we propose and compare approaches to learn strategies for crawling Linked Data on the Web by predicting whether a newly discovered URI will lead to an RDF data source or not.
Introduction Linked Data extends the principles of the World Wide Web from linking documents to that of linking pieces of data to weave a Web of Data. This relies on the well-known linked data principles  [10, 1]  including the use of HTTP URIs that can be dereferenced and the provision of useful and linked description upon access so that we can discover more things. Recently, a large amount of data are being made available as linked data in various domains such as health, publication, agriculture, music, etc. , and the Web of Linked Data is growing exponentially  [8]. In order to harvest this enormous data repository, crawling techniques for Linked Data are becoming increasingly important. Different from conventional crawlers, crawling for Linked Data is performed selectively to collect structured data connected by RDF links.
The target of interest makes it distinct from focused crawlers which select the collection of Web pages to crawl based on their relevance to a specific topic. The design objective of Linked Data crawlers is to fetch linked RDF data in different formats -RDF/XML, N3, RDFa, JSON-LD, etc. -as much as possible within a reasonable time while minimizing the download of irrelevant URIs -i. e. leading to resources without RDF content. Therefore our challenge is to identify as soon as possible the URIs referencing RDF sources without downloading them. The research question here is: can we learn efficient URI selection criteria to identify sources of Linked Open Data?. To solve this problem, in this paper we propose and compare methods on real data to predict whether a newly discovered URI will lead to RDF data source or not.
We extract information from the targeting and referring URI and from the context (RDF data graph) where URIs and their links were discovered in order to produce features fed to learning algorithms and in particular to an FTRLproximal online method employed to build the prediction model. FTRL-proximal is a time efficient and space efficient online learning algorithm, which can handle significantly larger data sets. It is also effective at producing sparse models which is important when the feature space is huge. FTRL-proximal outperforms the other online learning algorithms in terms of accuracy and sparsity. The contributions of this work include: 1) we identify the features to predict whether a target URI will lead to some RDF data or not; 2) we adapt the FTRLproximal algorithm to our task and build an online prediction model; and 3) we implement a Linked Data crawler with the online prediction model and evaluate its performance.
Related Work Semantic Web/Linked Data Crawlers. Semantic web crawlers differ from traditional web crawlers in only two aspects: the format of the source (RDF format) it is traversing, and the means to link RDF across data sources. There exist some work  [6, 13, 12]  in the field of Semantic Web/Linked Data crawling. The two main representative crawlers for Linked Data are LDSpider  [13]  and SWSE crawler  [12]. They crawl the Web of Linked Data by traversing RDF links between data sources and follow the Linked Data principles  [10, 1]. They offer different crawling strategies such as breadth-first and load-balancing traversal. In order to reduce the amount of HTTP lookups and downloading wasted on URIs referencing non-RDF resources, these previous works apply heuristic rules to identify relevant URIs  [13, 12].
As a result, a huge volume of RDF data is missed by these methods. Focused Crawlers on the Web. Traditional focused crawlers on the Web aim to crawl a subset of Web pages based on their relevance to a specific topic  [4, 5]. These methods build a relevancy model typically encoded as a classifier to evaluate the web documents for topical relevancy. Our work here is different in the sense that we do not filter on the topics but on the type of content. Meusel et al. [17]  proposed a focused crawling approach to fetch microdata embedded in HTML pages. Umbrich et al. [18]  built a crawler focused on gathering files of a particular media type and based on heuristics. Online Prediction. Our problem is also related to the classic online binary prediction in which data becomes available in a sequential order and the learner must make real-time decisions and continuously improve performance with the sequential arrival of data.
Preliminary Knowledge In this section, we introduce some basic definitions and notations used throughout the paper. The Linked Data crawler targets a kind of structured data represented in RDF format on the Web. The Resource Description Framework (RDF) provides a structured means of publishing information describing entities and their relationships in RDF triples. The main concepts of RDF and Linked Data we need here are: Definition 1. (RDF Triple) A triple t = (s, p, o) ∈ (U ∪ B) × U × (U ∪ B) × (U × B × L) is called an RDF triple where U denotes the set of URI, B the set of blank nodes and L the set of literals. In such a triple, s is called subject, p predicate, and o object. Definition 2. (RDF Graph) An RDF graph G = (V, E) is a set of RDF triples such that V is the node set and E is the edge set of G.
Task description Since the task of Linked Data crawler is to fetch RDF data on the Web we are interested in a kind of URIs that we call RDF-relevant URIs: Definition 4. (RDF-Relevant) Given a URI u, we consider that u is RDFrelevant if the representation obtained by dereferencing u contains RDF data. Otherwise, u is called non RDF-relevant. We note U R the set of RDF relevant URIs and U I the set of non RDF-relevant URIs with U = U R ∪U I. For the URIs that have certain file extensions such as *. rdf/owl or HTTP Content Types Headers such as application/rdf+xml, text/turtle, etc. , it is trivial to know the RDF-relevance of them. In this work, we focus on a knid of URIs called hard URIs whose RDF-relevance cannot be known by these heuristics.
Definition 5. (Hard URI) We call u a hard URI if the RDF relevance of u cannot be known straightforwardly by its file extension or HTTP Content-Type Header. For example, URI u with HTTP Content-Type header text/html is a hard URI since RDFa data could be embedded in u. As reported in  [9] , the URIs with HTTP Content-Type Header text/plain may contain RDF data so they are hard URIs too. Then, for our prediction task, we consider four types of URIs involved in the prediction we want to make: the target URI, the referring URI, direct property URIs and sibling URIs. Definition 6. (Target URI) We call target URI and note u t the URI for which we want to predict if the deref (u t ) will lead to a representation that contains RDF data i. e. if u t is RDF-Relevant. Definition 7. (Context RDF Graph) Given an RDF relevant URI u r containing the RDF graph G r t , we define G r t = (V, E) as the context RDF graph of URI u t if u t appears in G r t as a node, i.
rdf/owl. Inversely, URIs with the file extension *. html/htm are included in the training set since they may contain RDFa data. The true class label y t is required to update the predictor online. In our scenario, to observe the true class label of a URI we have to download it and check whether it contains RDF data or not. We cannot afford to download all URIs because of the network overhead, and our target is to build a prediction model that avoids downloading unnecessary URIs. There, an appropriate subsampling strategy is needed. We found that the positive URIs are rare (much less than 50%) and relatively more valuable. For each round, if URI u t is predicted positive namely ŷt = 1, we retrieve the content of u t and observe the real class label y t. Then the predictor can be updated by new training instance (y t , x t ).
Implementation of Crawler We now detail the prototype we tested for the Linked Data crawler and explain the Algorithm 2 it implements. Initialization. As shown in Algorithm 2, the proposed crawler starts from a list of seed URIs and operates an breadth-first crawling since this often leads to a more diverse dataset instead of traversing deep paths within some given sites. The maximum crawl depth d max is set for crawling. The F rontier data structure is initialized by the seed URI list S. Politeness. At the beginning of each round, a naive crawler obtains a URI from F rontier to retrieve. However, it would lead to the problem that the crawler issues too many consecutive HTTP requests to a server and is considered "impolite" by the server. Thus, we group the URIs in F rontier into different sets pld 0.
Feature Set Evaluation In this experiment, we evaluate the predictive ability of different combinations of feature sets introduced in Section 4. 2 by several offline/batch classifiers. We firstly introduce the dataset, metrics and offline classifiers used in the experiment and then discuss the results of the experiment. Dataset. The dataset used in the experiment is generated by operating a Breadth-First Search (BFS) crawl. The crawl starts a set of 50 seed URIs which are RDF relevant and randomly selected from 26 hosts. During the crawl, we only keep the hard URIs whose RDF relevance cannot be known straightforwardly. Finally we generate a dataset with 103K URIs. The dataset includes 9,825 different hosts. For each URI, we generate features and the class label. To check RDF relevance for each URI, we use the library Any23 4.
Online Versus Offline In this experiment, we evaluate the performance of the proposed online prediction method against several baseline methods. Metrics. The aim of the Linked Data crawler is to maximize the number of RDF-relevant URIs collected while minimizing the number of irrelevant URIs downloaded during the crawl. For our proposed method, the crawler has to download a fraction of URIs even though they are predicted negative. To better evaluate the performance of our approach, we use a percentage measure that equals the ratio of retrieved RDF-relevant URIs to the total number of URIs 5  crawled: percentage = #Retrieved RDF relevant U RIs #All retrieved U RIs Methods. We implemented the proposed crawler denoted by LDCOC (Linked Data Crawler with Online Classifier). The decision threshold τ in Algorithm 1 is set to 0.
5 and the parameter of LDCOC is set to 0. 17 according to the ratio of the number of positive URIs to the number of negative URIs in the training set used in Section 6. 1. As baselines, we also implemented three crawlers with offline classifiers including SVM, KNN and Naive Bayes to select URIs. The classifiers are pre-trained with two training sets (with size 20K and 40K). The BFS crawler is another baseline method to be compared to. As suggested in Section 6. 1, we use the feature set F +t+r+p for the experiment. Results. Table  2  shows the percentage of retrieved RDF relevant URIs by different crawlers after crawling 300K URIs. The results show that the crawlers with offline classifiers perform slightly better than BFS crawler. Considering that the Linked Data Web is a dynamic environment, the crawlers with offline classifiers pre-trained by a small size training set would not improve the performance a lot.
Conclusion We have presented a solution to learn URI selection criteria in order to improve the crawling of Linked Open Data by predicting their RDF-relevance. The prediction component is able to predict whether a newly discovered URI contains RDF content or not by extracting features from several sources and building a prediction model based on FTRL-proximal online learning algorithm. The experimental results demonstrate that the coverage of the crawl is improved compared to baseline methods. Currently, this work focuses on crawling linked RDF (RDFa) data. Our method can now be generalized to crawl other kinds of Linked Data such as JSON-LD, Microdata, etc. For future work, we are investigating more features such as the subgraphs induced by URIs and additional techniques such as graph embedding to further improve the predictions.
The contributions of this paper are three-fold. First, we provide a comprehensive overview of existing metrics for summary evaluation. We discuss several limitations of existing frameworks for summary evaluation. Second, we introduce an automatic framework for the evaluation of metrics that does not require any human annotation. Finally, we evaluate the existing assessment metrics on a Wikipedia data set and a collection of scientific articles using this framework. Our findings show that the majority of existing metrics based on vocabulary overlap are not suitable for assessment based on comparison with a full text and we discuss this outcome.
However, in turn, the assessment metric should be meticulously chosen. Hence, a second contribution of this paper is to introduce an automatic framework to evaluate assessment metrics that does not require any human annotation. Finally, as a third contribution, we evaluate the main existing metrics from the literature on two different data sets: a Wikipedia data set and a collection of scientific articles. Our experiments show that the majority of existing metrics based on vocabulary overlap are not suitable to assess the quality of produced summaries when compared to the original full text. The rest of the paper is organized as follows. Firstly, in Section 2, we provide a description of the most important international evaluation programs and collections for text summarization. Then, in Section 3 we present an overview of existing assessment metrics.
Evaluation Programs and Collections An acute need for automatic text compression has led to the emergence of a number of evaluation programs sponsored by research and governmental institutions. These programs include reference data and metrics. Shortly after the Internet explosion in the late 90s, the first annual summary evaluation campaign DUC (Document Understanding Conference) appeared. The first evaluation programs were general-purpose, while more recent ones cover specific domains, e. g. biomedical literature, search engine snippets, or are targeting specific applications, e. g. tweet contextualization within the Initiative for the Evaluation of XML Retrieval (INEX). These assessment programs provide evaluation frameworks containing text collections, evaluation metrics, and gold standards (also called reference summaries or ground truth).
Reference summaries were written by 4 different NIST assessors. In 2005, the task was also answering complex questions, but the documents were chosen from Financial Times of London and Los Angeles Times  [13]. The update task was introduced in 2007 and aimed to produce 100-words multi-document update summaries of news articles under the assumption that the user has already read earlier articles. This task was kept in TAC campaign. The DUC campaign saw large improvements in participants' automatic systems with regards to annotators' proposals. However, there were limitations to participants' summaries since they were topic-dependent and extractive. The DUC data set contains: documents, manually created summaries, automatic baselines, summaries submitted by the participants, evaluation results, additional supporting data and software.
In 2011, participating metrics should also be applicable on the level of individual summaries. Besides content evaluation, in 2011 the readability measuring should be also considered. Participants obtained the test data from the TAC Guided Summarization task, the humanauthored and automatic summaries from that task, namely: • topic statements; • two sets of 10 documents for each topic; • four human written reference summaries for each document set; • summaries evaluated in the TAC Guided Summarization task; • list of topic categories and target aspects. The output of participants automatic metrics was compared against manual metrics: (1) the Pyramid score for content evaluation; (2) Overall Readability for linguistic quality evaluation; and (3) Overall Responsiveness, which is a combined measure for content and linguistic quality evaluation.
Pearson's, Spearman's, and Kendall's correlation coefficients between scores assigned by each automatic metric and the three manual metrics were computed. The assumption under the evaluation framework was that a good automatic metric should make the same significant difference between automatic summarizing systems as the manual metrics, but should not provide a ranking to two summarizing systems contradicting the manual metric. Thus, each run was evaluated by (1) the correlation with the manual metric and (2) its discriminative power compared with the manual metric. TAC Summarization Track was not running in 2012 and 2013. In 2014, TAC restricted the summarization task with only biomedical literature in the Biomedical Summarization track. Nowadays, this task is extremely important since the integration of big data into health care could save around $1000 a year per person according to recent studies  [1].
SanJuan et al. took another step by introducing Tweet Contextualization (TC) Track In 2011, the Question Answering Track aimed to evaluate TC in terms of (1) relevance of the retrieved information to tweets, and (2) readability of the presented results  [21]. In 2012, this track was renamed Tweet Contextualization. In 2012, answers needed to be a concatenation of textual passages from an external textual resource providing background information that helped to understand the tweet content. Providing an answer to these kinds of task (retrieval and concatenation of relevant textual passages), thus, implies using text summarization. Associated test collections are described in the table  1. In 2011, the query data set included 132 tweets. Each tweet consisted of the identifier (id), the title (title), and the first sentence (txt) of a New York Times article published in July 2011.