dataesr/scientific-paragraphs-categorization huggingface dataset:
For simplification, in this first report only those stations with validated hourly sampling data have been used. This has reduced significantly the number of stations (79). This number will increase in future reports when adequate standard data processing is applied to all the time series. This simplification, unfortunately, may bias the results when basin-wide results are presented, as now there is a lack in information from many tide gauges in the most important storm surge regions (e. g. the Netherlands, Germany or France). For the 79 stations available (displayed in Figures  56 57 58 ) tidal constants, tide prediction and surge component (hourly sea levelhourly tide) were computed. The latter, analysed and presented here, represents mainly atmospherically forced components of sea level (atmospheric pressure and wind), providing information on storm surge events and storm variability.
The storms also caused surges on the west coast as well as in the fjords (e. g. Limfjorden) in Denmark. Concerning the adequacy of sea level time series available in CMEMS, it has become evident that there is a need for establishing homogeneous requirements on data sampling within CMEMS in order to avoid the diverse sampling intervals along the same time series. This is important because different physical processes can be adequately resolved with these different sampling strategies. On the other hand, it was found that only about 18% of the stations had data spanning 1993-2015 and that many existing stations were not yet available in the service. All these issues will be discussed in detail in future.
Exceptionally strong sea surface temperatures occurred in the entire Northern hemisphere (Section 1. 1). One of the lowest Arctic summer sea ice extent ever measured over the past 20 years was reported (Section 1. 7). In the North-West Shelf area, higher surface currents and strong frequency of extreme storms and sea level events were reported (Section 3. 2). In the Baltic Sea, basin-wide higher sea level, strong frequency of extreme events like storms and sea    4, 3. 4, 4. 3 and 4. 4). As discussed in more detail in the introduction, this report is only a firstbut importantstep towards a more comprehensive and systematic reporting of the state of the marine environment. Firstly, the report greatly improved the scientific exploitation of CMEMS comprehensive marine environmental data products.
The current decade is a witness to an enormous explosion of data being published on the Web as Linked Data to maximise its reusability. Answering questions that users speak or write in natural language is an increasingly popular application scenario for Web Data, especially when the domain of the questions is not limited to a domain where dedicated curated datasets exist, like in medicine. The increasing use of Web Data in this and other settings has highlighted the importance of assessing its quality. While quite some work has been done with regard to assessing the quality of Linked Data, only few efforts have been dedicated to quality assessment of linked data from the question answering (QA) perspective. From the linked data quality metrics that have so far been well documented in the literature, we have identified those that are most relevant for QA.
We apply these quality metrics, implemented in the Luzzu framework, to subsets of two datasets of crucial importance to open domain QA -DBpedia and Wikidata -and thus present the first assessment of the quality of these datasets for QA. From these datasets, we assess slices covering the specific domains of restaurants, politicians, films and soccer players. The results of our experiments suggest that for most of these domains, the quality of Wikidata with regard to the majority of relevant metrics is higher than that of DBpedia.
Quality can generally be defined as "fitness for use", but there are a lot of concrete factors that influence a dataset's fitness for use in question answering 2 settings and in specific application domains. Recently, a number of research activities have been concerned with automating the assessment of linked data quality by benchmarks such as Luzzu  [7] , RDFUnit  [25] , and others. In this paper, we compile a list of metrics that are applicable for assessing the quality of Linked Open Data (LOD) for question answering. We support this list by determining the quality of various subsets of Wikidata 3 and DBPedia 4 , of which both are widely used in linked data based question answering, such as SINA  [36] , BioASQ  [38] , QALD tasks  [39] , and others. To put the reader into the context of our work, we present a scenario and discuss it from two different perspectives: the data consumer's (i.
e. application developer's) perspective and the user's perspective. Data consumer's perspective: Data Caterer GmbH, a data science company, has been providing search and discovery services in various domains, including films and soccer. To cover these domains, they have access to a wide range of cross-domain datasets. To serve users' needs, they specialise on mobile question answering applications on top of a generic, open domain core engine. Data Caterer would now like to enter the market of restaurant recommendations and is facing fierce competition from other providers such as Zomato 5. To provide the best answers to the users' questions, it is crucial to use the right dataset, i. e. one that is rich in domain-specific information and correct. This task poses two challenges: 1. How to perform an initial data quality assessment? -Different datasets from different sources, which may have been created using different techniques, usually vary in quality w.
r. t. level of detail, redundancy and correctness. Assessing the quality of the datasets in an initialisation step, before actually answering questions, supports the system in delivering faster output, which, moreover, is precise. 2. Which subsets of a given dataset are specifically suitable for answering questions related to given topic? -Cross-domain datasets, such as datasets obtained from Wikipedia, cover many domains, but how well do they cover the specific domain of the question, i. e. , here, the restaurant domain? We need to assess the quality not of the dataset as a whole, but of certain subsets (slices) of it. User's perspective: Walter enjoys savouring a variety of cuisines. He came to Berlin for Christmas holidays and wants to find the best Fränkische Bratwurst in town. He uses the Data Caterer mobile app, types in a question and gets the precise location of the Frankenstüberl restaurant as a quick response.
STATE OF THE ART In this section, we summarise the state of the art of datadriven question answering and data quality assessment, and highlight the gaps that our research aims to address. The goal of a question answering system is to satisfy the user's information need (represented as a natural language query or keywords) by returning the best answer (a ranked list of documents, URIs, specific excerpts from text, etc. ). Closed domain question answering systems use context/application specific datasets, ontologies and thesauri (for instance, MeSH 6  ). The coverage or scope of information in the dataset is a specific application domain, e. g. , medical, financial, geographical. Such datasets are typically collected, curated and validated by domain experts such as, in the medical domain, doctors, clinicians, or physicians, to ensure correctness of the results.
In contrast, open domain question answering relies on a wide spectrum of datasets, ontologies and thesaurus to answer any question regardless of its category. Using multiple data sources poses greater challenges to the evaluation and the performance of the question answering system, mainly because of differences in the structure (structured, semi-structured or unstructured) and quality of the data. Furthermore,  [39]  and  [37]  report a number of challenges, related to data quality issues, affecting the overall system performance of QA systems and advocate the need to formally consider these factors for question answering systems. A number of studies have identified, defined and classified data quality dimensions and metrics  [3, 4, 29, 43]. Zaveri et al. present an exhaustive state-of-the-art survey on quality dimensions and metrics relevant for assessing linked data  [44].
However, only a handful of these dimensions and metrics are relevant for answering questions (tackling Obj 1, Obj 2 -cf. Section 1. 2) over linked datasets on the Web, as we will explain in Section 3. The question answering community has invested substantial effort into evaluating the performance of their systems, targeting both open and closed domain question answering. Regular challenges include the NTCIR Question Answering Challenge (QAC)  [14] , TREC  [41] , CLEF  [21] , BioASQ  [38] , and QALD series  [27]. The overall performance of a questions answering system is typically evaluated by measures such as precision, recall, F-score, and also, in certain tasks, by the global F-score w. r. t. the total number of questions, as in QALD-5  [40]. However, these metrics for evaluating question answering systems do not provide any information on the quality of data being used to answer the questions (tackling Obj 3, Obj 4 -cf.
General Terminology Before presenting those quality dimensions and metrics that we have identified as relevant to open domain question answering, we introduce the basic terminology used in the remainder of this section. • A quality dimension is a characteristic of a dataset. The importance of a dimension depends on the context in which a dataset is intended to be used. For instance, availability of dataset is crucial for a question answering system that depends on a SPARQL endpoint for retrieving answers but not for a question answering system that consumes offline data indexed locally. • Bizer and Cyganiak define a metric as a "procedure for measuring a[n. ] quality dimension"  [4]. Metrics "rely on quality indicators and calculate an assessment score from these indicators using a scoring function"  [4].
Availability Dimension Zaveri et al. define availability of a dataset as "the extent to which data (or some portion of it) is present, obtainable and ready for use"  [44]. Flemming refers to availability as the proper functioning of all access methods  [12]. We adopt a hybrid definition of both of these: Definition 1 (Availability). Availability is the degree of readiness of a dataset, ontology or thesaurus for consumption. Availability can be established, e. g. , by providing the dataset as an RDF dump, exposing it via a SPARQL endpoint, or via some other API, package or library -i. e. in whatever form a question answering system may prefer to consume. Some of these forms may be provided by third parties. We use the following metrics to measure availability: Metric A. 1 (Dereferenceability).
This metric measures the number of valid 303 or hash URIs among all resource URIs in a dataset, following the LOD best practices  [35]. As typical datasets for question answering are large, an approximate metric using statistical sampling technique  [8]  was used. Metric A. 2 (Dereferenceability of Forward Links). This metric assesses the extent to which a resource includes all triples from the dataset that have the resource's URI as the subject  [23]. To do so, it computes the ratio between the number of triples that are "forward-links" a. k. a. "out-links" (i. e. that have the resource's URI as their subject) and the total number of triples in the RDF graph served as a description of each resource.
Metric A. 3 (No Misreported Content Types ). This metrics checks whether RDF data is served with the right content type, e. g. , whether RDF/XML content is served as application/rdf+xml (cf. [44] ). This is done by checking the content type returned when dereferencing a URI and checking whether the content can be processed by a parser for the expected format. Metric A. 4 (RDF Availability). This metric checks whether a syntactically valid RDF representation of a dataset can be downloaded (cf. [44] ). Metric A. 5 (Endpoint Availability). This metric checks whether a SPARQL endpoint for a dataset is available, i. e. queryable. Relevance: In the context of open domain question answering systems, the relevance of the availability dimension is system dependent. The dimension is crucial, for example, for systems that leverage a remote SPARQL endpoint.
The output of such systems relies on the availability of the data in a specific form to be able to query the dataset and return the result. If the endpoint is not responsive or responds erroneously, then the system returns no answer and thus becomes unable to serve the user's request. However, systems that maintain offline copies of datasets and locally computed indexes are less prone to the availability of the original datasets. More generally, the availability dimension is crucial for any system that relies on a remote third party package, tool or library that gives access to a specific data source.
Interlinking Dimension Interlinking refers to connections between two terms with different names (URIs) that have the same meaning. For instance, "heart attack" and "myocardial infarction" are terms from different subdomains of medicine but have the same meaning. Such terms may occur in the same dataset, or across different datasets. New datasets are typically interlinked with "reference" datasets to aid customers in finding further relevant information. Large, widely known datasets such as DBpedia, but also high-quality curated domainspecific datasets typically serve as such references. In LOD, interlinking is typically implemented as RDF triples that connect a subject and an object by the owl:sameAs property. The following definition is paraphrased from  [44]. Definition 2 (Interlinking).
Metric I. 6 (Interlink Detection). Guéret et al. reuse five network measures to assess a dataset's degree of interlinking [18]: The same-as measure detects open owl:sameAs chains. owl:sameAs resources should be symmetric, in a sense that if X owl:sameAs Y then Y owl:sameAs X. The latter is called a closed owl:sameAs chain. The descriptive richness measure assesses the amount of new properties added to a resources through an owl:sameAs relation. The degree measure checks the number of in-links and out-links of a resource. A high degree measure means that agents can find information easily by means of traversal. The centrality measure checks the dependency of a resource in a dataset. Finally, the clustering coefficient measure aims at identifying how well resources are connected, by measuring the density of the resource neighbourhood.
A network has a high clustering cohesion when a node has a large number of neighbouring nodes, all of which are connected to each other. This means that links may end up being meaningless  [18]. We have described an approach to approximating the clustering coefficient measure in  [8]. Metric I. 7 (External Link Data Providers). The external links to data providers metric assesses the extent of connectivity between the dataset under assessment and external sources. In  [8] , we have implemented a approximate version of this metric.
Metric I. 8 (Dereferenceable Backlinks ). This metric measures the extent to which a resource includes all triples from the dataset that have the resource's URI as the object. This allows browsers and crawlers to traverse links in either direction  [23]. The metric is defined as the ratio between the number of triples that are "back-links" a. k. a. "in-links" (i. e. that have the resource's URI as their object) and the total number of triples in the RDF graph served as a description of each resource. Relevance: Interlinking is relevant for open domain answering systems, since it is concerned with data integration. Consider the following scenario. Let us assume a user (who is a football fan) searching for a list of all German soccer players. Here, the instances of the terms German and Soccer should be interlinked with the instances Deutschland and Football respectively in order to return the correct and accurate list of all the German football players.
Data diversity Dimension IRIs as the identifiers of entities in datasets are not necessarily human-friendly, or, if they are, they are often words in the dataset author's favourite language. However, humancomprehensible information in arbitrary formats and languages can be attached to entities via appropriate annotation properties. Definition 3. Data diversity refers to the availability of data in formats that are accessible to a wide range of human end users (numbers, images, audio, videos, etc. ), and in different international languages. We use the following metrics to measure the data diversity of a dataset: Metric D. 9 (Human Readable Labelling). This metric measures the percentage of entities that have a humanreadable label (rdfs:label). Although different datasets might different label properties (cf.
[10] ), we assume, for now, that the most commonly used standard label property (i. e. rdfs:label) is employed. Metric D. 10 (Multiple Language Usage). This metric checks whether literal objects are available in different languages. We check all literals having a language tag and ignore those without a language tag. The value of this metric is the average number of different languages used per resource, throughout the dataset. Relevance: Data diversity is an important dimension for question answering systems because of the diverse needs of their human end users -who might, e. g. , speak different languages. Moreover, rich natural language labels aid natural language approaches that disambiguate input words to uncover the semantics intended by the user and finally map them to the right entities in the underlying datasets.
Generally, any question answering system has to deal with lexical and structural differences between the words used in questions and resources in the available datasets and terms in the ontologies used as vocabularies by these datasets. Labels of classes and properties play a great role in the process of matching question words with the dataset's terminology. We also refer to data diversity in terms of variety of the data formats available to the question answering system, other than text, for instance images, audio (e. g. , for systems that support speech recognition), or videos. In addition, the final answer may need to be presented to the end users in natural language or as a visual graph -once more a situation in which human-readable labels play an important role.
Consistency Dimension A knowledge base is consistent if it contains no contradicting information  [22]. One can define the semantics of RDF data by defining classes and properties using RDFS and OWL. Reasoning can then use the semantics of the defined classes and properties to interpret the data and infer new knowledge -if the axioms that define the classes and properties and the triples in the datasets are consistent, or, in more formal terms, if the overall knowledge base has a model, i. e. , an interpretation that satisfies all axioms. We adopt the definition of consistency given by  [44]  as follows: Definition 4. Consistency means that a knowledge base is free of (logical/formal) contradictions with respect to particular knowledge representation and inference mechanisms. Consistency metrics includes: entities as a member of disjoint classes, misplaced classes/properties, misuse of owl:DatatypeProperty and owl:ObjectProperty, ontology hijacking, and incorrect domain and range usage.
We consider the following two metrics relevant for question answering systems: Metric C. 11 (Ontology Hijacking). Ontology hijacking refers to a dataset redefining classes or properties reused from an external vocabulary (which uses a different URI namespace). Metric C. 12 (Misused OWL Datatype/Object Property). This metric is based on the metric defined by  [23]. It detects properties that are defined as a owl:DatatypeProperty but is used like an object property, i. e. with resources in the object position, and vice versa. The metric computes a ratio of misused properties. Undefined properties are ignored by this metric. Relevance: Redefinition of external classes or properties by third parties could lead to an incorrect interpretation of the data by ontology-based question answering systems.
However, current works do not address how to compute initial ranking or trust values; they are provided by users or just taken for granted. Hence, without an initial preprocessing to assign initial values, trust metrics are simply inapplicable in the current environment of Linked Data. Provenance, on the other hand, is used to annotate data with metadata about its process of creation (when was it created, who did it, using what previous data, etc. ). There are currently several proposals for annotating metadata: using named graphs  [5, 13] , using a framework to deal with multiple annotations on each triple  [45] , making use of the PROV-O ontology  [26] , attaching a provenance graph to the dataset  [19] , or devising a model to represent statements about statements  [30, 11]. Implementing a provenance metric that takes into account such an heterogeneous ecosystem of provenance solutions is not straightforward.
Here we present three simple metrics to evaluate provenance. Metric P. 13 (Basic Provenance). This metric, defined in  [7] , checks whether a dataset has at least a triple with the dc:creator or the dc:publisher property, to describe a dataset. Metric P. 14 (Extended Provenance). This metric, defined in  [7] , checks if each entity in a dataset has the required provenance information such that an agent can identify the entity's origin.
Metric P. 15 (Provenance Richness). The Provenance Richness metric provides a measure of the information that a dataset has about itself, using the proportion of metadata statements in the dataset. Relevance:In the context of a QA system that extracts the information from Linked Data, trust and provenance are key factors to providing answers a user can have confidence in. In an environment where everyone can publish data, any given query can find disparate or even contradicting answers in different datasets. While trust provides a (general or personalised) measure of the trustworthiness a user can have in the data, provenance allows to relate data with their author and process of creation.
EVALUATION OF SELECTED MET-RICS 4. 1 Datasets: DBpedia and Wikidata For our pilot study we chose the most widely used crossdomain linked open datasets DBpedia and Wikidata. 8  DBpedia is a community endeavour to extract structured information from Wikipedia into RDF for further consumption. For working with the dataset, one can either load it into a local triple store, or access it via a SPARQL endpoint 9. Wikidata has lately emerged as one of the most famous user-curated source for structured information inside Wikipedia. Wikidata serves as the pivotal data management platform for Wikipedia and the majority of its sister projects  [42]. Wikidata can be queried/accessed using a public endpoint 10 deployed by third parties. However, the uptime of this endpoint is debatable for reasons unknown to the authors, one of them possibly being heavy user (request) traffic.
There are the following key differences between these two datasets: • DBpedia makes use of language dependent (i. e. human-readable) Wikipedia article identifiers for creating IRIs for concepts in every language (in Wikipedia language editions). DBpedia uses RDF and Named Graphs as its native data model. 8 Freebase used to be another popular cross-domain dataset but support for it has expired, which is why we did not consider it; cf. https://www. freebase. com/. • Wikidata, on the other hand, uses languageindependent numeric identifiers and its own data model, which provides a furnished platform for representing provenance information. Every subject on which Wikidata has structured data is called an entity, and every entity has a page. There are two types of entities: terms, which represent individuals and classes, and properties, which represent RDF properties.