dataesr/scientific-paragraphs-categorization huggingface dataset:
Every item page contains a label, a short description, a list of aliases, a list of statements and a list of site links. Wikidata statements are not just triples but can have additional quantifiers and references which support the claim. Each statement (typed as rdfs:Statement) has its own UUID identifier, which is used to connect to items, qualifiers and references. We prepared four slices of DBpedia and Wikidata, respectively, to cover the categories Restaurants, Politicians, Films and Soccer players. The choice of categories for slices was based on internal voting of the authors from a list of popular topics. We prepared the Wikidata slices by querying our local SPARQL endpoint which uses the RDF dumps 11 of November 30, 2015. For DBpedia we instead queried the publicly available SPARQL endpoint maintained by the DBpedia community.
The detailed statistics of the prepared slices are presented in Table  2  and Table  3  respectively. The slices prepared for our study, from both the datasets, are publicly available for research purposes and can be downloaded from https: //goo. gl/Kn6Fom (DBpedia slices), https://goo. gl/5aTkLp (Wikidata slices). The justification for evaluating quality metrics over slices of a dataset is: on the one hand, cross-domain datasets contain data on different domains, each of these datasets vary in quality of information (our obj 3). We would therefore like to compare the quality of datasets by looking at domain-specific subsets than the overall datasets. On the  other hand, knowing the "fitness to use", as in where to find high quality data on a specific topic, helps federated question answering systems to choose the right source at query time.
Experimental setup For slice preparation we initially relied on querying the publicly available SPARQL endpoints of DBpedia and Wikidata. However, in the case of Wikidata, due to a lack of reliability of the endpoint (in terms of uptime and constant request timeout errors), we preferred deploying our custom endpoint locally and run a local slicing tool  12. For preparing the slices and assessing their quality, we used a machine with the following specification: • Triple store: Virtuoso OpenSource 7. 2. 1 • OS: Ubuntu 14. 04 LTS • Storage: 1 TB disk space • RAM: 32 GB DDR3 • CPU: 12 core Intel Xeon E5-2690 v2 (@3. 0 GHz) A step by step guide for the endpoint setup and preparing slices is publicly available at https://goo. gl/y5xryk. Furthermore, we are planning to provide readers access to our virtual machine, as an interactive playground to run and evaluate datasets, on request.
The Luzzu Quality Assessment Framework We carried out our evaluation using the Luzzu  [7]  quality assessment framework developed previously by some of the authors. Luzzu provides an integrated platform that:  (1)  assesses Linked Data quality using a library of generic and user-provided domain specific quality metrics in a scalable manner; (2) provides queryable quality metadata on the assessed datasets; (3) assembles detailed quality reports on assessed datasets. A number of metrics defined in  [44]  were implemented for the Luzzu framework 13.
Observations from slices statistics We first discuss observations from the slice statistics; please refer to figures 1a, 1c, and 1c respectively. • Restaurants: The sizes of the slices obtained from both datasets are very similar. The DBpedia slice has 1, 585 more triples than the Wikidata slice. However, it is interesting to see that the number of instances of restaurants in Wikidata is almost double that of DBpedia. • Politicians: The politicians slice sizes of both the datasets shows high variation. Wikidata outperforms the DBpedia slices in the total number of triples and number of unique instances per slice by, 8, 684, 545 and 262, 532 respectively. This advocates for the information richness of Wikidata "politicians" category is far better than that of DBpedia. Thus, from the slice statistics, it will be advisable to fire a query related to politics or politicians on the Wikidata dataset rather than DBpedia.
• Films: The case of the films slices of both datasets is similar in terms of the number of triples and the number of unique instances per slice. Wikidata provides more information, with almost 2. 5 million more triples than DBpedia and 70, 949 more unique instances as compared to DBpedia. • Soccer players: However, for soccer players slices, DBpedia is richer in information than Wikidata, with almost 2 million triples and 82, 278 more unique instances per slice respectively. From the above observations, we can primarily infer that, the information richness of DBpedia is comparably reasonable to Wikidata for category Restaurants, very poor for categories Politicians and Films, and far better in case of the Soccer players category. Thus, for the user scenario we presented in Section 1. 1, it will be debatable to consider one dataset over the other, as DBpedia has 1, 585 more triples than Wikidata, however, the latter has 591 more unique instances as compared to DBpedia in the restuarant category.
Observations from computed scores of data quality assessment metrics Next, we discuss observations from assessing the quality of the slices, by quality dimension: • Availability : This dimension comprises of six metrics. The RDF Availability metrics gives a binary value: 0 meaning that an RDF dump is not accessible or 1 meaning that an RDF dump is accessible. This metric checks the usage of the void:dataDump property in the dataset's metadata and whether the linked URL is dereferenceable. For each slice we found 0 because the slices did not include VOID metadata. Similarly, Endpoint Availability checks the existence of the value of the void:sparqlEndpoint property, and its accessibility by firing an ASK query. For DBpedia, we manually checked this metric on the publicly available endpoint and found a VOID description  14  , but it does not include the void:dataDump property.
For Wikidata, we could not find such information in the public endpoint. The Dereferenceability A. 1 and Dereferenceability Forward links A. 2 metrics report almost constant scores for DBpedia slices, between [0. 012, 0. 013] for A. 1 and purely constant at 0. 027 for A. 2. Whereas for Wikidata slices, A. 1 scores are reported to increase with the increase in the size of data slice; refer to figure  2. This implies that Wikidata slices are better in dereferentiability as compared to DBpedia slices. Also, there is no proportional relation between the size and the scores for either of the datasets. Hence, on a one-to-one comparison basis, the average scores of Wikidata slices for availability are comparably better than that of the DBpedia slices. However, Wikidata only minorly outperforms DBpedia and both the datasets return poor results in this dimension.
One of the primary reason for such a low performance could be pin-pointed to the lack of regular maintainance (i. e. updation and cleaning) of the dataset's internal uri's/links and/or the lack of proper description in the subjects resource uri. • Interlinking : This dimension comprises of three metrics: Interlink Detection metric I. 6, Linking to External Providers metric I. 7 and Dereferenceable Back Links metric I. 8 respectively. Comparing the scores of these metrics, for I. 8 the assessment results show considerable improvement of Wikidata slices as compared to DBpedia (e. g. the politicians slice of DBpedia reports only 1. 4% whilst the Wikidata politicians slice scores 9. 8%). This implies that Wikidata is more traversible than compared to DBpedia, since resources in DBpedia report more broken/missing links which hinder its traversivity (via browser or crawlers).
We were not be able to compute the scores for I. 6 for Wikipedia and DBpedia. The reason behind this can be considered to be the lack of ontology information about Wikidata during the quality assessment process. Moreover, scores for the I. 7 metric was not computed, further invetigation is needed for the same. Potential reason could be some configuration or parsing error in computation of the metric. The highest score for I. 7 metric for Wikidata slices is reported for the politicians slice, i. e. 11, since the politican slice has the highest number of referred external links as compared to other slices. • Consistency : The scores of the metrics in this dimension, i. e. Ontology Hijacking C. 11 and Misused OWL Datatype Or Object Properties C. 12 are observed to be identical for all the slices.
The reason is the same as mentioned above for interlinking: i. e. the lack of ontological definitions in the slices. Since the slices are prepared from the whole DBpedia and Wikidata dataset using a specific SPARQL query patterns (e. g. , {?s a dbr:Restaurant. ?s ?p ?o}), they do not have any schema information. Therefore these scores can be ignored for the quality comparison of both datasets. However, for completeness we propose to load the schema from the whole dataset and compute these scores on DBpedia and Wikidata as a whole. • Data diversity : This dimension comprises of two metrics: Human Readable Labelling D. 9 and Multiple Language Usage D. 10. The one-to-one comparison of evaluation results for Human Readable Labelling on both datasets suggests that DBpedia remarkably outperforms Wikidata.
The reason behind this is, for these metrics Luzzu checks the existence of rdfs:label for every instances types. This means, it also considers having no label for rdf:Statements, their qualifiers and reference objects used for reification in Wikidata is a minus, which does not make sense to give labelling for reifications. This is one of the key differences between the data models of DBpedia and Wikidata, as mentioned in Section 4. However, Wikidata slices yield better scores for the Multiple Language Usage metric, as compared to DBpedia slices, on average. One of the reason better scores of Wikidata slices could be the language support (multilingual resource support) or popularity (in terms of relying on a particular dataset for updates). It can be observed from tables 4 and 5 that the average number of languages in Wikidata is greater or equal than that in DBpedia for all of the four slices studied here.
However, the scores for D. 10 of all the slices across the datasets the same. This could again be avoided for quality assessment judgement. For this dimension, we do not assess the diversity of data value types, such as images, audio, or videos. This is because we do not have implementation support in Luzzu at the moment for the same, but we are planning to implement them in the near future. • Trust and Provenance: We include three metrics for the Provenance dimension. Basic Provenance P. 13 and Extended Provenance P. 14 show that none of the datasets makes use of standard vocabularies for provenance. It is worth to be noted that, considering that the slicing method could provide a biased account of those metrics, we performed an evaluation against the whole DBpedia and Wikidata datasets with the same results.
The metric Provenance Richness P. 15 reports a negative result for DBpedia, while showing comparable figures for Wikidata. This result is expected because the Wikidata data model aims to provide statements about statements. Among the Wikidata slices, it can be observed that politicians have a greater provenance richness, while films and soccer players have lower values. It could be argued that it is more common to provide references for political activities, but the actual reason should be further studied.
RELATED WORK A majority of the work in this area focuses on the broader domain of LOD quality assessment, with very little or no specific coverage of the question answering perspective. The  organisers of community wide question answering challenges such as NTCIR Question Answering Challenge (QAC)  [14] , TREC  [41] , CLEF  [21] , BioASQ challenge  [38] , QALD series  [27] , etc. , do discuss the extreme importance of having gold standard (manifold cross-validated) data at disposal, and also discuss about the challenges in curating such good quality datasets for the question answering domain. However, there is, so far, not much dedicated work for developing an autonomous framework for establishing and enforcing quality assurance from the questions answering perspective. Preparing datasets of high quality (referred to as "fitness for use") requires administration from field experts.
For instance, the preparation of CLEF eHealth track datasets demands a huge man-power effort from medical domain experts, such as clinicians, physicians, lab attendants (wet workers), etc. The judgements (relevance judgements) for a question set have to be cross verified by more than one subject expert with utmost care. This is very expensive in terms of time and finance required to curate a high quality dataset. Crowdsourcing has also become a viable option in the  present decade especially from the quality assessment perspective; consider dedicated research efforts such as DBpe-diaDQCrowd  [1]  or TripleCheckMate  [24]. Also, general quality assessment tools and frameworks such as our own Luzzu  [7] , and others including one by Flemming  [12] , as well as Sieve  [28] , RDFUnit  [25] , LinkQA  [18] , and LiQuate  [33]  advocate the criticality of data quality assessment.
However, for none of the quality assessment frameworks and tools mentioned earlier the specific requirements for benchmarking datasets from a question answering perspective have been discussed so far. Also, the quality requirements or guidelines to be followed while preparing a gold standard for question answering systems (keeping in mind the standard quality assessment tools and frameworks) has, so far, not materialised formally. Dimension Metric Restaurants Politicians Films Soccer Players Availability EstimatedDereferenceabilityMetric 0.
CONCLUSION AND FUTURE WORK With question answering becoming increasingly reliant on open domain datasets, the quality of such datasets crucially influences the quality of answers to questions. So far, question answering systems have not yet been evaluated by the quality of the data they use, and while quality dimensions and metrics for linked data have been well documented, it has not been clear which of these are relevant for question answering scenarios. We have addressed these shortcomings of state-of-the-art research on question answering and data quality with a comprehensive review of linked data quality dimensions and metrics relevant to open domain question answering, and applied these metrics to subsets of the popular DBpedia and Wikidata datasets. The results of our experiments suggest that for most of these domains, the quality of Wikidata with regard to metrics relevant for open domain question answering is higher than that of DBpedia.
A major limitation of our work so far is that we have focused on those quality metrics that had been implemented for our Luzzu linked data quality assessment framework already, whereas further metrics would help to better cover the quality dimensions that we have identified as relevant to open domain question answering. Furthermore, we have so far only covered four subsets of DBpedia and Wikidata each, whereas these linked open datasets cover a lot more domains in which users would like their questions to be answered, and there exist further datasets that are widely used for open domain question answering. In our near future work we plan to address these shortcomings, in particular i) to extend and automate the process of rigorously assessing data quality, carried out by Luzzu. We plan to implement the dimensions that our current evaluation has not yet covered, such as Timeliness, Licensing, and Semantic accuracy, and also those metrics whose values could not be calculated due to technical problems of our implementation, such as the estimated interlink detection metric for the slices of both DBpedia and Wikidata, and the approximate implementation of the external links to data providers metric for DBpedia slices.
Obj 3 -How well do popular cross-domain datasets serve the needs of question answering applications in certain specific domains? Obj 4 -How can data quality from the perspective of open domain question answering be practically assessed using available tools and techniques? The remainder of this paper is structured as follows: In section 2, we present the state of the art for quality assessment of linked datasets and question answering in general. Obj 1 -What dimensions of data quality are relevant to open domain question answering systems? Obj 2 -By what data quality metrics can the quality of data for open domain question answering systems be assessed precisely?.
Instrumental data and model outputs Coral growth data were compared to the extent of the ozone depleted area, the wind stress, the zonal wind, the SST, and the PDO (Pacific Decadal Oscillation). The extent of the ozone depleted area over Antarctica (in million km 2 ) is defined as the maximum daily area in October of each year between 1979 and 2008. Data were obtained from NASA Ozone Watch at the link ozonewatch. gsfc. nasa. gov. Ozone depletion over Antarctica was discovered in the mid-1980s  (Farman et al. , 1985)  and was attributed to the positive trend in atmospheric halogenated hydrocarbons released by human activities  (Albritton and Kuijpers, 1999; Solomon, 1999). Gradual ozone depletion recovery occurred just from the beginning of the XXI century (almost 10 years after the implementation of the Montreal Protocol in 1989 that controls emissions of ozone depleting substances).
Results and discussions The three coral cores collected in Abrolhos exhibit a decline in annual growth rate towards the present (Fig. 2a, b, c ). Corals used in this study were found in a healthy condition, with no sign of bleaching at the time of sampling. To blend data from the three coral cores, we used a simple Gaussian z-score transformation (Fig. 2d ). Growth anomaly from individual coral cores exhibits a transition from positive to negative approximately after the mid-1960s and the 1970s, a trend highlighted by the z-score compilation. Figure  2d  indicates that the decline in coral growth follows the SST anomaly, Fig. 2e , inversely, for the Abrolhos site. A temporal difference exists at the time when patterns change from positive to negative phases and vice versa; nevertheless it is within the uncertainties of the modeled SST and z score of growth rate.
Also, slopes of linear trends of SST and coral growth also differ, being steeper for SST. Other works have suggested that mean annual SSTs throughout the tropics and subtropics have increased between 0. 4 and 1 • C in the past 4 decades  (Kleypas et al. , 2008). For Abrolhos, the mean annual SST changed from ∼ 24. 8 to ∼ 25. 8 • C between 1948 and 2006 (NCEP-NCAR reanalysis). Evidence of sharp decline in coral growth rate since the late 1970s was also reported in comparable latitudes away from South America, like that reported for genus Porites from 44 reefs in Australia  (Lough et al. , 2002). In a subsequent work, these authors have expanded their database to include five other reef sites  (Lough, 2008). Their average time serie depicts a sharp drop in the calcification rate (g cm -2 yr -1 ) starting in the late 1970s.
tween the National Centers for Environmental Prediction and the National Center for Atmospheric Research (NCAR). It uses a state-of-the-art analysis and forecast system to perform data assimilation using data from 1948 to the present. Its latest products have time coverage of four times daily while data from 1948 to 1957 are presented in the regular (non-Gaussian) gridded data format. SST and sea level were provided by NCEP-NCAR reanalysis and the Simple Ocean Data Assimilation (SODA) (Carton and Giese, 2008) product. These parameters spanned the full existing database since 1948. PDO data are available at http://www. jisao. washington. edu. PDO is a robust, recurring pattern of ocean-atmosphere climate variability centered over the midlatitude Pacific basin, which influences a significant part of the globe, especially South America.
Acknowledgements. We thank  IBAMA  for coral sampling licenses (nos. 283/2006  and  040/2007CGREP );  Fundação de Amparo à Pesquisa do Estado de São Paulo-FAPESP ;  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior-CAPES ,  Conselho Nacional de Pesquisa-CNPq  (Grant no. 300223/93-5 ) and  CNPq-MCTI/INCT-Criosfera  (Grant no. 573720/2008 ). The  Brazil-France scientific programs LMI PALEOTRACES, PRIMO (CNPq)  and the  IRD-CNPq (CLIMPAST). NCEP Reanalysis data provided by the  NOAA/OAR/ESRL PSD, Boulder, Colorado, USA. U / Th dating was supported by grants from  Taiwan ROC MOST  ( 103-2119-M-002-022  and  104-2119-M-002-003  to C. -C. S. ) and the  National Taiwan University  ( 105R7625  to C. -C. S. ).
Edge distortion between the multi-scale representation subbands of the reference image and the DIBR-synthesized image is measured precisely using mean squared error. In this way, areas around edges that are prone to synthesis artifacts are emphasized in the metric score. Two versions of morphological multiscale metric have been explored: (a) Morphological Pyramid Peak Signal-to-Noise Ratio metric (MP-PSNR) based on morphological pyramid decomposition, and (b) Morphological Wavelet Peak Signal-to-Noise Ratio metric (MW-PSNR) based on morphological wavelet decomposition. The performances of the proposed metrics have been tested using two databases which contain DIBR-synthesized images: the IRCCyN/IVC DIBR image database and MCL-3D stereoscopic image database. Proposed metrics achieve significantly higher correlation with human judgment compared to the state-of-the-art image quality metrics and compared to the tested metric dedicated to synthesisrelated artifacts.
The proposed metrics are computationally efficient given that the morphological operators involve only integer numbers and simple computations like min, max, and sum as well as simple calculation of MSE. MP-PSNR has slightly better performances than MW-PSNR. It has very good agreement with human judgment, Pearson's 0. 894, Spearman 0. 77 when it is tested on the MCL-3D stereoscopic image database. We have demonstrated that PSNR has particularly good agreement with human judgment when it is calculated between images at higher scales of morphological multi-scale representations. Consequently, simplified and in essence reduced versions of multi-scale metrics are proposed, taking into account only detailed images at higher decomposition scales. The reduced version of MP-PSNR has very good agreement with human judgment, Pearson's 0.
The evaluation of still images is important scenario in the case when the user switches the video in pause mode  [2]. For the comparision of DIBR algorithms, virtual views synthesized from the uncompressed data which contain only synthesis artifact need to be evaluated. When encoding either depth data or color sequences before performing the synthesis, compression-related artifacts are combined with synthesis artifact. In this paper, the distortions introduced only by view synthesis algorithms are evaluated using the IRCCyN/IVC DIBR image dataset  [3, 4]  and part of the MCL-3D image dataset  [5, 6]. DIBR algorithms introduce new types of artifacts mostly located around disoccluded regions  [2]. They are not scattered in the entire image such as 2D video compression distortions. As DIBR algorithms involve geometric transformations, most of them introduce mainly geometric distortions affecting edges coherency in the synthesized images.