dataesr/scientific-paragraphs-categorization huggingface dataset:
However, none of these funded studies were found in our search, probably because most of them were not interventional studies or interventional without IMPs which registration is desirable but not mandatory. Nevertheless, screening the information provided by FCT we found at least one interventional study with IMP that was not properly classified, consequently not registered in a database by the investigators  [8]. The level of participation of Czech Republic and Portugal in international grants was similar (n = 38 and n = 30, respectively), most of these involving trials with IMPs in both cases (Fig. 3a, b ). The organizations receiving national funds, thus acting as sponsor, in higher number of trials in the Czech Republic were hospitals (32%: n = 35/109) and universities (22%: n = 24/109).
National Policies for Clinical Research Both countries present national public policies implemented (or not) for clinical research, in particularly IICTs, which are summarized in Table  2. The data were collected from national legal decisions, analysis of the call's eligibility criteria for funding national infrastructures and projects as well as higher education institutions in each country. There have been legal decisions to establish clinical research as a priority that included the membership of ECRIN-ERIC and the definition of the clinical investigator status in both countries  [10] [11] [12] [13] [14] [15]. Since 2015, ten trials were initiated by ECRIN and conducted in the Czech Republic in this time period: POEM_H2020, FAIRPARK II_H2020, NISCI_H2020, SECURE_H2020, EuroHYP_7FP, PrOOf_ H2020, VISION_DMD_H2020, DISCHARGE_FP7, PopART_H2020, TENSION_H2020.
Today, also general-purpose mixed-integer solvers such as Gurobi (Achterberg 2019) and CPLEX  (Klotz 2017 ) can solve problems including these bilinear nonconvexities. In bilevel optimization, very often the assumption is made that the linking variables, i. e. , those upper-level variables that also appear in the lower-level constraints, are bounded integers. In this case, the bilinear terms λ Cx can be linearized if upper bounds on λ are available. Note, however, that finding these upper bounds is the same task as finding big-M values for the KKT reformulation. Nevertheless, if such a big-M is at hand, in  Zare et al. (2019)  it is shown that in case of large lower-level problems (measured in terms of the number of constraints), the strong-duality based reformulation (10) outperforms the KKT-based approach.
The authors conducted a computational study on a set of benchmark instances shown in Table  1  (except CLIQUE and IMKP) and optimal solutions have been reported for 822 out of 874 instances. The code of  Fischetti et al. (2017a)  is publicly available  (Fischetti et al. 2017b) , and represents the current state-of-the-art exact method for general bilevel MILPs. The code is integrated within the commercial solver CPLEX. An alternative open-source implementation that includes features of  Fischetti et al. (2017a) , but also many additional ones, has been developed by  Tahernejad et al. (2020)  and is available online  (Ralphs 2018). Unsurprisingly, specialized approaches for solving particular interdiction problems, like those of  Fischetti et al. (2019)  and  Furini et al. (2020b) , are outperforming the generic approaches by  Fischetti et al.
Acknowledgments This research has been performed as part of the  Energie Campus Nürnberg  and is supported by funding of the  Bavarian State Government. The authors thank the  DFG  for their support within projects  A05  and  B08 in CRC TRR 154. Martine Labbé  has been partially supported by the  Fonds de la Recherche Scientifique -FNRS  under Grant(s) no PDR  T0098. 18.
The quality of a Knowledge Graph (also known as Linked Data) is an important aspect to indicate its fitness for use in an application. Several quality dimensions are identified, such as accuracy, completeness, timeliness, provenance, and accessibility, which are used to assess the quality. While many prior studies offer a landscape view of data quality dimensions, here we focus on presenting a systematic literature review for assessing the completeness of Knowledge Graph. We gather existing approaches from the literature and analyze them qualitatively and quantitatively. In particular, we unify and formalize commonly used terminologies across 56 articles related to the completeness dimension of data quality and provide a comprehensive list of methodologies and metrics used to evaluate the different types of completeness.
I. INTRODUCTION The development of Semantic Web technologies like the Resource Description Framework (RDF) 1 has led to unprecedented volumes of data published on the internet as Linked Open Data (LOD) 2   [1]. The collection and publication of such vast amounts of data into a Knowledge Base (KB) is certainly a progression in the right direction towards the Web of Data. However, the evolution of KBs exposed as Linked Data such as in the LOD Cloud 3 is generally unrestrained  [2] , which leads to a variety of quality issues, at various levels; for example, at the schema or at the instance level. An empirical study carried out by Debattista et al. [2]  shows that datasets published in the LOD cloud have a reasonable overall quality, but significant issues remain concerning some The associate editor coordinating the review of this manuscript and approving it for publication was Wajahat Ali Khan.
1 https://www. w3. org/RDF/ 2 http://linkeddata. org/ 3 http://lod-cloud. net/ quality dimensions, such as data provenance and completeness. Therefore, by studying only one dimension such as completeness, we have the ability to explore completeness quality issues more thoroughly. For instance, we can detect whether the completeness problem is better dealt with during data collection or integration process. The Semantic Web promotes the reuse and sharing of this data, as well as its automatic processing by computer agents. The data represented in this way make sense and allow, in theory, for consensual interpretation by all actors (producers and consumers). Linked Data is sometimes called Knowledge Graph as referenced by Google in 2012. 4 ata quality is also a challenge for traditional information systems, hence, rigorous research on ensuring adequate quality of data in relational databases have been carried out, even before the onset of Knowledge Graph.
This development has led to a positive impact on the data quality organizational processes for relational databases  [3] ,  [4]. Thus, the applicability of this approach in the context of Web of Data provides an avenue to leverage the experience gained from traditional information systems. Since high quality of data ensures its fitness for use  [5]  in a wide range of applications, having the right metrics to assess and improve the quality of Knowledge Graph is of great importance. Several frameworks and approaches have been proposed to evaluate varying dimensions of Linked Data quality; Zaveri et al. [6]  conducted a comprehensive Systematic Literature Review and identified 18 different quality dimensions that can be applied to assess the quality of a Knowledge Graph. In our article, we focus on how to assess the completeness of Linked Data.
A. OVERVIEW OF KNOWLEDGE GRAPH COMPLETENESS Completeness is a data quality measure that refers to the amount of information present in a particular dataset  [6]. For example, the instance Albert Einstein might suffer from a data completeness problem when his birth place is missing in the dataset. Assessing data quality is one of the challenges that data consumers and providers are facing  [11]. It is a multifaceted challenge, for instance, the term quality is commonly described as fitness for use  [5]  which encompasses several dimensions such as accuracy, timeliness, consistency, correctness, completeness, etc. Nevertheless, these dimensions maybe subjective as completeness implies that the amount of data is sufficient for the consumer's needs which can vary substantially. It can be measured as the percentage of data available divided by the data required, where 100% is the best value.
Later, a new type of completeness is introduced for Linked Data called (iv) interlinking completeness, which checks the existence of links between datasets via their linksets  [14]. Thus, we classified the selected articles into one of these types of completeness as illustrated in Section II. As a result of our SLR, we identified three new types of completeness, namely, (v) currency, (vi) metadata and (vii) labelling completeness making it a total of seven types of completeness. It should be noted that assessment requires a reference or gold standard with which to assess against and such a gold standard should ideally operate on Closed World Assumption. However, the gold standards in Linked Data still operate on Open World Assumption  [15] , but, for the purposes of measuring completeness, we assume that the gold standard is complete.
D. RESEARCH QUESTIONS In this SLR, we aim to answer the following general research question: How can we assess the completeness of Knowledge Graph, which includes different types of completeness considering several approaches? We divide this general research question into sub-questions: • what types of completeness currently exist for Knowledge Graphs? • what are the proposed approaches and metrics to identify and measure the completeness of Knowledge Graphs? • what are the data completeness problems being discussed by researchers? • what tools are available to detect completeness of Knowledge Graphs? Search strategies in a systematic review are usually iterative and are run separately by two or more reviewers to avoid bias and to maximize coverage of all related articles. We performed a search on Google Scholar as a search engine and the following Databases: IEEE Xplore, ACM Digital Library, Science Direct and Springer Link.
II. KNOWLEDGE GRAPH COMPLETENESS ANALYSIS Through our methodology, we identified 56 core articles related to Knowledge Graph or Linked Data completeness. We recognize the challenge of trying to explicitly elaborate on all core articles, hence, we categorized and summarized the main ideas in all the articles with different subsections focusing on different ideas. Section II-A discusses simple statistical analysis and trends from the core articles. Section II-B provides a summary of the core literature categorized according to the type of completeness found. The categories are grouped according to broad problems and approaches recognized. Consequently, Table  5  focuses on the core ideas of the metrics for the core articles. The purpose is to provide a full overview of metrics over all the types of completeness succinctly and avoid clogging.
B. QUALITATIVE ANALYSIS In this section, we analyzed the 56 articles qualitatively to extract relevant information regarding Knowledge Graph completeness. After analyzing the selected articles in detail, we identified and extracted 23 ubiquitous metrics which are presented in Table  5  that can be applied to assess the completeness of Knowledge Graph, categorizing them based on the type of completeness covered. As mentioned previously, (i) schema, (ii) property, (iii) population and (iv) interlinking completeness were already identified by  [6]. As part of our SLR we identified three more types, namely, (v) currency, (vi) metadata and (vii) labelling completeness. In the following, we describe each of the seven types by providing a definition and discussing the problems and approaches that they address.
1) SCHEMA COMPLETENESS The schema of a dataset is considered complete, if it contains all the classes and properties needed for a given task. It is also called ontology completeness  [62]. Fürber and Hepp  [62]  defined schema completeness as the degree to which classes and properties are represented in a schema. In a similar sense but under a different name, Mendes et al. [61]  defined intensional completeness which is the existence of all the attributes in a dataset for a given task. For example, the dataset suffers from a schema completeness problem when the property capital is missed from the instance France. Definition 1 (Schema Completeness): Schema completeness is the degree to which the classes and properties of an ontology are represented in a LD dataset.
The first step aims to find the properties patterns that are most shared by the subset of instances extracted from the triple store related to the same category Maximal Frequent Patterns (MFP). This set, that is called ''transaction'', will be then used to calculate a completeness value regarding these patterns. The second step carries out for each transaction a comparison between its corresponding properties and each pattern of the MFP set regarding the presence or the absence of the pattern. An average is, therefore, calculated to obtain the completeness of each transaction t and, hence, the completeness of the whole dataset.
b: PROBLEMS Several articles address the challenge of development of new tools and frameworks to assess and improve completeness and other data quality dimensions  [55] ,  [63] ,  [66]. The authors in  [20]  were interested in how to apply first order logic predicates and developed a capacity function (i. e. , a fuzzy measure) to express completeness. Reference  [59]  investigated how to employ the similarity between entities in a dataset to determine completeness. In  [31] , the authors built transaction vectors constituted of sequence of properties that deduced from instances to use them as an input to generate frequent patterns in order to compute the completeness.
c: APPROACHES AND METRICS There are 13 articles that propose some approaches of metrics about schema completeness  [20] ,  [22] ,  [24] ,  [27] ,  [31] ,  [53] ,  [55] ,  [59] ,  [61] -  [63] ,  [66] ,  [67]. These existing approaches defined a set of metrics to assess schema completeness such as applying fusion methods or defining quality indicators, or assessing completeness based on extracting a set of frequent/required predicates. Several metrics measure completeness as the ratio of the number of classes/properties presented in a dataset to the total number of classes/properties  [22] ,  [53] ,  [61] ,  [63] ,  [66] ,  [67]. Other metrics take into account only the mandatory properties to assess the completeness  [24] ,  [31] ,  [72]. [59]  measured ratio of similar instances/subjects missing same properties.
2) PROPERTY COMPLETENESS Property completeness as defined by  [6]  is the measure of the missing values for a specific property. This is similar to the definition of  [13]  which referred to it as column completeness. Property completeness is measured by determining if a specific property has missing values. For example, the dataset suffers from a property completeness problem when the property capital of the instance France does not have a value, namely, Paris. Definition 2 (Property Completeness): Property completeness is the degree to which values for a specific property are available for a given task.
a: OVERVIEW One of the earliest and influential research which addresses property completeness is the work of Mendes et al. [61]. They developed a versatile tool for quality assessment (Sieve) as a part of a Linked Data Integration Framework which deals with data access, schema mapping and identity resolution. This work provided one of the foremost easily extensible framework for Linked Data integration and data fusion. While the tool covers various data quality dimensions for assessment, it adequately provides one of the more widely used implementation for assessing property completeness. Sieve provides a customizable module based on a conceptual model of assessment metrics and scoring functions. The assessment metric is a procedure for measuring the score with regards to a data quality dimension such as property completeness using selected schema elements as a data quality indicator.
c: APPROACHES AND METRICS There are 22 articles that addressed property completeness either by proposing an approach or metric to measure completeness  [17] ,  [18] ,  [22] ,  [23] ,  [25] ,  [29] ,  [34] ,  [42] ,  [44] ,  [46] -  [48] ,  [53] ,  [55] ,  [57] ,  [58] ,  [61] -  [65] ,  [69]. A prominent methodology for assessing property completeness focuses on the development of novel frameworks towards measuring the level of completeness of a knowledge base such as  [22]  where the authors use the Goal Question Metric (GQM) method to define metrics for inherent qualities of a dataset. A set of metrics based on measurement-theory have been proposed  for the inherent quality characteristics of a dataset where property completeness is evaluated the ratio of the sum of the number of presented properties per instance to the total number of instances in the dataset.
3) POPULATION COMPLETENESS A dataset is complete if it contains all of real-world objects for a given task, which is also called the completeness at data (instance) level  [62]. Population completeness is also termed extensional completeness  [61]. For example, the dataset suffers from a population completeness problem if it does not have all the French cities. Definition 3 (Population Completeness): Population completeness is the degree to which all real-world objects of a particular type are represented in a LD dataset. In this paper, the authors compared Brazilian municipalities in English and Portuguese DBpedia datasets according to all 5565 Brazilian municipalities.
b: PROBLEMS The popular challenge in this type is to check a KB to see whether it contains all entities of a given type. Reference  [25]  was interested in how to maintain the quality of KB that evolves and changes frequently. The proposed approach provides an overview of the change of a given KB and a fine-grained analysis. Consequently, the authors in  [68]  used completeness metrics to assess the quality of newly published Linked Data for cultural heritage. Reference  [33]  was focused on disambiguation problem, the authors provided a method to scalably resolve entity co-reference in structured datasets. Moreover, efforts have been made to include completeness information in KBs. This is achieved by adding true facts such as Adele has one brother. This information is essential to evaluate query completeness and soundness  [37] ,  [42] ,  [45].
c: APPROACHES AND METRICS A set of 17 articles have been proposed to enhance population completeness  [25] ,  [26] ,  [33] ,  [37] ,  [39] ,  [42] ,  [43] ,  [45] ,  [47] ,  [53] ,  [56] ,  [57] ,  [60] -  [62] ,  [64] ,  [68]. Various metrics that check KBs to see whether they contain all entities of a given type in comparison to real-world data  [25]  or deal with query completeness via hybrid computation  [43] , include completeness information as part of the KB that can be used for validation  [42]. On the other hand,  [26]  used a Bayesian Network to model the dependencies among resources that belong to a set of linked datasets and represent the joint probability distributions of relationships among resources. The probability of an individual resource is considered the likelihood of redundancy or indicator of completeness regarding the resource.
4) INTERLINKING COMPLETENESS This type particularly focuses on data integration which is a core tenet of Knowledge Graph. It refers to the instances that are interlinked in the dataset for disambiguation with regards to a reference dataset  [30]. For example, the instance France linked from French national dataset to another instance French Republic in the United Nations dataset. Definition 4 (Interlinking Completeness): Interlinking completeness is the degree to which instances are interlinked in a LD dataset with respect to some reference dataset(s).
b: PROBLEMS In the work of  [17] , the authors were interested in how to develop a standard data quality model for quality specification and assessment. The authors identified an indicator that are in a specific context of use, how to measure degree of instances associated with all expected and related entities. In the first step, the authors defined base measures for quality evaluation, then combining different base measures to get derived measures and metrics that are obtained by integrating base and/or derived measures. These measures and metrics are used to assess data quality covering various quality dimensions. Albertoni et al. [19]  explored how to assess the value of interlinks of datasets in terms of information gain via what they refer to as linkset importing. References  [33] ,  [70]  proposed methods to resolve entity co-reference and completing links in KBs.
c: APPROACHES AND METRICS We identified 13 articles that focus on the interlinking completeness in LD  [14] ,  [17] -  [19] ,  [21] ,  [26] ,  [28] ,  [30] ,  [49] ,  [52] ,  [61] ,  [68] ,  [70]. Several approaches have been proposed to assess interlinking completeness. Reference  [26]  analyzed the quality of data and links in LOD cloud using Bayesian Networks. Additionally,  [19]  estimated the completeness of a dataset by complementing SKOS thesauri with their skos:exactMatch related information. In  [49] , the authors gathered the essential predicates of data sources using their covering and discriminative abilities. Then, they selected the most suitable alignments based on their confidences and finally, comparing the instances based on the selected alignments.
Definition 5 (Currency Completeness): Currency completeness is the degree to which elements of a knowledge base are available as it is updated over time. a: OVERVIEW Currency completeness mainly deals with checking for outdated data which are usually inappropriate for most tasks. Rula et al. [35]  were the first to refer to currency completeness as a concept to evaluate the completeness of currency measurement. They developed the first dataset independent framework for assessing the currency of Linked Open Data (LOD) graphs. In order to measure the currency of data, the temporal information about the creation and modification of LOD resources and documents, called versioning metadata is of uttermost importance. Unfortunately, it is difficult to collect versioning metadata in LOD, especially because there are no widely adopted models to represent such metadata, and their characterization.
| resources set with currency values > 0 | | all resources set in dataset | (4) b: PROBLEMS The issue of how to develop frameworks and metrics for assessing the currency of RDF data is the focus of research such as  [35] ,  [73]  and currency completeness is the byproduct of evaluating the proposed frameworks for assessing currency. Reference  [36]  also dealt with currency completeness while investigating an approach to deal with the mutual propagation of the changes between a replica and its origin dataset termed as co-evolution.
6) METADATA COMPLETENESS Descriptive metadata about datasets enables dataset discovery, and as such  [41]  provided a comprehensive overview on metadata termed as dataset profiling where they also assess metadata completeness. Accordingly, metadata is considered complete if it contains all the fields with values required to properly describe a dataset e. g. a dataset with technical identifiers such as title or description without any metadata context is incomplete and reduce the quality of the dataset. Metadata/description of a dataset is expected to be Findable, Accessible, Interoperable and Reusable (FAIR)  [74]. For instance, indicating the description of the dataset about France that it captures intrinsic properties of the country or all the editors that modified the dataset, thus, making sure the metadata is available and complete.