dataesr/scientific-paragraphs-categorization huggingface dataset:
Background Pharmacogenomics (PGx) studies how individual gene variations cause variability in drug responses  [1]. Well established knowledge in PGx constitutes a basis for implementing personalized medicine, i. e. , a medicine tailored to each patient by considering in particular her/his genomic context. The state of the art of this domain lies both in the biomedical literature and in specialized databases  [2, 3] , but a large part of it is controversial, and not yet applicable to medicine. Indeed, this results from studies difficult to reproduce and that do not fulfill statistical validation standards for two main reasons: the knowledge represented within semantic web ontologies  [6]. For these reasons, LOD offer novel opportunities for the development of successful data integration and knowledge discovery campaign, as required for the discovery of novel pharmacogenes.
Pharmacogenomics data and linked data PharmGKB is a comprehensive database about PGx that includes manually annotated gene-drug relationships  [3]. Recently, annotations of PharmGKB have been completed with a level of evidence going from 1 to 4, distinguishing well validated gene-drug relationships (level= 1 -2) from insufficiently validated ones  (3) (4) , thus pointing at knowledge in need for additional investigations  [14]. PharmGKB does not provide its data in RDF, but parts of PharmGKB have been transformed and published in RDF by contributors of the Bio2RDF project, thus enabling SPARQL queries  [15]. Clinical annotations of PharmGKB are however not freely available. Their usage is granted through a license agreement, preventing the data from being redistributed, thus published as Linked Open Data.
Many other databases provides data that are indirectly relevant to PGx. For instances, DrugBank  [16]  provides drug-target relationships; Clin-Var  [17]  provides gene variant-phenotype relationships; SIDER  [18, 19]  and Medi-Span provides drug-phenotype relationships such as drug adverse events or indications  [20]. Medi-Span is a proprietary database of Wolters Kluwer Health (Indianapolis, IN) aiming at providing drug clinical data to clinicians. DGIdb (The Drug Gene Interaction database) is another interesting initiative that integrates quasi-exhaustively data about gene-drug relationships, considering 15 distinct sources  [21]. DisGenet is a data integration initiative that focuses on gene-disease relationships and provides data in RDF, including parts of ClinVar and OMIM  [22]. Data integration effort clearly oriented to PGx applications are less common, particularly if considering semantic web approaches  [23].
Hoehndorf et al. integrated and made available a set of PGx related data that includes PharmGKB, DrugBank and CTD (the Comparative Toxicogenomics Database), using semantic web technologies  [24]. They used the integrated dataset to identify pathways that may be perturbed in PGx. In this effort of publishing PGx data, Coulet et al. extracted about 40,000 PGx relationships from the biomedical literature and published them in the form of RDF statements  [25].
Data preparation Data selection Initial step is to select a set of data that include relevant data about PGx gene-drug relationships. Figure  1  gives a general overview of the type of data we consider for this study: three types of entities, gene, phenotype and drug; and relationships between them, i. e. , genephenotype, phenotype-drug and gene-drug relationships. Fig. 1  Overview of the type of entities and relationships considered and their origin. Entities are of three distinct types: Gene, Phenotype and Drug. Gene-Phenotype relationships are coming from ClinVar and DisGeNET, Phenotype-Drug relationships from SIDER and Medi-Span, Gene-Drug relationships from DrugBank. In addition, we included gene and drug entities from PharmGKB to enable building the training and test sets. Equivalence mappings are defined between entities of the same type but of different origin.
In addition to entity-entity relationships, we consider some attributes that are specific to entities, such as the ATC class of drug that is a drug attribute. Naming of different parts of the data (e. g. , G-P links, gene attributes) is used later in the step of formatting of the linked data. The detailed schema of the data is provided Fig. 2  We selected data sources manually but oriented our selection to sources providing typed relationships and limited ourselves to two sources per relationship. As a result, we selected ClinVar and DisGeNET for gene-phenotype; SIDER and Medi-Span for phenotype-drug; DrugBank for gene-drug relationships. PharmGKB completes the set of data sources to enable building the training and test sets (see "Training and test sets" subsection).
Data RDFization The second step is about turning selected data in a standardized RDF graph, available at https://pgxlod. loria. fr. We benefit from the fact that Dis-GeNET  [44] , SIDER  [45]  and DrugBank  [46]  are already available online in the form of LOD and reused them. DisGeNET includes data from ClinVar, but because it includes only a part of it, we made our own RDF version of ClinVar following guidelines and scripts of the Bio2RDF project. We completed the Bio2RDF version of PharmGKB locally with gene-drug relationships manually annotated by PharmGKB but not openly distributed  [15]. Similarly, we transformed drug indications and side-effects from Medi-Span in the form of RDF triples and loaded them into our SPARQL server. For the management of RDF data, we rely on Blazegraph, a graph database system that provides support for RDF and SPARQL.
Medi-Span data, as PharmGKB clinical annotations are protected by a license agreement and can not be redistributed. This explains why we are providing a controlled access to our set of PGx linked data. We propose to open this dataset, on demand, with licensees. Figure  2  presents the detailed schema (i. e. , type of entities and relationships) of the linked data we selected and consider for mining. Figure  3  presents an example of data from the PGx linked data, instantiating the schema presented Fig. 2. The SPARQL query returning data presented in Fig. 3  is provided in Additional file 1. Other SPARQL queries, such as the one provided in Additional file 2, may be built by considering the partial data schema presented Fig. 2.
Mapping definition To define mappings, we first relied on standard identifiers such as NCBI Gene ID found in DisGeNET and ClinVar URIs and UMLS CUI found in DisGeNET, ClinVar, SIDER and Medi-Span. We defined regular expressions over URIs to isolate identifiers and when two match, we define a mapping. Figure  3  shows two Fig. 2  Schema of the pharmacogenomic linked data selected for this study. Entities are related to either Genes, Phenotypes (or Diseases) or Drugs. We artificially enriched the data with an additional type of entity: gene-drug pairs. These entities link exactly one gene and one drug and are the nodes of the graph we classify either as associated or not associated from a PGx point of view, to valid candidate pharmacogenes. For mapping purposes, we added to our dataset Gene references from UniProt and Drug references from PubChem.
Because part of Medi-Span and part of PharmGKB data are protected, we restricted the online access to the data Second, when no standard identifier exists, we relied on services provided by biodb. jp to obtain cross-references between identifiers and, accordingly, define mappings  [47]. We implemented a tool named biojp2rdf that transforms the cross-references provided by biodb. jp in RDF  [48]. For drugs, we also relied on the API provided by RxNav to obtain mappings between Medi-Span identifiers and UMLS CUIs  [49]. We loaded all mapping data into our SPARQL server to enable the resolution of identity between entities of the same type.
Learning task, training and test sets Learning task Our learning task is a supervised classification of specific nodes from the data graph. Nodes considered for classification represent candidate pairs of one gene and one drug, which are binary classified as either pharmacogenomically associated or not. Each candidate pair is added to the data graph as a new node linked to both the gene and the drug constituting the pair. Figure  3  provides an example of such a pair and its links to its constituents. Training set To constitute our training set we defined two sets of instances: positives and negatives. Our set of positives is gene-drug pairs annotated as associated according to PharmGKB (version of October 1 st , 2015) and that are annotated with a high level of validation in PharmGKB, i. e.
, level= 1 or 2  [14]. PharmGKB clinical annotations are relating gene variants to drugs, not gene to drugs. We generalized these relationships to manipulate gene-drug relationships. When 2 variants of a same gene are associated with two distinct levels of evidence to the same drug, we conserve only the highest. For instance, the VKORC1 gene has several variants associated to warfarin with level of evidence from 1 to 4. We conserve only that the VKORC1 gene is associated to warfarin with a level of evidence 1. Accordingly, we generated 91 positive instances. To constitute our set of negatives, we randomly generated gene-drug pairs from those listed in PharmGKB, but checked to be absent from DGIdb (the Drug Gene Interaction database), which collects gene-drug relationships from various sources  [21] , including PharmGKB.
This approach enables a better accuracy and generalization of the model, and counterbalances the instability of decision trees, a forest being more stable to slight changes in the data. Data formatting with RF Linked data are in the form of graphs, whereas machine learning algorithms such as RF take a feature matrix as an input. Consequently, our PGx linked data requires to be formatted in the form of such a matrix. Each line of a feature matrix represents an instance and each column represents a feature describing the instances. We propose encoding parts of the RDF graph by observable paths that start from the gene and the drug of a gene-drug pair. These paths start from the gene or the drug and potentially reach each others. To contain the size of the matrix, we simplify paths from genes and drugs in path of length 1, hereafter named G-D link, G-P link or D-P link, depending on the entity that are connected.
RF algorithm performs correctly (F-m= 0. 73) in the frame of our case study, but presents several limitations. First, RF is limited by our usage of a multi-instance representation of data, i. e. , data about one gene-drug pair is represented in the feature matrix by several lines  [52]. Because our dataset contains much more data about positive examples than data about negatives, it results that if we balance the number of positive and negative examples in the training set (respectively 91 and 91), the actual number of lines (i. e. , instances) in the matrix describing positive examples is much larger than for negatives (respectively 108,038 and 3,197). However, our experiments showed that initial selection of negative examples, as well as keeping a certain unbalance, is important. This large unbalance lead the RF to an overfitting, i.
e. , an instance to classify will most probably be similar to one of the numerous descriptions of positive examples and be classified as positive. We overcome this drawback by doubling the number of negative examples in our training set. This unbalancing of examples (respectively 91 positives and 182 negatives) resulted in a reduction of the unbalance in the feature matrix (respectively 108,038 and 57,885). Here we can note that the second set of negative example is described by much more instances than the first one, what let us see that our random pick of negative examples in the large set of gene-drug relationships not referenced by DGIdb impacts to some extent the results of our approach. Another limitation of RF is that it requires a formatting of the graph data and then to select a set of features.
We achieved this selection manually to retain 6 features, and then apply a standard feature selection method (Information Gain) that enabled us to filter one useless feature (disease attribute such as the MeSH class of the disease). Our experiment shows that GK achieves globally better than RF, and that the Mustard library offers many facilities to mine RDF data. In addition, it provides us several insights on the features that are more relevant to consider when mining our PGx linked data. For example, it seems that considering only the close neighborhood (d=4) of instances to classify is sufficient in our case. Also, in the case of a constrained graph, as the one we designed, considering substructure in the neighborhood of instance may not be of primary importance. This may be associated to the fact that we limited the size and connectivity of the data graph, and consequently knowing solely the label of a set of edges and of vertices may enable to reconstruct a path or a subgraph in the neighborhood of an instance.
Our selection of data sources may be discussed, particularly because some of those are not open. Of course adding new sources would be of interest. Indeed, our choice for the linked data framework is motivated by the fact that we want to ease the addition/removal of data sources for enabling the selection of best features out of many sources without considering if they are open or not. In regards with the results from previous works  [40, 41] , we think that sources of triples extracted from the literature would be particularly valuable, such as those extracted in  [25]. Because GK considers data directly in the form of a graph, one could want to mine directly LOD resources, without particular selection. However, the large number of available data in the LOD, including many noninformative metadata, makes this still challenging.
We decided to use multiple data sources, with various license agreement. Further work could evaluate the impact of adding/removing data sources, then offering the opportunity to compare the importance of open data vs. not open data. A limitation to our approach is related to the field of PGx itself since only few (91) gene-drug relationships have a high level of evidence according to PharmGKB, making our training set relatively small. One way of enlarging the size of the training set would be to consider gene variant-drug relationships, instead of gene-drug, that have two advantages: being more numerous, but also rendering more precisely the state of the art of PGx knowledge. Indeed, a gene may host two (or more) variants, one that impacts drug response and one that does not. Two biases are to consider when interpreting the results.
First, negative examples are gene-drug relationships not listed in DGIdb, which includes known and predicted candidates from many databases. Consequently, negatives are likely not to be related, instead of not being related, but to our knowledge no existing resource lists negative gene-drug relationships. Second, tested examples are likely to be related since they are listed in PharmGKB with a low level of confidence. However, our goal is prioritize these candidates, instead of detecting negatives from those.
Conclusion This article is a proposal to help validating candidate pharmacogenes by learning from PGx linked data. More precisely, we selected and interconnected data relevant to the PGx domain in the form of a large RDF graph. Then, we formatted these data to train and compare a RF and a GK classifier. These two classifiers were evaluated and used to identify and rank candidate pharmacogenes. GK achieves a F-measure of 0. 81, whereas RF reaches 0. 73. Top candidate pharmacogenes pointed out by our approach are provided and interpretated in this article. Top candidates that are not already extensively studied will be further investigated by PGx experts. Results we obtained with the GK library named Mustard are particularly promising both for our application domain, i. e. , validating pharmacogenes, and more broadly for the mining of biomedical linked data.
To the best of our knowledge, ours is one of the largest empirical studies that analyzes the impact of coverage on post-release bugs in open-source software. In this study, we examine 100 large open-source Java projects that use the JIRA 1  bug tracking service, that provides support for bug tracking and project management. We download these 100 projects that are hosted on GitHub and use Maven. GitHub is one of the largest software repositories, which hosts millions of software projects including some popular projects such as spring-roo 2  from Spring, the WildFly Application server  3  (previously JBoss application server) from the WildFly community, and Maven 4  from Apache, all of which are present in our dataset. We execute test cases and calculate coverage for our 100 projects, considering cases where a method is called either directly or indirectly by a test case, and examine the relation between code coverage and the number of bugs found after the release of the software.
2) We measure the test adequacy by executing these test cases and analyse the correlation between code coverage and post-release bugs at the project and file level. 3) We draw on statistical methods and graphs to understand the impact of metrics such as lines of code and cyclomatic complexity on the correlation between code coverage and post-release bugs. 4) We make our dataset publicly available for other researchers to replicate our experiments and conduct future studies. In this paper, we describe code coverage, and the tools we use to collect information from our dataset in Section II. We explain the methodology of our study in Section III. We perform several statistical tests on the data to answer the two research questions and we provide results for these tests in Section IV. In Section V and Section VI, we provide several threats to validity and related work, respectively.
A. Code Coverage Software testing is used to test different functionalities of a program or system and to ensure that given a set of inputs the system produces the expected results. A test adequacy criterion defines the properties that must be satisfied for a thorough test  [12]. Code coverage, which measures the percentage of code executed by test cases, is often used as a proxy for test adequacy. The percentage of code executed by test cases can be measured according to various criteria, including the percentage of executed source code lines (line coverage), and the percentage of executed branches (branch coverage). Sonar combines these measures into a hybrid measure, referred to as coverage. 5 This coverage measure is efficient to compute, while still incorporating information about branches, which are important, because they may lead the program to very different behaviors.
B. Sonar Sonar 6 is an open-source platform that helps to the manage software quality of a project. Sonar can either be used as a standalone web based application or can be integrated into a Web Application Container such as Tomcat. Sonar uses various tools, such as JavaNCSS, 7 JaCoCo, 8 Cobertura, 9 and Surefire, 10 to extract software metrics such as cyclomatic complexity, lines of code (LOC), number of test cases, and code coverage. In our empirical study, we collect software metrics, such as cyclomatic complexity, lines of code, and code coverage using Sonar.
C. Maven Maven 4 is a software project management tool that supports building and running the software and its test cases. Maven uses information that is present in the project object model (POM) file, pom. xml. The POM file contains information about the project such as its dependencies on libraries and the order in which the different components of the project should be built. Maven primarily supports Java projects and for such projects it dynamically downloads all dependencies from a central Maven repository. Sonar makes use of Maven's project directory structure to get various information, such as the number of classes, the number of test cases, the number of packages and the overall lines of code. It also uses this structure to run test cases to collect the coverage of the project.
D. JIRA JIRA 1 is a project tracker used for issue tracking, bug tracking and efficient project management. To be able to uniformly obtain bug information for the different projects in our dataset, we focus on projects that use JIRA for reporting bugs. For each bug, JIRA records the affected and fixed version of the software, which represent the version in which bug was found and the version in which bug was fixed or resolved, respectively. This information ensures that we are collecting only post-release bugs i. e. , those bugs logged after the release of the particular version of the software. We collect information about all the closed and resolved bugs for a particular affected version of the software. JIRA also assigns each bug an identifier that is unique for the given software project.
E. GitHub GitHub is one of the largest project-hosting platforms and uses the git 11 version control system. GitHub is similar to a social network, where software developers spread across the globe can collaborate. Currently, GitHub has more than 11 million users and over 28 million repositories. We clone the repositories of software projects using the command git clone {url}. We only download projects that contain a Maven pom. xml file, implying that they are compatible with Sonar.
III. METHODOLOGY & STATISTICS In this section, we describe the methodology we use to collect data for this study. Furthermore, we also present several statisics to describe our dataset. A. Methodology a) Project Information: First, we search for open-source projects that use JIRA issue tracking system and allow public access to all of the issues filed in the tracking system. We find several examples of projects using public instances of JIRA  12 such as projects developed by the Apache Foundation, Spring Project, the WildFly (formerly JBoss) Community, etc. While these projects are popular and have a large base of contributors, they also cover a wide variety of programs ranging from build management, database, big data, etc. For this, we had to manually find the official web page of each project (>300) and verify whether the project's source code is available on GitHub and to identify their JIRA name.
We further restricted the projects to those that use Maven for project management. We, then, collect the source code of projects that are hosted on GitHub and use JIRA issue tracking system. For each project, we visited its website to confirm the major and stable releases and checked out the latest release of the software that was made at least 6 months prior to the month of collection of data (August 2013). For some of the projects, the stable release was made one or two years before August 2013, which gave ample time for users to use the release and report bugs. After collecting the releases, for each one of them, we run Sonar on these projects to collect metrics such as LOC, cyclomatic complexity, code coverage etc. We filter out projects with less than 5,000 lines of code as these projects are small and do not contain many test cases and have even fewer numbers of bugs.
In the end, we select top 100 projects sorted by size. Our dataset contains projects of different sizes ranging from 5,000 LOC to more than 100,000 LOC. Initially, to set up the project, we use the command mvn clean install in the root of each project repository. The clean command removes any files compiled during the prior builds that might be present in the repository and the install command builds a dependency tree for all the components specified in the pom. xml (the root POM). The install command also compiles the. java files present in the components specified in pom. xml into corresponding. class files. After the install phase, we use the command mvn sonar:sonar to collect coverage and other metrics. Before running this command, we need to start the Sonar web server, which has its own Maven repository, data repository, web services and Sonar plug-ins.