Package: openbiolink
OpenBioLinkOpenBioLink is a resource and evaluation framework for evaluating link prediction models on heterogeneous biomedical graph data. It contains benchmark datasets as well as tools for creating custom benchmarks and evaluating models.DocumentationPaper preprint on arXiv•Peer reviewed paper in the journal Bioinformatics (for citations)•Supplementary dataThe OpenBioLink benchmark aims to meet the following criteria:Openly availableLarge-scaleWide coverage of current biomedical knowledge and entity typesStandardized, balanced train-test splitOpen-source code for benchmark dataset generationOpen-source code for evaluation (independent of model)Integrating and differentiating multiple types of biological entities and relations (i.e., formalized as a heterogeneous graph)Minimized information leakage between train and test sets (e.g., avoid inclusion of trivially inferable
relations in the test set)Coverage of true negative relations, where availableDifferentiating high-quality data from noisy, low-quality dataDifferentiating benchmarks for directed and undirected graphs in order to be applicable to a wide variety of link prediction methodsClearly defined release cycle with versions of the benchmark and public leaderboardBenchmark DatasetTheOpenBioLink2020 Datasetis a highly challenging
benchmark dataset containing over 5 million positive and negative edges.
The test set does not contain trivially predictable, inverse edges from the training set
and does contain all different edge types, to provide a more realistic edge prediction
scenario.OpenBioLink2020: directed, high qualityis the default dataset that should be used for benchmarking purposes. To allow anayzing the effect of data quality as well as the directionality of the
evaluation graph, four variants of OpenBioLink2020 are provided -- in directed and undirected setting,
with and without quality cutoff.Additionally, each graph is available inRDF N3format (without train-validation-test splits).OpenBioLink 2020 datasetsAll datasets are hosted onzenodo.OpenBioLink2020: directed, high quality//RDF(default dataset for benchmarking)OpenBioLink2020: undirected, high quality//RDFOpenBioLink2020: directed, no quality cutoff//RDFOpenBioLink2020: undirected, no quality cutoff//RDFDatasets summaryDatasetTrainTestValidEntitiesRelationsdirected, high quality8.503.580401.901397.066184.73228undirected, high quality7.559.921372.877357.297184.72228directed, no quality cutoff51.636.9272.079.1392.474.921486.99832undirected, no quality cutoff41.383.0932.010.6621.932.436486.99832Previous versions of the Benchmark (click to expand)OpenBioLink 2020 alpha-releaseOpenBioLink2020: directed, high quality(default dataset)OpenBioLink2020: undirected, high qualityOpenBioLink2020: directed, no quality cutoffOpenBioLink2020: undirected, no quality cutoffPlease note that the OpenBioLink benchmark files contain data derived from external ressources. Licensing terms of these external resources are detailedbelow.Baseline resultsModelHits@10Hits@3Hits@1RESCAL0.6150.4790.407TransR0.5920.4510.369DistMult0.5340.3310.184ComplEx0.5250.3140.166RotatE0.5220.3150.156TransE0.4410.2680.128Hyperparameter used to achieve these results usingdglke:learning rateembedding   sizeregularization   coefficientgammaiterationsRESCAL0.053003.00E-07350000   on 2 GPUsTransR0.12201.00E-0812550000   on 2 GPUsComplEx0.13802.00E-06360000   on 8 GPUsDistMult0.13804.00E-07950000   on 2 GPUsRotatE0.051281.00E-0712550000   on 2 GPUsTransE0.13603.00E-098550000   on 2 GPUsInstallationPipInstall a pytorch version suitable for your systemhttps://pytorch.org/pip install openbiolinkSourceclone the git repository or download the projectCreate a new python3.7, or python3.6 virtual environment(note: under Windows, only python3.6 will work)e.g.:python3 -m venv my_venvactivate the virtual environmentwindows:my_venv\Scrips\activatelinux/mac:source my_venv/bin/activateInstall a pytorch version suitable for your systemhttps://pytorch.org/Install the requirements stated in requirements.txt e.g.pip install -r requirements.txtManualThe OpenBioLink framework consists of three parts:Graph creationDataset splitEvaluationThe creation of the graph and the splitting of the created graph in training, testing and an optional validation set can be performed by either via the GUI or the command line interface. The evaluation of a trained model is served as part of theopenbiolinklibrary.Graph creation & Dataset splitGUIBy callingopenbiolinkfrom the command line a graphical user interface is started, providing an interface to create a graph and perform a dataset split. Step by step instructions on how to use the GUI can be found in thewiki.Command line interfaceopenbiolink-pWORKING_DIR_PATH[-action][--options]...Graph CreationTo generate the default graph (with all edges of all qualifies) in the current directory, use:openbiolinkgenerateFor a list of arguments, use:openbiolinkgenerate--helpDataset SplitTo split the default graph using the random scheme, use:openbiolinksplitrand--edgesgraph_files/edges.csv--tn-edgesgraph_files/TN_edges.csv--nodesgraph_files/nodes.csvFor a list of arguments, use:openbiolinksplitrand--helpSplitting can also be done by time withopenbiolinksplittimeMore documentation will be provided later.EvaluationTo ensure a standardized evaluation of different methods applied to the OpenBioLink dataset, an evaluator is provided in the packageopenbiolink. For examples how to evaluate a model, seehere.DataloaderAll versions of the OpenBioLink datasets can be easily accessed within Python via theDataLoader, which downloads all required files automatically.fromopenbiolink.evaluation.dataLoaderimportDataLoader# Name of the Dataset, possible values HQ_DIR, HQ_UNDIR, ALL_DIR, ALL_UNDIR. Default: HQ_DIRdl=DataLoader("HQ_DIR")train=dl.training.mapped_triplestest=dl.testing.mapped_triplesvalid=dl.validation.mapped_triplesFile descriptionGraph GenerationTSV WriterDefault File NameDescriptionColumnsALL_nodes.csvAll nodes present in the graphNode Id, Node typeedges.csvAll true positive edgesNode 1 ID, Edge type, Node 2 ID, Quality score, Sourceedges_list.csvList of edge types present in edges.csvEdge typenodes.csvAll nodes present in edges.csvNode ID, Node typenodes_list.csvList of node types present in nodes.csvNode typeTN_edges.csvAll true negative edgesNode 1 ID, Edge type, Node 2 ID, Quality score, SourceTN_edges_list.csvList of edge types present in TN_edges.csvEdge typeTN_nodes.csvAll nodes present in TN_edges.csvNode ID, Node typeTN_nodes_list.csvList of node types present in TN_nodes.csvNode typeids_no_mapping.tsvID's of nodes that could not be mapped to other ontology systemsNode ID, Node typetn_ids_no_mapping.tsvID's of nodes that could not be mapped to other ontology systemsNode ID, Node typestats.txtStatistics about edges.csv and nodes.csv(See column headers of file)tn_stats.txtStatistics about TN_edges.csv and TN_nodes.csv(See column headers of file)Biological Expression Language (BEL) WriterThe Biological Expression Language (BEL) is a domain specific language that enables the expression of
biological relationships in a machine-readable format. It is supported by thePyBELsoftware ecosystem.BEL can be exported with:openbiolinkgenerate--output-formatBELDefault File NameDescriptionpositive.bel.gzAll true positive edges inBEL Scriptformat (gzipped) for usage in PyBEL or other BEL-aware applications)positive.bel.nodelink.json.gzAll true positive edges inNodelink JSONformat (gzipped) for direct usage withPyBELnegative.bel.gzAll true negative edges in BEL Script format (gzipped)negative.bel.nodelink.json.gzAll true negative edges in Nodelink JSON format (gzipped)Example opening BEL Script usingpybel.from_bel_script():importgzipfrompybelimportfrom_bel_scriptwithgzip.open('positive.bel.gz')asfile:graph=from_bel_script(file)Example opening Nodelink JSON usingpybel.from_nodelink_gz():frompybelimportfrom_nodelink_gzgraph=from_nodelink_gz('positive.bel.nodelink.json.gz')There's an externally hosted copy of OpenBioLinkherethat contains
the exports as BEL.Train-test split creationDefault file nameDescriptionColumn descriptionstrain_sample.csvAll positive samples from the training setNode 1 ID, Edge type, Node 2 ID, Quality score, TP/TN, Sourcetest_sample.csvAll positive samples from the test setNode 1 ID, Edge type, Node 2 ID, Quality score, TP/TN, Sourceval_sample.csvAll positive samples from the validation setNode 1 ID, Edge type, Node 2 ID, Quality score, TP/TN, Sourcenegative_train_sample.csvAll negative samples from the training setNode 1 ID, Edge type, Node 2 ID, Quality score, TP/TN, Sourcenegative_test_sample.csvAll negative samples from the test setNode 1 ID, Edge type, Node 2 ID, Quality score, TP/TN, Sourcenegative_val_sample.csvAll negative samples from the validation setNode 1 ID, Edge type, Node 2 ID, Quality score, TP/TN, Sourcetrain_val_nodes.csvAll nodes present in the training and validation set combinedNode ID, Node typetest_nodes.csvAll nodes present in the test setNode ID, Node typremoved_test_nodes.csvAll nodes which got removed from the test set, due to not being present in the trainingsetNode IDremoved_val_nodes.csvAll nodes which got removed from the validation set, due to not being present in the trainingsetNode IDCURIE'sAll node ID's in the graph are CURIES, meaning entities can be easily looked up online by concatenatinghttps://identifiers.org/with the ID, f.e.:CURIEIdentifiers.orgGO:0006915https://identifiers.org/GO:0006915REACTOME:R-HSA-201451https://identifiers.org/REACTOME:R-HSA-201451Detailed information of how the Identifiers are resolved can be found herehttps://registry.identifiers.org/Train-test-split creationRandom splitIn the random split setting, first, negative sampling is performed. Afterwards, the whole dataset (containing positive
and negative examples) is split randomly according to the defined ratio. Finally, post-processing steps are performed to
facilitate training and to avoid information leakage.Time-slice splitIn the time-slice split setting, for both of the provided time slices, first, negative sampling is performed. Afterwards,
the first time slice (t-1 graph) is used as training sample, while the difference between the first and the second time
slice serves as the test set. Finally, post-processing steps are performed to
facilitate training and to avoid information leakage.Generally, the time slice setting is trickier to implement than the random split strategy, as it requires more manual evaluation and
knowledge of the data. One of the most difficult factors is the change of the source databases over time. For example,
a database might change its quality score, or even its ID-format. Also, the number of relationships stored might increase
sharply due to new mapping files being used. This might also result in ‘vanishing edges’, where edges that were present
in the t-1 graph are no longer existent in the current graph. Although the OpenBioLink toolbox tries to assist the user with
different kinds of warnings to identify such difficulties in the data, it is unfortunately not possible to automatically detect nor solve all these problems, making some manual pre- and post-processing of the data inevitable.Negative samplingFirst, the distribution of edges of different types is calculated to know how many samples are needed from each edge type.
For now, this distribution corresponds to the original distribution (uniform distribution could a future extension).
Then, subsamples are either – where possible – taken from existing true negative edges or are created using type-based sampling.In type-based sampling, head and tail node are randomly sampled from a reduced pool of all nodes, which only
includes nodes with types that are compatible with the corresponding head- or tail-role of the given relation type.
E.g., for the relation type GENE_DRUG, one random node of type GENE is selected as head node and one
random node of type DRUG is selected as tail.In most cases where true negative edges exist, however, their number is smaller than the number of positive examples.
In these cases, all true negative samples are used for the negative set, which is then extended by samples created by type-based
sampling.Train-test-set post-processingTo facilitate model applicationEdges that contain nodes that are not present in the training set are dropped from the test set. This facilitates use of embedding-based models that usually cannot make predictions for nodes that have not been embedded during training.Avoiding train-test information leakage and trivial predictions in the test setRemoval of reverse edgesIf the graph is directed, reverse edges are removed from the training set.
The reason for this is that if the original edge a-b was undirected, both directions a→b and a←b are materialized in the directed graph.
If one of these directed edges would be present in the training set and one in the test set, the prediction would be trivial.
Therefore, in these cases, the reverse edges from the training set are removed. (Note that edges are removed from the training set instead of the test set because this is advantagous for maintaining the train-test-set ratio)Removal of super-propertiesSome types of edges have sub-property characteristics, meaning that relationship x indicates a generic interaction between two entities (e.g.protein_interaction_protein),
while relationship y further describes this relationship in more detail (e.g.,protein_activation_protein). This means that the presence of x between two nodes does not imply
the existence of a relation y between those same entities, but the presence of y necessarily implies the existence of x. These kinds of relationships
could cause information leakage in the datasets, therefore super-relations of relations present in the training set are removed
from the test set.True Negative edgesAs randomly sampled negative edges can produce noise or trivial examples, true negative edges (i.e., relationships that were explicitly mentioned to not exist) were used wherever possible.
Specifically, for disease_drug and disease_phenotype edges, true negative examples were extracted from the data source directly, as they were explicitly stated. For gene-anatomy relationships, over-expression and under-expression data was used as contradicting data. For other relationship-types, e.g., gene_activation_gene and drug_inhibition_gene, this indirect true negative sample creation could not be applied, as the relationship does not hold all information necessary (the same substance can have both activating and inhibiting effects, e.g. depending on dosage).Source databases and their licensesSource typeSource nameLicenseTrue neg.Scoreedge (gene-gene)STRINGCC BYNoYesedge (gene-go)GOCC BYNoYesedge (gene-disease)DisGeNetCC BY-NC-CANoYesedge (gene-phenotype)HPOCustom:HPONoNoedge (gene-anatomy)BgeeCC 0YesYesedge (gene-drug)STITCHCC BYNoYesedge (gene-pathway)CTDCustom:CTDNoNoedge (disease-phenotype)HPOCustom:HPOYesNoedge (disease-drug)DrugCentralCC BY-SAYesNoedge (drug-phenotype)SIDERCC BY-NC-CANoNoontology (genes)GOCC BYontology (diseases)DOCC 0ontology (phenotype)HPOCustom:HPOontology (anatomy)UBERONCC BYmapping (UMLS-DO)DisGeNetCC BY-NC-CAmapping (STRING-NCBI)STRINGCC BYmapping (ENSEMBL/UNIPROT-NCBI)UniProtCC BYid (genes)NCBIPublic Domainid (go)GOCC BYid (anatomy)UBERONCC BYid (disease)DOCC 0id (drug)PubChemPublic Domainid (phenotype)HPOCustom:HPOid (pathway)REACTOMECC BYid (pathway)KEGGCustom:KEGG(True neg.: whether the data contains true negative relations; Score: whether the data contains evidence quality scores for filtering relations)The OpenBioLink benchmark files integrate data or identifiers from these sources. The provenance of data items is captured in the benchmark files, and licensing terms of source databases apply to these data items. Please mind these licensing terms when utilizing or redistributing the benchmark files or derivatives thereof.All original data in the benchmark files created by the OpenBioLink project (not covered by the licenses of external data sources)  are released asCC 0.We offer the benchmark files as-is and make no representations or warranties of any kind concerning the benchmark files, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law.Citation@article{10.1093/bioinformatics/btaa274,author={Breit, Anna and Ott, Simon and Agibetov, Asan and Samwald, Matthias},title="{OpenBioLink: a benchmarking framework for large-scale biomedical link prediction}",journal={Bioinformatics},volume={36},number={13},pages={4097-4098},year={2020},month={04},issn={1367-4803},doi={10.1093/bioinformatics/btaa274},url={https://doi.org/10.1093/bioinformatics/btaa274},eprint={https://academic.oup.com/bioinformatics/article-pdf/36/13/4097/33458979/btaa274.pdf},}