Package: bluegraph
Unifying Python framework for graph analytics and co-occurrence analysis.AboutBlueGraph is a Python framework that consolidates graph analytics capabilities from different graph processing backends. It provides the following set of interfaces:preprocessing and co-occurrence analysis API providing semantic property encoders and co-occurrence graph generators;graph analytics API providing interfaces for computing graph metrics, performing path search and community detection;representation learning API for applying various graph embedding techniques;representation learning downstream tasks API allowing the user to perform node classification, similarity queries, link prediction.Using the built-inPGFramedata structure (currently,pandas-based implementation is available) for representing property graphs, it provides a backend-agnostic API supporting the following in-memory and persistent graph backends:NetworkX(for the analytics API)graph-tool(for the analytics API)Neo4j(for the analytics and representation learning API);StellarGraph(for the representation learning API).gensim(for the representation learning API).This repository originated from the Blue Brain effort on building a COVID-19-related knowledge graph from theCORD-19dataset and analysing the generated graph to perform literature review of the role of glucose metabolism deregulations in the progression of COVID-19. For more details on how the knowledge graph is built, explored and analysed, seeCOVID-19 co-occurrence graph generation and analysis.bluegraphpackageBlueGraph’s API is built upon 4 main packages:bluegraph.coreproviding the exchange data structure for graph representation that serves as the input to graph processors based on different backends (PGFrame), as well as basic interfaces for different graph analytics and embedding classes (MetricProcessor,PathFinder,CommunityDetector,GraphElementEmbedder, etc).bluegraph.backendsis a package that collects implementation of various graph processing and analytics interfaces for different graph backends (for example,NXPathFinderfor path search capabilities provided by NetworkX,Neo4jCommunityDetectorfor community detection methods provided by Neo4j, etc).bluegraph.preprocessis a package that contains utils for preprocessing property graphs (e.g.SemanticPGEncoderfor encoding node/edge properties as numerical vectors,CooccurrenceGeneratorfor generation and analysis of co-occurrence relations in PGFrames.)bluegraph.downstreamis a package that provides a set of utils for various downstream tasks based on vector representations of graphs and graph elements (for example,NodeSimilarityProcessorfor building and querying node similarity indices based on vector representation of nodes,EdgePredictorfor predicting true and false edges of the graph based on vector representation of its nodes,EmbeddingPipelinefor stacking pipelines of graph preprocessing, embedding, similarity index building, etc).Main components of BlueGraph’s API are illustrated in the following diagram:cord19kgpackageThecord19kgpackage contains a set of tools for interactive exploration and analysis of theCORD-19dataset using the co-occurrence analysis of the extracted named entities. It includes data preparation and curation helpers, tools for generation and analysis of co-occurrence graphs. Moreover, it provides several interactive mini-applications (based onJupyterDashandipywidgets) for Jupyter notebooks allowing the user to interactively perform:entity curation;graph visualization and analysis;dataset saving/loading fromNexus.servicespackageCollects services included as a part of BlueGraph. Currently, only a mini-service for retrieving embedding vectors and similarity computation is included as a part of this repository (see embedder service specificREADME).InstallationIt is recommended to use a virtual environment such asvenvorconda environment.Installing backend dependenciesIf you want to usegraph-toolas a backend, you need to manually install the library (it cannot be simply installed by runningpip install), as it is not an ordinary Python library, but a wrapper around a C++ library (please, seegraph-tool installation instructions). Currently, BlueGraph supportsgraph-tool<=2.37.Similarly, if you want to use thebluegraph.downstream.similaritymodule for building similarity indices (on embedded nodes, for example), you should install the FacebookFaisslibrary separately. Please, seeFaiss installation instructions(condaandconda-forgeinstallation available).You can install bothgraph-tooland the FacebookFaisslibrary by creating a new environment with the right dependencies usingconda, as follows:conda create --name <your_environment> -c conda-forge graph-tool==2.37 faiss python=<your_python>
conda activate <your_environment>The same holds for the Neo4j backend: in order to use it, the database should be installed and started (please, seeNeo4j installation instructions). Typically, the Neo4j-based interfaces provided by BlueGraph require the database uri (the bolt port), username and password to be provided. In addition, BlueGraph uses the Neo4j Graph Data Science (GDS) library, which should be installed separately for the database on which you would like to run the analytics (seeinstallation instructions). Current supported Neo4j GDS version is>=1.6.1.Installing BlueGraphBlueGraph supports Python versions >= 3.7 and pip >= 21.0.1. To update pip from the older versions run:pip install --upgrade pip wheel setuptoolsThe stable version of BlueGraph can be installed from PyPI using:pip install bluegraphThe development version of BlueGraph can be installed from the source by cloning the current repository as follows:git clone https://github.com/BlueBrain/BlueGraph.git
cd BlueGraphBasic version including only the NetworkX backend can be installed using:pip install bluegraphThe prerequisites for using thegraph-toolbackend can be found in ‘Installing backend dependencies’. You can also install additional backends for Neo4j and StellarGraph by running the following:pip install bluegraph[<backend>]Where<backend>has one of the following valuesneo4jorstellargraph.Alternatively, a version supporting all the backends can be installed by running the following commands:pip install bluegraph[all]In order to use thecord19kgpackage and its interactive Jupyter applications, run:pip install bluegraph[cord19kg]Getting startedTheexamples directorycontains a set of Jupyter notebooks providing tutorials and usecases for BlueGraph.To get started with property graph data structurePGFrameprovided by BlueGraph, get an example of semantic property encoding, see thePGFrames and semantic encoding tutorialnotebook.To get familiar with the ideas behind the co-occurrence analysis and the graph analytics interface provided by BlueGraph we recommend to run the following example notebooks:Literature exploration (PGFrames + in-memory analytics tutorial)illustrates how to use BlueGraphs’s analytics API for in-memory graph backends based on theNetworkXand thegraph-toollibraries.NASA keywords (PGFrames + Neo4j analytics tutorial)illustrates how to use the Neo4j-based analytics API for persistent property graphs.Embedding and downstream tasks tutorialstarts from the co-occurrence graph generation example and guides the user through the graph representation learning and all it’s downstream tasks including node similarity queries, node classification and edge prediction.Create and run embedding pipelinesillustrates how embedding pipelines can be built and executed using BlueGraph.Finally,Create and push embedding pipeline into Nexus.ipynbillustrates how embedding pipelines can be created and pushed toNexusandEmbedding service APIshows how embedding service that retrieves the embedding pipelines from Nexus can be used.Getting started with cord19kgThecord19kgpackages providesexamplesof CORD-19-specific co-occurrence analysis. Please, see more details on the CORD-19 analysis and exploration pipeline of the Blue Brain Projecthere.We recommend starting from theCo-occurrence analysis tutorialnotebook providing a simple starting example.TheTopic-centered co-occurrence network analysis of CORD-19notebook provides a full analysis pipeline on the selection of 3000 articles obtained by searching the CORD-19 dataset using the query“Glucose is a risk factor for COVID-19”(the search is performed usingBlueBrainSearch).TheNexus-hosted co-occurrence network analysis of CORD-19notebook provides an example for the previously mentioned 3000-article dataset, where datasets corresponding to different analysis steps can be saved and loaded to and from aBlue Brain Nexusproject.Finally, thegenerate_10000_network.pyscriptallows the user to generate the co-occurrence networks for 10’000 most frequent entities extracted from the entire CORD-19v47 database (based on paper- and paragraph- level entity co-occurrence). To run the script, simply executepython generate_10000_network.pyfrom the examples folder.Note that the generated networks are highly dense (contain a large number of edges, for example, ~44M edges for the paper-based network), and the process of their generation, even if parallelized, is highly costly.LicensingBlue Graph is distributed under the Apache 2 license.Included example scripts and notebooks (BlueGraph/examplesandBlueGraph/cord19kg/examples) are distributed under the 3-Clause BSD License.Data files stored in the repository are distributed under theCommons Attribution 4.0 International license (CC BY 4.0)License.Funding & AcknowledgementsThe development of this project was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.Copyright (c) 2020-2021 Blue Brain Project/EPFL