Bayer’s Approach to Modelling and Loading Data at Scale | by Daniel Crowe | Towards Data Science

Back in April we hosted an online conference for our community, Orbit 2021, and in listening to Henning Kuich, Dan Plischke, and Joren Retel from Bayer Pharmaceuticals, the community got a glimpse into how a team within Bayer Pharmaceuticals uses TypeDB to accelerate their drug discovery pipelines.
The team at Bayer fundamentally want to understand diseases better so that they can create better therapeutic interventions. A deeper understanding of diseases enables the identification and development of novel therapeutic interventions that have little to no side effects.
In order to achieve this, they need to build a model that accurately represents the biological system of diseases, and map that to the large biomedical datasets that they have available.
To model diseases in TypeDB, they started with concepts that can be modulated. Modelling these as entities in a TypeDB schema would be as follows:
definegene sub entity;variant sub entity;protein sub entity;complex sub entity;indication sub entity;
Here, gene, variant, protein and complex were chosen as entities as they can be associated in some form with a disease. The disease or diseases of interest are modelled through the entity indication. In practice, they need to capture the inherent interrelations within the biological system, and biology is complex! Of course, this means that we need to worry about an enormous amount of other things within the biological system.
For example, genes are expressed in cells, and cells are part of tissues, while tissues are in an organisam. These data could come from assays and experiments in clinical studies. Putting all of that together and modelling the right relations might lead to something that looks like this:
Why did the Bayer team choose to use TypeDB as the database for this type of modelling?
Everything in biology is extremely connected and context-dependent, which makes representing data in a traditional relational database very difficult. Because of this, Henning and his team felt that TypeDB’s type system was a much more natural way to model biology.
Biomedical experiments are highly context-specific, so when building a biomedical knowledge graph you want to be able to quickly interrogate what parameters were used, what is the actual assay, is it in-vitro, is it in a cell? This requires a database that can capture that level of complexity and a query language that allows us to easily ask complex questions over that data. With TypeDB, Henning’s team was able to do exactly that.
Bayer also leverages the benefits of the native inference engine in TypeDB. Inferencing can do a lot for teams with similarly complex biomedical data. For example, Henning and his team use TypeDB’s inference engine to infer variant to gene connections based on their locations on the chromosome.
Genes are essentially just sequences in the genome, literally locations along a long stretch of other locations. Variants are variations of these sequences, also along the same stretch of numbers and locations. Using TypeQL, (TypeDB’s query language) the team is able to map variants to genes based on overlapping locations, rather than having to depend on other databases that require them to import the relations between variants and genes. Crucially, this avoids the need to maintain those other databases, as it takes a long time to keep those updated and migrated on a regular basis. All of this is avoided by using TypeDB’s inference engine.
TypeDB also allows you to redefine a schema at any point in time during the database’s lifecycle. For example, we could add a new definition on what they mean by a variant being associated with a gene, depending on a gene location. This is important, especially when we add in new datasets. All of this enables Henning’s team to operate much more efficiently and accelerate their drug discovery process.
You can see an example of a fully fleshed out schema in the biology domain via the BioGrakn-Covid repo on github.
Now that there is a schema that represents the system we are working with, loading data at scale becomes the next challenge. To do this in the way Bayer wants to, Henning and his team built and open-sourced TypeDB Loader (formerly GraMi), a data migration tool for TypeDB. Here, we describe how it works and how the Bayer team uses it.
The reason Henning’s team decided to build their own custom loader, is that they wanted a better and more scalable way to load a large amount of data into TypeDB. To load data, the original TypeDB docs suggest building a function that reads in a file (for example in a tabular format), and building a templating function to generate the insert query. We can then use one of the client drivers to insert that into a TypeDB database. However, when doing this at scale, introduces a number of challenges that TypeDB Loader seeks to solve:
Handle the repeating logic of the templating functions needed
Taking care of the potential for dirty data, preventing errors during insert
The parallelisation and batching of insert queries per transaction
Big data fault tolerance
Using the defined TypeDB schema, our first step is to map the data to the schema. This is handled in two configuration files. The first is the processor config, which contains logic between the data and schema; for example, which attributes must be generated for each type. This determines whether or not certain attributes are required, and ensures that the data ingested produces the right attribute of the right type.
The second configuration file is the data configuration file. Here we can specify which data file is being read, what kind of file it is, what kind of processor you are using out of the processor config, and then map the columns in your data to the actual processor in the processor config. This ensures that everything is inserted properly.
The Bayer team also wanted to be able to monitor the data migration as it was happening. What has been inserted, where errors may have occurred and where it stopped for whatever reason. This allows you to restart the migration at the exact point where you left off, removing the risk of duplicating or losing data.
TypeDB Loader is a Java application and library, enabling you to use it as a CLI tool or use it within your own Java project. Once you decide which of the two ways you will use it, you specify what TypeDB instance you are talking to and which database you are writing into. You then provide your data configuration, your processor configuration, TypeDB schema, and then determine where you want the migration status to be tracked. This can be as simple as a JSON file.
This is it, TypeDB Loader takes it from there. Henning and team are continuing to release updates and encourage anyone interested to follow the developments on the Discord in the #typedb-loader channel.
Import nested relations with attributes as role players
Append attributes to existing types
Append or insert data to existing data
Update schema including updating rules
Lots of tests on data insertion to prevent data loss or errors
Processor and data config validation that catches errors before migration starts
Auto-generate processor config skeleton for quick start
Improving data warning logging, a consolidated output that is easier to understand and parse
Rule management, making it easier to apply and define or undefined rules
Reading from SQL tables directly instead of just files
Building a Kafka connector so it can be a part of a larger data processing infrastructure