Evolving Data Models with JanusGraph
By: Ryan Stauffer
Tags: Graph, Data Model, Music Analytics, Tinkerpop, Janusgraph

Over the next few weeks, we’ll be taking an episodic look at building a graph data system to tackle interesting problems in interesting ways. There’s no better place to start than right in the middle — the data model. After all, a key feature of a graph data system is the tremendous flexibility it gives us to explicitly describe our data. In its ideal state, our graph data system would offer infinite flexibility for data storage and retrieval. A lot of this flexibility comes from being able to take an existing graph and change its data model to reflect new understandings about our data.

To demonstrate this, we’ll iteratively build our data model, discover some pitfalls to avoid, and check out how to improve the data model in a working database without needing to drop and rebuild the entire thing. We’ll look at some sample code and real data. All our experiments can be run with a simple in-memory JanusGraph instance running from the Gremlin Console. Production systems will be a bit more complex, but this setup works just fine for our data modeling examples.

To begin our applied graph data modeling exercise, we of course start with some data. Let’s choose something from a domain that’s a bit different from most of the examples out in the wild — data on concert performances by U.S symphony orchestras. The Baltimore Symphony Orchestra has done a great job compiling some of this data, which they use to analyze different aspects of professional orchestras’ program choices. For this example, we’ll be looking at a single season of data from 2016-2017.

As we can see, the raw data is pretty straightforward.

To build an appropriate schema, we start by conceptualizing our data into different logical categories. Broadly, we care about what was played, as well as who play it. The what are Concerts consisting of multiple individual Pieces, each written by a Composer. The who is the Orchestra (a group of many players), and a couple individual artists of interest — the Conductor and the Soloist.

This means we’re modeling 6 distinct elements:

Concert

Piece

Orchestra

Composer

Conductor

Soloist

But we can immediately find some areas for improvement. It’s clear that some Conductors might also be Composers (like Esa-Pekka Salonen) or Soloists (like Itzhak Perlman). We prefer to not have two separate vertices representing the same distinct individual, so we’ll roll these up under a single Artist label. We’ll use the relationship of that person to a Concert to describe their role in the proceedings. We can keep Orchestra separate because it is an organization made up of different individual artists. We’ll also prefer the more generic term Work over Piece, to encompass a broader variety of music.

This leaves us with 4 vertex labels:

Concert

Work

Orchestra

Artist

We can begin to connect everything together with edges like this:

Our initial schema design

This is our starting point, though there is certainly room for improvement. If some of the pitfalls of this model have already caught your eye — be patient! Evolving the schema on our graph is the key part of this exercise.

Defining an Initial Data Model

Let’s get to coding. We’ll be defining our schema in the Gremlin Console with the JanusGraph Management API.

We start by opening up the Gremlin Console and providing a simple inline configuration for our JanusGraph instance. We also open the JanusGraph Management API. There are many more things to say about configuring JanusGraph — like removing schema defaults and turning on schema constraints — but we’ll tackle those in a separate episode.

$ bin/gremlin.sh

gremlin> graph = JanusGraphFactory.build().

set('storage.backend', 'inmemory').open()

gremlin> mgmt = graph.openManagement()

Going forward we’ll assume we’re already working in our Gremlin Console, so we’ll leave out the gremlin> prompt for readability and conciseness.

First we define our Vertex and Edge labels. Edge Multiplicity choices can be tricky, but our choices here — SIMPLE and MANY2ONE — should serve us well for this example.

// Vertices

Orchestra = mgmt.makeVertexLabel('Orchestra').make()

Artist = mgmt.makeVertexLabel('Artist').make()

Work = mgmt.makeVertexLabel('Work').make()

Concert = mgmt.makeVertexLabel('Concert').make() // Edge

COMPOSER = mgmt.makeEdgeLabel('COMPOSER').

multiplicity(MANY2ONE).make()

SOLOIST = mgmt.makeEdgeLabel('SOLOIST').

multiplicity(SIMPLE).make()

CONDUCTOR = mgmt.makeEdgeLabel('CONDUCTOR').

multiplicity(SIMPLE).make()

ORCHESTRA = mgmt.makeEdgeLabel('ORCHESTRA').

multiplicity(SIMPLE).make()

INCLUDES = mgmt.makeEdgeLabel('INCLUDES').

multiplicity(SIMPLE).make()

We successively define all the properties for each Vertex label, and attach them to their respective labels. String is a reasonable choice for a default data type. If it looks like there may be parsing errors, and we just want to begin testing our schema, Strings work just fine! A cardinality of SINGLE is likewise a convenient default, and is perfect for this data model. We can always iteratively improve these over time. Finally, we create the connections between edges and vertex labels.

// Define Vertex Property Keys

// Orchestra

name = mgmt.makePropertyKey('name').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

mgmt.addProperties(Orchestra, name) // Artist

lastName = mgmt.makePropertyKey('lastName').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

firstName = mgmt.makePropertyKey('firstName').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

gender = mgmt.makePropertyKey('gender').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

nationality = mgmt.makePropertyKey('nationality').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

deceased = mgmt.makePropertyKey('deceased').

dataType(Boolean.class).cardinality(Cardinality.SINGLE).make()

mgmt.addProperties(Artist, lastName, firstName, gender,

nationality, deceased) // Work

title = mgmt.makePropertyKey('title').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

compositionDate = mgmt.makePropertyKey('compositionYear').

dataType(Integer.class).cardinality(Cardinality.SINGLE).make()

soloInstrument = mgmt.makePropertyKey('soloInstrument').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

mgmt.addProperties(Work, title, compositionDate, soloInstrument) // Concert

firstDate = mgmt.makePropertyKey('firstDate').

dataType(String.class).cardinality(Cardinality.SINGLE).make()

numShows = mgmt.makePropertyKey('numShows').

dataType(Integer.class).cardinality(Cardinality.SINGLE).make()

mgmt.addProperties(Concert, name, firstDate, numShows)

// Define connections as (edgeLabel, outVertexLabel, inVertexLabel)

mgmt.addConnection(COMPOSER, Work, Artist)

mgmt.addConnection(SOLOIST, Work, Artist)

mgmt.addConnection(CONDUCTOR, Work, Artist)

mgmt.addConnection(ORCHESTRA, Concert, Orchestra)

mgmt.addConnection(INCLUDES, Concert, Work) mgmt.commit()

We finish off by committing our management transaction. Don’t forget this last part, or you’re certain to encounter some frustrating errors later. Also note that variables referencing vertex and edge labels and property keys are only valid in the current management API transaction. If we commit or close our transaction, we’ll need to open up another one if we plan on doing anything with these variables.

A Quick Aside on Schema Naming

Choosing a schema naming convention fundamentally comes down to three things:

The particulars of the data you’re modeling The conventions of other data systems or applications you might be interacting with (don’t change existing conventions just for the sake of changing them — it ruins any existing documentation and just makes for more work all around) Personal preference

I personally lean toward “PascalCase” vertex labels, “ALL_CAPS_UNDERSCORE” edge labels, and “camelCase” property names. This clearly distinguishes between the different graph elements during discussion, and also happens to align with the common Neo4j style conventions. I’m admittedly a bit obsessive about good naming, but I try not to get too pedantic about the parts of speech I use for edge labels. If a noun is the clearest way to express the relationship — go for it. So we’re using COMPOSER and SOLOIST as some of our edge labels, because those are the simplest and most concise ways of expressing the relationships in question.

Answering Questions with our Graph

For an initial peek into our model’s effectiveness, we’ll load the following data into our graph: