Automate Neo4j Database Construction with Python
By: Madison Gipson
Tags: Coding, Python, Neo4j, Data Science, Graph Database

Coding Time

Note: the full code for this project is in the following repository, so you can reference the context of the gists below or check out the data I was importing.

Script Skeleton

Our main Python script that imports the data or builds the nodes and relationships will live inside that handy Python folder and virtual environment.

This code block does the bare minimum:

1. Import py2neo

2. Establish a database connection

3. Run a single query

With Py2Neo, you can run Cypher queries as-is or use the Py2Neo functions to perform similar actions. My personal preference is running Cypher queries as-is because I can simply copy and paste the code to and from the Neo4j browser.

If you want something super simple, you could repeat the graph.run() call with your different queries and it would get the job done. However, there are some things I discovered when developing my own script that helped tremendously; keep reading if you want to see if these will be useful to you.

Batching & Auto-Committing

I leveraged periodic commits (aka batching) because I kept running into memory issues with my heavier-lifting queries.

If you do run in batches, you’ll want to use an open transaction (accomplished with the begin(), run(), commit() sequence in the code below) instead of a closed transaction (standalone run() call).

This is because a closed transaction auto-commits, meaning that if you use batching and a closed transaction, it will execute and commit your first batch but not continue to do the other batches. Definitely not what we want.

#Without Periodic Commits graph.run("""LOAD CSV WITH HEADERS

FROM 'file:///womeninstem.csv' AS line

MERGE (person:Person { name: line.personLabel })

ON CREATE SET person.birth = line.birthdate, person.death = line.deathdate""") #With Periodic Commits query_list = [] query_list.append("""CALL apoc.periodic.iterate("

LOAD CSV WITH HEADERS

FROM 'file:///womeninstem.csv' AS line

RETURN line

","

MERGE (person:Person { name: line.personLabel })

ON CREATE SET person.birth = line.birthdate, person.death = line.deathdate", {batchSize:1000, parallel:false, retries: 10}) YIELD operations""") #would have more queries in query_list for query in query_list:

g = graph.begin() # open transaction

result = g.run(query).to_data_frame() # execute query

g.commit() # close transaction

Query Failure Alerts

The thing about running a Cypher query through Python code instead of in the browser is that it doesn’t always tell you if the query fails- it will alert you of syntax errors, but that’s about it.

For that reason, it’s important to have something implemented that alerts you if a query doesn’t finish completely.

query_list = [] query_list.append("""CALL apoc.periodic.iterate("

LOAD CSV WITH HEADERS

FROM 'file:///womeninstem.csv' AS line

RETURN line

","

MERGE (person:Person { name: line.personLabel })

ON CREATE SET person.birth = line.birthdate, person.death = line.deathdate", {batchSize:1000, parallel:false, retries: 10}) YIELD operations""") #would have more queries in query_list for query in query_list:

g = graph.begin() # open transaction

result = g.run(query).to_data_frame() # execute query

try:

if result['operations'][0]['failed'] > 0: #means batch failed

print('Could not finish query')

else: #means batch succeeded

g.commit() # close transaction

print('Completed query ')

except Exception as e: #means there's a syntax or other error

g.commit() # close transaction

print('Completed query')

Parameterizing Queries