Exploring the full-text search index in Neo4j on a movies dataset | by Tomaz Bratanic | Towards Data Science

The full-text search index was introduced in Neo4j version 3.5 with Apache Lucene powering the index. The agenda of this blog post is to introduce basic Lucene query language and describe how to use it in Neo4j. As you will see, we can also combine Lucene with Cypher to manipulate results.
I got most of the inspiration for this blog post from Christophe Willemsen, who is part of the Graphaware team. He has described the use of FTS very nicely for noobies like me, thank you!
Blog: Deep Dive into Neo4j 3.5 Full Text Search
Webinar: To be or not to be. Full Text Search tips and tricks.
We will be using the Movie dialog corpus dataset available on kaggle. It is rich with meta-data as it contains all the conversations between the characters in 617 different films. I have a couple of blog posts lined up using this dataset, but for starters, we will only use the data about the movies to explore the full-text search index capabilities.
We have two labels(Movie, MovieTag) in our graph. They are connected with the HAS_TAG relationship. Movies can have additional attributes like release year, IMDB rating, and the number of votes on IMDB.
Create constraints
To optimize import and later queries, we define unique constraints for both movies and movie tags.
CREATE CONSTRAINT ON (m:Movie) ASSERT m.id IS UNIQUE;CREATE CONSTRAINT ON (m:MovieTag) ASSERT m.id IS UNIQUE;
Copy the movie_titles_metadata.tsv file to $Neo4j/import folder before running the import query.
LOAD CSV FROM "file:///movie_titles_metadata.tsv" as row FIELDTERMINATOR "\t"MERGE (m:Movie{id:row[0]})SET m.title = row[1],    m.release_year = toInteger(row[2]),    m.imdb_rating = toFloat(row[3]),    m.no_votes = toInteger(row[4])WITH m, apoc.convert.fromJsonList(          replace(row[5]," ",",")) as tagsUNWIND tags as tagMERGE (mt:MovieTag{id:tag})MERGE (m)-[:HAS_TAG]->(mt)
In simplified terms, the lucene indexing pipeline consists of two steps. The first step is the “Analyzer” step, which takes care of the preprocessing of the text. In the second step, the “IndexWriter” stores the results of the “Analyzer” to the index.
In this blog post, we will be using the default “standard” analyzer in Neo4j. It tokenizes on non-letter and filters out English stop words and punctuation. Does no stemming, but takes care to keep likely product names, URLs, and email addresses as single terms.
You can check out other available analyzers.
CALL db.index.fulltext.listAvailableAnalyzers;
First, we have to transform all of the attributes, that contain numbers, to strings as we can’t index integers or floats. The indexing process silently ignores numbers.
IMDB rating attribute is a number with a single decimal point. To convert it to a string, we will first multiply it by ten and then convert it to a string. For example, rating 6.1 will be saved as “61”.
MATCH (m:Movie)SET m.string_rating = toString(toInteger(m.imdb_rating * 10)),    m.string_release_year = toString(m.release_year)
When using range queries, we have to be careful as the ordering is done lexicologically(alphabetically). In our case, all of the ratings are between “00” and “99”, so there should be no issue. If, for example, the range spanned to “150”, we would have an issue, as looking for intervals between “50” and “150” will not work by default. The workaround is described in Lucene range blog post. To allow range search between numbers with different count of digits, we need to preprocess the values by prepending zeros to them to ensure that all values have the same count of digits.
In this example, we preprocess all the numbers to have seven digits.
WITH 7 as total_lengthMATCH (m:Movie)WHERE exists (m.imdb_rating)WITH m, total_length,         toString(toInteger(m.imdb_rating * 10)) as string_ratingWITH m, total_length — length(string_rating) as zeros, string_ratingWITH m, apoc.text.join([x in range(1,zeros) | “0”],””) +                                      string_rating as final_ratingSET m.range_rating = final_rating
Now we create the “MovieIndex” index, which contains nodes with label Movie and the four specified attributes. The first parameter is the name of the index. The second parameter defines the labels of the nodes, and the third parameter the attributes we want to index.
CALL db.index.fulltext.createNodeIndex("MovieIndex",["Movie"],["title","string_rating","range_rating","string_release_year"])
Let’s take a look at some basic lucene query operators.
Specific attribute
Search for movies that have the word “dream” in their title.
CALL db.index.fulltext.queryNodes("MovieIndex", "title:dream") YIELD node, scoreRETURN node.title as title, score
Results
Logical operator
There are two logical operators available, “OR” and “AND”.
Search for movies that were released in 1999 or 2000.
CALL db.index.fulltext.queryNodes("MovieIndex",      "string_release_year:1999 or 2000") YIELD node, scoreRETURN node.title as title, scoreLIMIT 5
Results
Single-character wildcard
The single character wildcard operator ? looks for terms that match that with the single character replaced.
CALL db.index.fulltext.queryNodes("MovieIndex", "title:th?") YIELD node, scoreRETURN node.title as title, score
Results
One thing to note is that “the matrix” movie does not show up in results because the standard analyzer removes the stop words like “the”.
Multi-character wildcard
The multi-character wildcard operator looks for zero or more characters. You can also put the operator in the middle of the term like dre*am. To prevent extremely slow wildcard queries, a term should not start with the wildcard *dream .
CALL db.index.fulltext.queryNodes("MovieIndex", "title:drea*") YIELD node, scoreRETURN node.title as title, score
Results
Fuzzy search
Fuzzy search works by using mathematical formulae that calculate the similarity between two words. A commonly used method for calculating similarity is Levenshtein distance.
CALL db.index.fulltext.queryNodes("MovieIndex", "title:dream~") YIELD node, scoreRETURN node.title as title, scoreLIMIT 5
Results
Range query
Lucene differentiates between range operators with inclusive or exclusive ends.
Operators:
{} ->excluding edges
[] -> including edges
CALL db.index.fulltext.queryNodes("MovieIndex", "string_rating:[50 TO 99}") YIELD node, scoreRETURN node.title as title,scoreLIMIT 5
Results
To demonstrate why we need numbers with the same amount of digits when doing range search, let’s try the following query.
CALL db.index.fulltext.queryNodes("MovieIndex", "string_rating:[50 TO 100]") YIELD node, scoreRETURN node.title as title,score
Intuitively you would expect it to work just fine, but because of lexicological ordering, this is not the case. The workaround we used to solve this issue is the range_rating attribute, where values are prepended with zeros to allow alphabetical sorting between numbers with a different order of magnitude.
CALL db.index.fulltext.queryNodes(“MovieIndex”, “range_rating:[0000050 TO 0000150]”) YIELD node, scoreRETURN node.title as title,scoreLIMIT 5
Results
Lucene query language also supports boosting the score of the results with the caret (^) operator.
Boost results that have the string_rating between 50 and 99.
CALL db.index.fulltext.queryNodes("MovieIndex", "title:dream string_rating:[50 TO 99]^2") YIELD node, scoreRETURN node.title as title,scoreLIMIT 5
Results
Time decay
I saw a lucene time decay query in the Nodes 2019 webinar by Christophe Willemsen and wanted to replicate it. Basically, we combine many boosting parameters to achieve a time decay effect. To show what that means we will use apoc.text.join() to generate nine boosting parameters.
RETURN apoc.text.join([ x in range(0,8) |  "string_release_date:" + toString((date().year - x)) + "^" + toString(10-x)]," ")
Results
string_release_date:2019^10 string_release_date:2018^9 string_release_date:2017^8 string_release_date:2016^7 string_release_date:2015^6 string_release_date:2014^5 string_release_date:2013^4 string_release_date:2012^3 string_release_date:2011^2
As you can see movies with the release year 2019 will be boosted by a factor of ten, movies from 2018 will be boosted by nine and so on.
To wrap this blog post, let’s combine the rating-boosting and the time decay effect in a single query. And just so you know you can always manipulate results later with cypher, let’s add a filter that will return only thrillers.
WITH apoc.text.join([x in range(0,10) | "string_release_date:" + toString((date().year — x)) + "^" +     toString(10-x)]," ") as time_decayCALL db.index.fulltext.queryNodes("MovieIndex", "title:dream string_rating:[50 TO 99]^2 "+ time_decay) YIELD node, score// filter only thrillersMATCH (node)-[:HAS_TAG]->(:MovieTag{id:'thriller'})RETURN node.title as title,scoreLIMIT 5
Results
Full-text search is a very welcomed feature, that can help us develop better and faster Neo4j applications and also solve other problems. Join the Neo4j community forum if you have any questions or feedback.
All the code is available on Github.