Getting Started with Elasticsearch Query DSL | by Niels D. Goet | Towards Data Science

In this post, I’ll introduce the basics of querying in Elasticsearch (ES). We’ll look at how queries are structured (e.g. the filter vs. query context, and relevance scoring) in Elasticsearch Domain Specific Language (DSL) and apply them with the Python Elasticsearch Client. (And, if DSL makes your head spin, skip to the final section of this post, where we’ll go through the basics of running SQL queries against ES). All code used in this blog post is available in this GitHub repo.
For this post, I’m assuming you are familiar with the basics of ES. I’m also assuming you have already provisioned and deployed your own Elasticsearch cluster (either “from scratch” or using a managed service), and that you’ve written data to an index.
For a quick primer on the basics of ES and instructions on how to provision a cluster, and set up an index with Python, check my earlier post on Creating and Managing Elasticsearch Indices with Python.
towardsdatascience.com
For the examples shown below, I’ll be using the neflix_movies index I set up in my earlier post. This index is constructed from data on 7787 Netflix shows available on Kaggle. The original data is in CSV, and contains information on movies and series available on Netflix, including metadata such as their release date, their title, and cast. The mapping for our index is defined as follows:
This mapping can be further improved and is not optimised for disk usage or tuned for search speed. However, it’ll suffice for the queries that we’ll be running below.
If you’d like to learn how to create better ES indices that are optimised for disk usage, check out my earlier post on Optimising Disk Usage in Elasticsearch.
towardsdatascience.com
Before we dive into the hands-on sections of this blog post, let’s review some of the basics of querying data from Elasticsearch. The ES search API accepts queries that use Elasticsearch Domain Specific Language (DSL), which is based on JSON. The ES documentation describes DSL as an Abstract Syntax Tree (AST) of queries that consists of two types of clauses:
leaf query clauses that look for a specific value in a specific field (e.g. a match or range); andcompound query clauses that are used to logically combine multiple queries (such as multiple leaf or compound queries) or to alter the behaviour of these queries.
leaf query clauses that look for a specific value in a specific field (e.g. a match or range); and
compound query clauses that are used to logically combine multiple queries (such as multiple leaf or compound queries) or to alter the behaviour of these queries.
When you run a query against your index (or indices), ES sorts the results by a relevance score (a float) that represents the quality of the match (the _score field shows its value for each “hit”). An ES query has a query and a filter context. The filter context — as the name suggests — simply filters out documents that do not match the conditions in the syntax. However, unlike the match in the bool context, it will not affect the relevance score.
Let’s look at a simple example of a query with a query and a filter context. The example below filters on release year (using a range query), and runs a match query against the movie type in the query context.
{    "query": {        "bool": {            "must": [                {"match": {"type": "TV Movie"}},            ],            "filter": [                {"range": {"release_year": {"gte": 2021}}}            ]        }    }}
We can run this query on our index directly using our ES Console (assuming you’re using a cloud-based platform such as Elastic.co, Bonsai, or Amazon Elasticsearch Service), or through the Python Elasticsearch Client. The queries themselves look the same regardless of whether you use the client or the console. In the remainder of this post, my examples will use the Python ES Client with DSL.
To get started, we first set up the ES client connection (for further details, please see my earlier blog post and this GitHub repo):
In a second step, we define and run our query against the netflix_movies index:
The print statement at the end of the code prints the title and the associated relevance score as tuples. The output looks like this:
[ ('Bling Empire', 0.96465576), ('Carmen Sandiego', 0.96465576), ('Cobra Kai', 0.96465576), ('Disenchantment', 0.96465576), ('Dream Home Makeover', 0.96465576), ("Gabby's Dollhouse", 0.96465576), ('Headspace Guide to Meditation', 0.96465576), ('Hilda', 0.96465576), ('History of Swear Words', 0.96465576), ('Inside the World’s Toughest Prisons', 0.96465576)]
In this case the relevance scores are the same for all entries: this makes sense, since all of these entries have the exact same value for type (that is, “TV Movie”). Since we know that “TV Movie” is the value that the field can take, we may as well have used the filter context, since we’re looking for that exact value.
So when should you use the filter or query context? In general, filters are cheaper and should be used whenever you can (ES caches frequently used filters automatically to boost performance). They are used to search on binary cases (that have a clear “yes” or “no” answer) and on exact values (for example when searching on a numeric value, a specific range, or a keyword). The query context, in turn, should be used when the result can be ambiguous or for full text search (i.e. when searching analysed text fields).
Let’s revisit the query shown above with this information on when to use the filter and query contexts in mind: The query applies to a range of a short field type (release_year) and a keyword field type (type). For the latter, we know the exact value we want (“TV Show”). A better way of writing the query therefore is as follows, re-using the range filter, and applying a term query (more on this type of query below) for the latter, and placing both in the “filter” context:
Now that we’ve covered the basics of search and DSL, let’s first look at several basic queries.
First up is the “ match” query, which is the default option for full-text search. Let’s assume we’re looking for metadata on the “House of Cards” series. Because ES analyses the text when using a match query, it will return any show that has “House” or “Cards” in the title if we go with the default (DSL defaults to the “or” operator). Therefore, we’ll use the “and” operator:
The query produces a single hit (one document):
{  "total": {    "value": 1,    "relation": "eq"  },  "max_score": 15.991642,  "hits": [    {      "_index": "netflix_movies",      "_type": "_doc",      "_id": "rWHgLHgBIp2yHscf16v-",      "_score": 15.991642,      "_source": {        "show_id": "s2833",        "type": "TV Show",        "title": "House of Cards",        "director": null,        "cast": "Kevin Spacey, Robin Wright, Kate Mara, Corey Stoll, Sakina Jaffrey, Kristen Connolly, Constance Zimmer, Sebastian Arcelus, Nathan Darrow, Sandrine Holt, Michel Gill, Elizabeth Norment, Mahershala Ali, Reg E. Cathey, Molly Parker, Derek Cecil, Elizabeth Marvel, Kim Dickens, Lars Mikkelsen, Michael Kelly, Joel Kinnaman, Campbell Scott, Patricia Clarkson, Neve Campbell",        "country": "United States",        "date_added": "November 2, 2018",        "release_year": 2018,        "rating": "TV-MA",        "duration": "6 Seasons",        "listed_in": "TV Dramas, TV Thrillers",        "description": "A ruthless politician will stop at nothing to conquer Washington, D.C., in this Emmy and Golden Globe-winning political drama."      }    }  ]}
The relevance score for this hit is 15.991642. This may seem odd at first glance. However, keep in mind that the relevance score is not bounded and that it is used to evaluate the relevance of a document versus all others that match the query. In other words: the score represents how relevant a document is for a query compared to other “hits” (documents) for that same query, in the same search.
The term query is used to “find” documents for which a field matches an exact value. We have a number of keyword fields in our index that we can apply a term query to. The example below extracts five documents where the type field is equal to TV Show (note that the size parameter controls how many records ES returns, and it defaults to 10):
A term query should not be used for text fields. ES changes text fields as part of analysis, tokenising the text, stripping it of punctuation, and converting it to lowercase. If type had been a text field instead of a keyword field, the query shown above would not have returned a result because any instances of “TV Show” would have been turned into ["tv", "show"] as part of analysis.
Last but not least in our “common queries” series: the “range query”. As the name implies, the range query is used to find any documents for which a specific field’s value falls within a defined range. It can be used for, e.g. date and numeric field types. It can also be used on text and keyword fields, provided your cluster settings allow for expensive queries (the default). The query shown below runs a simple range query on release year, searching for 100 movies or TV shows released in 2012 or earlier:
There are several more complex queries that I tend to use. In this section, we’ll take a brief look at regex, metric aggregations, and custom filters.
The regexp query is designed for finding documents where a term in a specific field matches a regular expression. The example below looks for any documents in our netflix_movies index where the title field includes a word that in part consists of “war” (e.g. war, warror, Warfare, warmest, etc.):
Regexp queries can be made case-sensitive (using the case_insensitive option, with default set to false). However, title is a text field for which analysis has not been disabled. Therefore, we can expect the terms in the title field to have been lowercased (among other things — please refer to the previous sections for more details on “analysis”). Enabling or disabling case sensitivity would therefore make no difference given our current mapping.
ES aggregations are a great way to explore your data, and they come in three flavours: metric, bucket, and pipeline. Here we’ll look at one from the metric and one from the bucket category.
We’ll first take a look at the stats query (a metric aggregation), which comes in handy when you’re looking to review summary statistics (min, max, mean, sum) for numeric values. For example, if we’d quickly want to see how release_year is distributed in our data, we can run the following query:
For aggregations we set the size parameter to 0 in our queries. This means that the search API will not return any documents (if you omit this, the hits field will contain the documents). The response for this query gives us simple summary stats for the release_year values across our documents (e.g. min, max, mean):
'{  "release_year_stats": {    "count": 7787,    "min": 1925.0,    "max": 2021.0,    "avg": 2013.932579940927,    "sum": 15682493.0  }}'
Let’s now turn to our second aggregation query, this time from the bucket aggregation family. The terms aggregation is useful for getting a quick overview of the availability of observations across different categories. In our netflix_movies index, you may for example want review how many movies were released in any given year, or how many movies of each type are present in the dataset. In DSL, our query for the former (count of movies per release year) looks as follows:
There’s two things worth highlighting in this query. First, we specify a second size parameter in the terms object, which controls how many “term buckets” are returned (in this case: 5). Second, we order the output by the keys (i.e. the release years), in ascending order. The output from this query should look like this:
{  "release_years": {    "doc_count_error_upper_bound": 0,    "sum_other_doc_count": 7775,    "buckets": [      {        "key": 1925,        "doc_count": 1      },      {        "key": 1942,        "doc_count": 2      },      {        "key": 1943,        "doc_count": 3      },      {        "key": 1944,        "doc_count": 3      },      {        "key": 1945,        "doc_count": 3      }    ]  }}
Let’s quickly review this output. The buckets fields include the document count for each of our five term buckets (one object for each key). In turn, thesum_other_doc_count is the sum of all counts for the buckets that were excluded from the output. Finally, the doc_count_error_upper_bound value reflects the error associated with the bucket aggregation. At this point you might wonder: “error, what error?”. ES reports an error in the response because term aggregations are approximate. This stems from the fact that documents are distributed across shards, and each shard has its own ordered list of terms, which are combined in the aggregation.
The node that coordinates the search process requests a number of term buckets equal to size. Subsequently, the results from these shards are combined to produce the final response. We can therefore expect a greater chance of error if the specified size is less than the number of unique terms. In our example above, we have 73 distinct release_year values, so we may want to change the size parameter (and for large numbers of term buckets, the shard_size too, to address some of the additional overhead that stems from increasing size).
Onto our final “complex” query: the “script query”. A script query can be used if you want to use a filter of your own design on documents in your index. The script query is provided in the filter context. In the example below, I’ve written a simple script that extracts documents for which release_year is greater than 2018 (we could achieve the same result with a range query, see above).
The source field contains the script, and the params field includes the start_year value that is used by the script. The default language for script queries is “painless” (specified in the lang field), but other languages are available, including Lucene’s Expressions Language and Mustache Language. If you have custom scripts that you use often, you can use a store scripts on the cluster state using the _scripts endpoint, so you can retrieve them for later use. Storing scripts speeds up search since it often reduces the time needed for compilation.
For the SQL fans out there: you can use SQL syntax to query your ES indices. This feature is part of X-Pack and you can use to directly execute SQL queries against your ES indices in the console on your cloud-based ES service, such as AWS or Elastic.co, or using the Python Elasticsearch Client. (Note that this does not work on Bonsai as the X-Pack plugin is not included because it requires a license for commercial purposes. I’ve therefore run the below API calls for the SQL translate API on Elastic.co instead, which offers a 14-day free trial).
If you work on Elastic.co, you can use the SQL translate API to translate SQL syntax to native Elasticsearch queries. From the console, it’s as simple as using a POST request with your SQL syntax in a JSON document. For example, a query to get a list of titles of Netflix movies released in 2015 or later looks as follows:
{"query": "SELECT title FROM netflix_movies WHERE release_year >= 2015"}
If you send this document using a POST request on the translate endpoint (POST _sql/translate) on your API console, it will return the corresponding query in native a native ES query (DSL). For our example, the resulting ES query is:
{  "sort": [    {      "_doc": {        "order": "asc"      }    }  ],  "query": {    "range": {      "release_year": {        "to": null,        "include_upper": false,        "boost": 1,        "from": 2015,        "include_lower": true      }    }  },  "_source": false,  "fields": [    {      "field": "title"    }  ],  "size": 1000}
You can also get the output directly, skipping the “translate step” altogether. If you make a POST request with your JSON document with _sql?format=txt, the API returns the response data in a nicely formatted text format (for JSON format, use _sql?format=json):
title                                ------------------------------------- Black Crows                                                                     Black Earth Rising                                                              Black Lightning                                                                 Black Man White Skin                                                            Black Mirror                                                                    Black Mirror: Bandersnatch                                                      Black Panther                                                                   Black Sea                                                                       Black Site Delta                                                                Black Snow...........
Similarly, you can run simple GROUP BY queries (provided the field you perform the aggregation on is not of the text type). For example, the following query will return the number of movies for each release year (here I’m limiting the output to the first five years):
POST _sql?format=txt{"query": "SELECT release_year, COUNT(*) as n_entries FROM netflix_movies GROUP BY release_year LIMIT 5"}
Again, the output is nicely formatted:
release_year  |   n_entries    ---------------+--------------- 1925           |1               1942           |2               1943           |3               1944           |3               1945           |3
And that’s it! We’ve now reviewed the basics of DSL and have run a couple of basic and more complex queries.
Thank you for reading!
Support my work: If you liked this article and you’d like to support my work, please consider becoming a paying Medium member via my referral page. The price of the subscription is the same if you sign up via my referral page, but I will receive part of your monthly membership fee.
If you liked this article, here are some other articles you may enjoy:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Disclaimer: “Elasticsearch” is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. Description and/or use of any third-party services and/or trademarks in this blog post should not be seen as endorsement for or by their respective rights holders.
Please read this disclaimer carefully before relying on any of the content in my articles on Medium.com.