Evalcat — a library for the aggregation of search metrics
Tags: Search, Evaluation

Evaluating search systems

Building search systems have been made simple with the availability of open-source search engines like Elasticsearch and Solr. However, comparing the effectiveness of search systems when trying to make improvements can often be a laborious task. It usually consists of multiple steps of offline and online evaluations, including relevance assessments, log analysis and A/B testing.

There is often a step in search evaluation that precedes the aforementioned, and that is to examine the search results of a set of queries. This can be done by comparing the aggregated metrics of two search systems, typically the current system and an improved one. While this has no impact on any relevance judgements, it can be useful as a sanity check to monitor the changes between systems.

Introduction

Over the past year I interned at a Tokyo-based startup called Libora, where I worked on their identically-titled product, a search engine for online courses. After the completion of the search engine, the next step was to improve on the search effectiveness. As I made changes to the search algorithm, I realized that I was constantly reusing code to compare and evaluate the previous and new search results. As such, we decided to extract the functions and package them in a library evalcat .

Evalcat

We aim to provide a library where the search results can be aggregated and compared across systems and queries. Search results are aggregated by field values. For example, if the search results contain a price field, we can compute the mean and percentile values for the results. Comparisons can be performed across systems or queries and the aggregated metrics are returned in a pandas DataFrame. Utility functions like ranked-biased overlap can be used to compare search systems across queries.

Evalcat was created when we wanted to understand the changes in search results when experimenting with different search algorithms. We originally started from tabulating and visualizing Elasticsearch aggregations, but later extended it to be able to compute custom metrics.

Using evalcat

Let’s take a look at an example.

The search results of two systems are given in the following format, where the query orange is mapped to a ranked list of search items each containing the fields id and price .

search_results = { “system A”: {

“orange”: [{“id”:1, “price”:1.3}, {“id”:2, “price”:2.5}],

}, “system B”: {

“orange”: [{“id”:1, “price”:1.3}, {“id”:3, “price”:0.9}],

}, }

To retrieve metrics for the price field, we instantiate a NumericalField object indicating that price is a continuous variable. This class provides methods to compute the percentiles and mean values for that field.

>>> price_field = NumericalField(“price”)

The main class of evalcat is ResultList . By passing the search results and the field as arguments, we can view the computed metrics for the query “orange” using the get_system_metric_df() method.

>>> result_list = ResultList(search_results, [price_field]) >>> result_list.get_system_metric_df(‘price’, ‘orange’) +----------+---------------+---------------+-------+-------+

| | 25-percentile | 50-percentile | total | mean |

+----------+---------------+---------------+-------+-------+

| system A | 1.6 | 1.6 | 3.8 | 1.9 |

| system B | 1.0 | 1.1 | 2.2 | 1.1 |

+----------+---------------+---------------+-------+-------+

While this may be a simple example, the point is that search systems can be compared easily over defined metrics. The Field class is intended to be extended to provide custom definitions of metric computations.

Future additions

Evalcat is very new and currently lacks some functionality and refinement, but the plan is to continually add new functionality over time. At the time of writing, the next step is to implement a connection with Elasticsearch to allow for a pipeline from search algorithm to the calculated metrics.

We encourage anyone using this to file for bugs or feature requests.