Improving search relevance and engagement with text attributes
By: Pinterest Engineering
Tags: Elasticsearch, Search, Infrastructure, Query, Engineering

Matthew Fong | Pinterest engineer, Discovery

Nearly two years ago, we launched Guided Search, a way to search for questions with more than one right answer, and have been making improvements ever since. Pinterest Search was built on top of Lucene, utilizing basic functionalities, such as normal text fields and payload fields, without many major customizations. However, we learned it’s increasingly important to store and differentiate text from a wide variety of sources in order to more accurately rank the retrieved Pins. After iterating through many possible solutions, we designed and implemented a new field, attributed_text, which allows us to store attributes like source types and morphological forms for each piece of text.

Motivation

The goal of any search engine is to have both high precision and high recall at an acceptable latency. Over time, we’ve made notable changes to our index to address latency and recall issues.

While it’s easy to add a new text source when a search index is small, we needed to find a scalable solution as the number of sources grew. We created a field called merged_text which bucketed all text annotations, but it wasn’t able to differentiate between annotation sources at scoring time and made it difficult to experiment with new text sources. We also used stemming, which dramatically helped recall but hurt precision. For example, a Pinner searching for “animals” would also match documents that just had the token “animal,” but searches for “bowling” and “bowl” would both retrieve the same set of documents. Because we wanted to maximize both recall and precision, our goal was to design a solution that provided us with better text ranking signals and a similar level of recall.

An exploration of solutions

The first idea was to evaluate the cost of separating our merged_text back into several fields by having different fields for certain sources or morphological transforms. We tried the simplest case, comparing the performance of searching two fields to just searching one. When we tested one machine without any optimizations, we found the median latency of two fields is 2.5x the latency of just one, while the ratio of average latency is 1.5x. Because there’s such a big discrepancy when using two fields, we decided this wouldn’t be the best approach, no matter what optimizations we could make.

Another idea was using just one field, but storing different morphological forms of the same token within that field. For example, with the token “animals” we would store “animals” (exact), “animal” (lemmatized), and “anim” (stemmed). We can then indicate which type of match that token is in its payload. Although this solution is relatively easy to implement, it also comes with drawbacks. For instance, searching and scoring becomes much more complex since we have to look at multiple posting lists and position indexes, and computational costs would probably double.

This third solution is our actual design. We use the payload for the match type as well. Although it seems like there are too many possible morphological transforms to represent them efficiently in the payload, we solve this problem by using perfect hashing. For each stemmed token, we assign a hash value to every possible unstemmed form so there are no possible collisions. Before indexing, we create a dictionary that has entries such as the following:

We then use this dictionary at indexing time to store the hash value (which is the index of the term in the appropriate stemmed token list). We access the same dictionary at serving time so we can compare the exact form of the matched token to the exact form of the query token. Because the largest unstemmed token list has fewer than 150 entries, we can store this value in just one byte and are able to store both the source and the exact form in just two bytes. (Fun fact: the top 5 longest token lists are decor, fashion, design, inspir and color.)

You can see an example of the entire indexing flow below in Figure 1, including what the actual payloads look like. With a custom SpanQuery and Scorer, we’re able to apply any type of multiplicative boost to documents that match specific types of annotation source as well as stemmed or unstemmed match of queried token.

Figure 1. Sample flow for how we populate attribute text for one document

Looking ahead

These upgrades had their fair share of obstacles. The first issue we ran into involved computational latency. As we were now doing more work for each search, our serving machines had to work harder and were struggling to handle the load of our peak traffic times. We’re continuing to make the backend Lucene code more efficient for our specialized use of payloads by, for example, implementing lazy scoring for multiple-term queries. Instead of scoring each payload as we scan the posting list, we’ll wait until the end to score the payloads for the documents that match all of the terms of the query. With this optimization, we saw a 28 percent improvement in p99 latency.

These changes also unlock many more possibilities towards providing a better, more accurate search service. We can implement additional optimizations, such as requiring all query tokens to come from the same annotation source or be within a certain distance from each other. We can also have a more nuanced form of scoring for exact match vs. stemmed match for each token. For some words, such as “bowling,” there’s a big difference between it and its stemmed form “bowl.” For others, such as “men’s,” there’s little to no difference at all with “men”.

Attributed text in action

We’re still experimenting with this approach, and we’ve seen strong gains in both relevance and engagement in early tests. Along with a significant increase in the number of repins per search, we also saw relevance improvements, as shown in the side-by-side comparison below.

Figure 2. Improvement for the search query ‘anime art ideas’ from using attribute text.

If you’re interested in solving similar problems in search infrastructure, search ranking or query understanding, come join us!

Acknowledgements: Dong Wang, Xin Liu and Rui Jiang for their technical guidance and feedback.