Predicting your next query even before you type!
By: Flipkart Engineering
Tags: Search, Ecommerce, Backend, Startup, India

The CGF computes a relatedness score among pairs of candidate queries and ranks the destination queries for each source query based on a function of relatedness score. It operates in the following stages:

Finding related queries. Arriving at relatedness-score using blender. Ensuring candidate quality.

Finding Related Queries

We derive the related queries in two ways :

Immediate query to query reformulations in a user session Common (Extent of intersection between) products clicked on the search results page of two different queries: co-clicked products

Using immediate query to query reformulations

Representation: Probable destination queries for a source query

From the user journeys represented as a stream of events, we find the probability of a user reformulating to a next query using the ‘search’ events. For the ‘source query’ → ‘destination query’ pairs which have the same category intent, we compute the probability of reformulating to the ‘destination query’ from the ‘source query’. (Read more about Flipkart’s query understanding here).

Where R is the relatedness score of source query, destination query pair

Using co-clicked products

Common products being reached from different queries on Flipkart

Co-clicked products are an important signal of relatedness of queries as the users looking for the same set of products may express their intent through different queries. The extent of overlap between product clicks for different queries represents the extent of relatedness between the queries.

The relatedness score is a function of two probabilities :

Arriving at relatedness score with Blender

For the source query harry potter, we merge the candidates from the two sources as shown below, to arrive at a blended result with final scores.

While the current blending function weighs both sources equally, it is expected to evolve towards a weighted function with machine learnt weights.

Ensuring candidate quality

This phase eliminates poor quality queries to arrive at the final pool of related queries in the following steps:

Spell Correction

Often, queries in the actual user journeys are prone to misspellings, as the user may have written in a hurry, is unaware of the spelling, or may not be proficient with the spellings in the language. All query candidates go through Flipkart Search’s Semantic Spell Module (read more about it here) to rectify the misspelled queries.

Deduplication

We eliminate the duplicates or similar query suggestions. The current version of deduplication considers the following kinds of query duplicates :

Same stemmed tokens → ‘shirt for boys’ ~ ‘shirts for boy’

(We use porter stemmer here)

(We use porter stemmer here) Same character stream → ‘backpack’ ~ ‘back pack’

Among duplicate queries, we retain the one that has better relatedness score with the source query.

For example, in a source query, out of the 10 destination queries, if there are two duplicates, then we’ll retain only the one with a higher relatedness score from the two.

school bag → (backpack, 80), (back pack, 50), (barbie school bags, 40), .

Here, ‘back pack’ is eliminated from the list of destination queries as its score 50 < 80 (that of the corresponding duplicate query).

Performance Check

We fetch the historic performance numbers (click-through-rates, popularity, etc.,) of all destination queries and eliminate the ones that do not meet certain threshold values (based on historic distribution). This way, we ensure users see the apt search results which assist in the ‘buy’ decisions.

The following is a representation of the chosen related queries for a source query:

Representation of destination queries for a source query — Related Searches

The CGF module operates as a series of Spark jobs in our ETL pipelines that are orchestrated via Airflow and their frequency is determined based on freshness requirements of the result and how rapidly the input signals change. These jobs run over Flipkart’s Data Platform — FDP (read more about FDP here).

Understanding the need for Distributed Key-Value Store

The CGF operates as ETL pipelines with the output saved in HDFS. The scale of this data is about 10s of million rows. This section addresses how this data is made available for Autosuggest’s service to be consumed with each applicable request.

For data consumption by an application, we needed to transfer this data from HDFS to another data store that could support the scale of data size, rate of requests coming from Autosuggest, and the latency constraints.

We chose the data store on the following constraints:

Data retrieval should be fast enough to meet with Autosuggest’s latency constraint of single-digit milliseconds. The data store should be scalable in terms of the size of data it can store and requests per second (RPS) it can serve.

While in-memory alternatives such as Guava Cache can adhere to constraints of high RPS and low latency, these cannot scale for the size of data we get from the CGF. We thus concluded on using a distributed key-value data store which meets our needs on scalability, latency, and availability.

The distributed key-value data store is updated with the latest data from the CGF periodically. Let’s find out how this data is employed to render the ‘Discover More’ suggestions in Autosuggest.

Personalising ‘Discover More’ in Autosuggest

Consider a user who has searched for — blender, Harry Potter and Fitbit, in the recent past. When the user opens Flipkart app and searches, we’ll show her the following suggestions: