Just in Time Personalization — SlasnN 2018
By: Flipkart Engineering
Tags: Search, Slashn, Data, UX, Data Science

Introduction

With the growing popularity of online shopping, personalizing the choice of products displayed to assist the customer’s needs in real time, is the need-of-the-hour.

Vaibhav Srivastava and Arun Agarwal explain the concept of just-in-time personalization along with a glimpse into the technicalities of how Flipkart makes it possible.

What is Just-in-time Personalization?

A typical shopping activity online sometimes involves users who just know what they are looking for in the most generic terms. For instance, someone who is looking for a white car.

If we were to show results pertaining to the broad spectrum search term “White Car”, the statistically popular, or the most sold out white cars surface as results. This way, we will do justice neither to the overflowing product inventory nor the user’s preferred choice.

Shopper personas

The popular cars need not be the cars that the user seeks to buy often, because of the varied shopper personas:

Youngsters with varied tastes and interests having a huge purchase power: These users are lesser in numbers in the customer base, but they are the ones who do would go for expensive purchases with no restrictions on the price.

Professionals with common tastes and interests having a limited purchase power: Majority of users belong to this category. They are the ones who would base their purchase on the most popular products and have a budget to adhere to.

Executives with refined taste and enormous purchase power: These users are in moderate numbers with too little time for shopping and have a strong affinity towards Brands and Price. They are the ones who need a carefully curated catalog to strike a purchase immediately with fewer restrictions on the price.

That said, it is imperative for every Flipkart webpage user to see a personalized list of products that suit the need-of-hour or real-time. The same search string “White car” shows a different set of products intelligently chosen to suit the shopper persona’s shopping intent just then. This is just-in-time personalization and is of utmost priority in any e-commerce application.

How did we think?

Considering the varied user personas that wear the shopper’s hat with varied spending abilities, our search functionality aims to present the top 20 product choices curated based on the common affinities — Store, Brand, Geo-sensitivity, and Gender.

With the Structured Catalog, User’s Historical Activity such as Navigation on Flipkart, Product Page Views, Card Clicks, Purchases, Payments, and the User Persona aspects we track, the Search and User Insights team created clusters of users. For every store, we have defined N buckets and mapped users to those buckets.

Few Complexities

Unpredictable spending behavior in every user persona : People spend a high price for brands they are interested in, even if they exhibit frugal spending behavior in a new upcoming brand in the market.

: People spend a high price for brands they are interested in, even if they exhibit frugal spending behavior in a new upcoming brand in the market. Varied intent and Varied people : People browse the product catalog intending to buy for self or family, which means catering to the user of the product. For example, a user shopping for Nike shoes may also browse gifts for a girl child.

: People browse the product catalog intending to buy for self or family, which means catering to the user of the product. For example, a user shopping for Nike shoes may also browse gifts for a girl child. Every search or browse session may not lead to a purchase: People check the availability of products but do not always purchase immediately.

Need-of-the-hour

Despite all the complexities that exist, we must find out for whom the current browsing session is all about, to render or recommend the choicest products suitable to the user persona on the search pages.

Let’s consider taking into account, the historical user activities as recorded in various systems and running the data sets through trained ML algorithms to arrive at the user persona’s present need. With the volume of data processed for this activity, the time taken for rendering the informed product listing was longer and hence these results were no longer apt for the search-of-the-hour.

Conclusively, for the personalization to be useful, just-in-time and comprehensive for the current session, we need to understand the user view in Flipkart system and the user preferences based on the long-term history of session affinities.

Understanding User View and Session Affinities

Understanding the user personas and their needs as soon as they start a session with the Flipkart website requires the stitching of a user account with all the data pertaining to Demography, Behavior, Live Aggregates and Data, just-in-time.

Multiple systems across Flipkart derive information from these user views for their requirements.

For every user session, based on the user persona, the long-term history, price sensitivity, user preferences and the most popular products pertaining to the search, it is possible to find out the user’s affinity for display of products on the webpage. Among many affinities we track, the most judgemental affinities that can provide an insight into the user personas’ requirement are Brand, Store, Gender, and Geo-sensitivity.

Translating the Challenges to Capabilities

Wading through huge data sets to arrive at the just-in-time personalized set of products for display, brings forth a multitude of challenges around handling voluminous data, faster compute requirements, near real-time predictions, low latency outputs. We translated these challenges to the following capabilities:

Derive useful information from historical data: Gender, Brand are examples.

Conclude real-time intent of the user: Female looking for shoes is an example.

Scale up the prediction service in real-time as the search cannot take a long time to display predicted product choices. Example: Type n. Nike, Nikkon comes up based on past browse/purchase history.

Train ML models to ensure that the predictions are right.

While there are more and more capabilities needed in this space, we have started with these capabilities. It is challenging to ensure due attention and personalized search results for every user, where about 100 million products are available for a user base of over 100 million with both factors growing exponentially.

Engineering the J-I-T Personalization

Just before the Just-in-Time personalization came into being, MR jobs computed affinities. The batch processing results although correct, took a few hours to show up. The results were no longer fresh and relevant to the current user search session.

In a user session, there may be spikes in the search based on the change of intent. For example, the same user may look up an iPhone for self and Redmi Note consecutively for a family member. In batch processing, when session level aggregations happen, such minor yet significant spikes may go unnoticed.

The following illustration represents the placement of components and workflow that addresses the aforementioned issues:

In the re-engineered design, the following are the key points to note:

fStream platform with abilities to perform joins over large windows, calculate time-partitioned aggregates and record data over a time range.

Inputs from ML platform on previous data and user interactions to verify the predictions.

Many user insight ‘Writes’ to the Live data store.

The following system illustration displays the technical stack that makes it all possible: