E-commerce Search Re-Ranking as a Reinforcement Learning Problem
By: Atul Agarwal
Tags: Search, Machine Learning, Ecommerce, Search Engine Optimizati, Reinforcement Learning

Search as a term is overloaded: it describes both the desire to find something and the process for doing so.

Any Search Workflow has three main parts to it:

Search Pre-Processing: It involves the entire workflow of query understanding. It goes from language identification → character filtering → tokenization → spell correction → stemming and lemmatization to query re-writing (expansion, relaxation, segmentation, scoping). After query understanding, the query is passed to a search engine for information retrieval and ranking.

Information Retrieval & Ranking: It involves retrieving information(documents) indexed in the search engine and ranking them based on the query passed to the search engine.

Search Post-Processing: It involves re-ranking of the search results retrieved from the engine based on external signals, and displaying the search results.

Search Post-Processing — Re-ranking based on Clickstream data

One of the most readily available external signal/feedback for search is the user click data. We try to model the user clicks as a function of relevance and re-rank the search results.

An aggregated and processed clickstream data(time and position of click normalization) would be of the form:

Before we deep dive into methods of re-ranking, let us define some terms used in the article.

Deterministic Process: refers to the process where no randomness is involved in the future state of the system. If we start with the same initial state deterministic model would always give the same results.

Stochastic Process: refers to a randomly determined process. Stochastic optimization generates and uses random variables.

Markov Decision Process: refers to a discrete-time stochastic control process. It provides a mathematical framework for the decision-making process where outcomes are partly random and partly controlled by the decision-maker. It contains the following:

A Markov Decision Process(MDP) is a tuple M = (State Space, Action Space, Reward, State Transfer Function, Discount Rate). The objective of the MDP is to find a policy the maximizes the expected accumulative reward starting from any state s. If the reward is unknown, it is a reinforcement learning problem.

Reinforcement learning focuses on finding the balance between exploration (of uncharted territory) and exploitation (of current knowledge).

MDP follows Markov Property, meaning that the state is able to summarize past sensations compactly in such a way that all relevant information is retained and thus future state depends upon the current state only.

Multi Arm Bandit: is a special case of MDP where there is only one state. The term “multi-armed bandit” comes from a hypothetical experiment where a person must choose between multiple actions (i.e., slot machines, the “one-armed bandits”), each with an unknown payout. The goal is to determine the best or most profitable outcome through a series of choices. At the beginning of the experiment, when odds and payouts are unknown, the gambler must determine the machines to pull, its order, and the number of times.

Difference between MDP and Multi Arm Bandit

To differentiate between MDP and multi-arm bandit, we take the restaurant example: We may often go to our favorite restaurant and try some new sometimes to find a potentially better one. If our preference for every restaurant never changes, the bandit is a very suitable formulation. If our preference changes to Chinese food after having Japanese food, then modeling and exploring the state transition is more suitable, hence MDP.

Coming Back to the re-ranking problem

Learning to Rank methods are the gold standard for search result re-ranking based on external feedback. They work on the principle of optimizing NDCG or MRR or MAP over annotated data. These methods have two significant disadvantages:

It requires a massive amount of annotated data to train, and the introduction of new documents/products in the index/catalog further complicates things.

It does not explore new documents/products before having substantial feedback, thus introducing a bias.

Here comes reinforcement learning to take care of the above two problems.

In the search domain, if we had feedback data for an infinite amount of time, we could go with the frequentist approach. Still, since that is not possible, we always update our prior beliefs with current knowledge(Bayesian approach).

We can model search sessions as MDPs where the flow would go like search query 1, add to cart, search query 2, add to cart,…., checkout. There is a fantastic paper that describes this method. We take a more straightforward approach where we model the clicks for individual query(single state), i.e., Multi-Arm Bandit.

Search Re-Ranking as a Multi-Arm Bandit Problem

We treat the search results retrieved for a search query q as distinct arms and probability of clicking/adding to cart any result as the chances of pulling the arm. The reward, in this case, would be to maximize the total number of clicks/add to cart for the search query q.

There are multiple ways to solve the Multi-Arm Bandit problem. We analyze different ways in which it can be implemented.

1) E Greedy

We choose an epsilon, ε amount of user to show random results(pull random arms) while showing the best(pulling arm with the best score) for the rest of them. This helps us to explore all the possible combinations. The main challenge here is to choose the correct ε. If it is too small, we won’t be able to explore all the arms/products. If it is too large many users would find themselves with poorly ranked results.

2) Policy Gradient

In this method, we choose a policy, e.g., softmax policy, and the arm is chosen randomly from the softmax distribution. We maintain an average reward for each arm/document/product, and the gradient agent moves in the direction of maximum rewards. It would have been an ideal method if:

All the products/arms were introduced since the beginning of the observation &

There were no new entries in the environment(the search result page).

3) Upper Confidence Bound(UCB)

This method is based on Optimism in the Face of Uncertainty Principle. We take the upper bound of the confidence interval and choose products/pull arms accordingly. If we can update the table mentioned above in real-time, this is one of the best methods.

4) Thomson’s Sampling

One of the most effective approaches to the multi-arm bandit, which takes into consideration the probability distribution of each arm(aggregated table figure 1). The basic idea here being that from each arms tuple(good_samples, total_samples), a confidence interval can be generated, and a random value can be given between LCB and UCB.