Ranking Ads with Machine Learning
By: Chandan Routray
Tags: Search, Machine Learning, Learning To Rank, Search Ranking

Ranking Ads with Machine Learning

Introduction

OLX helps people upgrade their lives by making it super easy for anyone to buy or sell almost anything through our platforms. We operate in more than 40 countries with varied demographics, different languages and usage patterns.

Search is an integral part of any e-commerce website and one of the most important user journey towards the conversion. At OLX, search generates about 80% of total conversions and we strive to deliver the best search experience to the user. In this blog, we are going to talk about how we improved search results rankings using machine learning and we hope this could serve as a starting point for many companies who are starting their journey in search rankings.

Search: Recall and Ranking

The search functionality on any platform usually works in the following way, user enters a query(search string), might apply some filters like brand of the product, price range etc. This information is explicit and usually is not relaxed, so for e.g. if the user is looking for “iphone” in Cape Town, inside the “Cell Phones” category, we end up with pseudo query like:

This query hits our search engine and depending upon the query and filters we get back anywhere from a few to over ten thousands of relevant results, this is what we call “Recall” of the query(You can read more about how we improve the recall here). Now, in what order should we present this to our user as all of them match the user query, this is what we called the “Ranking” problem, i.e. given a list of items and user query, how should we rank the list of items such that we have the most relevant results at the top.

Recall vs Ranking

Structuring the project

Usually a machine learning project is structured in these following steps:

Defining a baseline approach : Deciding a decent baseline that your ML solution should outperform. It could be a previous solution to the problem or a solution based on business intuition

: Deciding a decent baseline that your ML solution should outperform. It could be a previous solution to the problem or a solution based on business intuition Formalising the ML problem : Transforming the business problem into a ML problem for e.g. whether its a classification or regression, defining the target variable, the objective function, losses etc.

: Transforming the business problem into a ML problem for e.g. whether its a classification or regression, defining the target variable, the objective function, losses etc. Dataset and Labeling: Generating a labeled(if required) dataset for training and evaluation.

Generating a labeled(if required) dataset for training and evaluation. Feature Engineering and Modelling: Creating the right features for training the model, cross validation, hyper-parameter tuning etc.

Creating the right features for training the model, cross validation, hyper-parameter tuning etc. Serving: Building infrastructure to serve the ML solution that meets certain performance requirements

Building infrastructure to serve the ML solution that meets certain performance requirements Evaluation and Experimentation: Defining metrics for evaluating the model offline as well as online in the A/B testing phase

Lets go step by step through each of these as we move ahead.

Defining a Good Baseline

In online classifieds, one of the important factors for conversion are:

Proximity of the item: Measure of closeness between user’s selected location and item’s location

Freshness of the item: Measure of how recently the item was listed on the platform.

These are also some of the sorting options that a user gets on our platform. We know from our past experience, that ranking by proximity or freshness is better than ranking randomly. So, an easy and intuitive step that we did in the past is to linearly combine these two factors to create a weighted score which we used to rank the items:

score = ⍺ * proximity + β * freshness

Now the question arises, how did we get the weights ⍺ and β. A simple approach would be to launch multiple A/B tests to see which weights are performing the best, a simple metric like Conversions per Search can be used to measure the performance.

Great! so why do we need machine learning if we can do this ?

There are multiple drawbacks of this approach like what if there are some combinations of weights that we didn’t explore, or can the score be better represented with some non-linear function of the factors, or what if we want to add other factors. And also, A/B test usually takes days and with many factors this could take forever to get decent weights.

This already tuned weighted formula served as our baseline and also the starting point. For the first version, we decided to include some new factors and learn these weights with historical search log data.

Formalising the ML problem

In a broad way, a ranking or learning to rank problem can be formulated as,

There are three popular approaches that are used across academia and industry to convert this to a learning problem, they are classified depending on how many items they consider in their loss function:

Point-wise approach: The algorithm learns by measuring loss over a single item, which is usually the difference in the actual relevance score and the predicted relevance score of the item. It reduces the ranking problem to classification or regression problem.

Pair-wise approach: The algorithm learns by measuring loss over a pair of items, which is the difference in the actual relative order and the predicted relative order of the two items. It reduces ranking to a pairwise classification problem.

List-wise approach: The algorithm learn by directly minimising the IR measures like NDCG or MAP.

In practice pair-wise methods have been found better than point-wise methods because predicting relative order is closer to the nature of ranking than predicting class label or relevance score independently. Though a big upside in point-wise ranking is that you can get started with your favourite classification or regression algorithm with no modification at all. That being said, feel free to try different approaches offline first and decide which suits you best.

Pair-wise vs Point-wise

Here forward, we are going to show how we used point-wise method as our first approach towards solving the ranking problem. Now, let us take a look on how did we get the labeled dataset.

Dataset and Labeling

There are two ways to create training data. The first one is by human judgments, in which human judges are asked to choose more relevant item given a pair of items and the query context, but to get a dataset of substantial size it could be very costly.

In the second approach, the one that we used, past search log data is used to create relevance labels. For example, suppose for a search query, we presented the user with 100 items, out of which user scrolled up to the first 8 items and interacted with them. We convert these interactions to relevance labels, for e.g. an impression has a relevance score 0 and click has 1. These labels serve as target for the classification problem, later during prediction time, the class probability of the relevant class(in the above example click) is used as the ranking score. In case you have multiple relevance levels you can use methods like McRank, Prank etc. to convert class probabilities into ranking score.

Creating labels from search log data

Features

We have various signals on our platform that can be used to determine relevance of a particular item given a search context, for e.g.

Features that provide proxy for the performance of the item like its CTR during the last day or the last week

Features that provide information about the activity of the item’s seller on the platform like number of items posted in the last week

Personalisation features like what was the price of the item that the user last bought, within a product category

Prioritising which features to use is very important as making the same features available online during the prediction time is a challenging task, which can be be very time consuming and can involve a lot of engineering work. A good approach would be to explore all the features offline first, by recreating them from the historical data and selecting the features which are providing the best uplift in the performance with least effort to make them available during the prediction time. Also, even before exploring all the features, a good exercise could be to use the features that are already available during the request time or are present in the search infrastructure. This will let you know early on, if you are optimising the right objective function offline or if the assumptions that you have made during the labelling step are okay, plus as the system matures it gets hard to debug. For us, the final training dataset looks something like this:

Evaluation Metric

Choosing the right evaluation metric is very critical and it should take into consideration various factors such as layout of the results page, business impact etc. There are many options available and you can pick one or multiple.

We chose NDCG@K as our evaluation metric for measuring the offline performance of the algorithm, K can be decided based on how many items does the user actually see in the search results on average. For e.g. if the user rarely goes to page 2 of your search results, then you can pick K as the number of the items you show on page 1, as you would really want the relevant item to show up in page 1 of the search result. To measure the performance online, we can look at metrics like conversions/search or any other key performance metric related to discoverability in search.

Modelling and Serving Infrastructure

For the first version of the ranking model, we used the features that we were already using in the hand-tuned formula but this time learned these weights by training a logistic regression model on the labeled dataset. These features were already being calculated online, so no separate infrastructure was required for testing, plus training a simple logistic regression helps us to quickly validate the coefficients whether they align with the business intuition or not.

Another big advantage of using logistic regression model is that the learned coefficients can directly be plugged into the ranking function of the existing search engine so that we don’t need to build additional ML serving infrastructure to test this model. You can also take advantage of plugins provided for ranking by popular search engines like Solr and ElasticSearch.

Results

With this minimal effort we were able to increase the key performance metric for search by 10%. These initial results validated our training process and laid down the foundation for the project.

To keep the blogpost as an introduction towards ranking, we deliberately didn’t cover certain aspects of the problem like the position-bias, improvements provided by pair-wise methods, designing a feature service and high performant ML serving architecture. We plan to cover these topics in future.

The Team

Special thanks to Cristian Martinez, Mouloud Lounaci, Andres Pipicello, Colin Smith, Jorge Corigliano, Vladan Radosavljevic, Data Science and Personalisation & Relevance team at OLX.

Resources and Further Reading

Libraries

Search Engine Plugins

Evaluation Metrics