Recursive Least Squares for Linear Contextual Bandits
By: Fedor Parfenov
Tags: Data Science, Contextual Bandit, Machine Learning, Artificial Intelligence, Reinforcement Learning

Recursive Least Squares for Linear Contextual Bandits

TL;DR — Contextual bandits can be modelled as a linear regression; you can sample arms with Thompson sampling; the recursive least square algorithm can be used to efficiently update the mean and variance of the linear regression weights.

I recently re-discovered the Recursive Least Square algorithm which I have not seen for a while. I was struck by its simplicity and elegance. I also noticed that this approach carried all the necessary features to be used in a linear contextual bandit algorithm. So I decided to write this blog laying out how I would apply it and why I think this is a neat approach.

Contextual bandit algorithms

As a quick reminder, the Contextual Multi-Armed Bandit algorithms (ConBandits) are an extension of the classic multi-armed bandit algorithms which takes into account the context at the time of decision. It is very often used by e-commerce companies to optimise their website to automatically explore different variants of the page to select the best one while minimising the opportunity cost of exploring. For more on this, please check my previous entry: Multi-Variate Web Optimisation Using Linear Contextual Bandits

Formally, the objective is to explore a set of arms A given possible context C. A is usually a set with K arms:

C is the context that can be represented by a vector of J variables:

In a given context c ∈ C, we define the optimal arm a* ∈ A as the one that maximises the expected value of R which is a random variable with an arbitrary distribution.

In practice, the algorithm will receive a series of randomly chosen contexts and the objective is to minimise the overall cumulative regret Δ by having a best possible strategy when selecting arms ã ∈ A and learning from the feedback:

ConBandits as a linear model

The contextual bandit can be driven by a linear model. We first need to map both A and C to the linear feature space 𝔛 with a mapping function m. The feature vector x ∈ 𝔛 is of length L containing an intercept; the vector of one-hot encoded a of length K; the context vector c of length J; and the first-order interactions between the context and the arms of length J × K:

Thus the length L = 1 + K + J + J × K. The first-order interactions are required to enable the model to learn context-specific optimal arms. The interactions between context variables is a possibility but this modelling decision is out of scope.

The linear relationship between an observed reward r_t and observed feature vector x_t is defined as:

With w — vector of weights of length L. Given a training set of size N with a vector of rewards r of dimension N×1 and the matrix of feature X of dimension N×L, the ordinary least square estimates of w would be:

We also know that, asymptotically, the OLS estimates will follow a multivariate Gaussian distribution:

This is a convenient property enabling the use of Thompson sampling we will describe below. For more details, those derivations can be found in many Statistics and Machine Learning textbooks.

Contextual bandit step-by-step

The algorithm below describes the 4 key steps of the ConBandits. Some of these steps can run asynchronously and often do in production; for clarity, they are presented as sequential steps: