Diving Head-First Into The Dark Pool Problem
By: Louise Ferbach
Tags: Trading, Research, Editors Pick, Reinforcement Learning, Machine Learning

Part Four : The Censored Exploration Algorithm

Now, time to get to the serious stuff. In this section I will present the paper’s main theoretical result, which is a polynomial-time, near-optimal algorithm for multi-venue exploration from censored data. I will first provide an overview of the algorithm and its analysis before diving into the technical details.

Presentation and Explanation of the Algorithm

If you understood the Greedy Algorithm, you should have no difficulties with the Censored Exploration Algorithm. As I said previously, the greedy process was identical over time ; the authors simply introduced timely updates of the tail distributions estimates, calculated with the censored observations from each venue collected at the previous timestep (i.e., the consumed units at each venue given the number that had been allocated). Thus the algorithm (Algorithm 2 below) implements a continuous allocate-reestimate loop.

The Censored Exploration Algorithm

Everything seems simple in my description of the procedure, but as you will have understood, the crux of the matter is updating at each time step the estimated probabilities according to the censored data collected at this step. This corresponds to the subroutine OptimisticKM, which specifies how we calculate the new estimated tail probabilities from the observed data.

The most natural choice would be the Maximum Likelihood Estimator (MLE) on the data, aka the Kaplan-Meier estimator (thus the KM in OptimisticKM, which is only an optimistically modified MLE that will be detailed later).

I present hereunder an implementation of this algorithm.

You can see that I can provide here any implementation for the update of the tail estimates. I will provide, further in the article, an example with the traditional implementation of the MLE, as well as with the optimistic version.

It’s all very dense, so we will proceed step by step at this stage.

Step 1 :

We first review the MLE for censored data and provide a new finite-sample convergence bound for this estimator.

This allows us to define a “cut-off point” for each venue i such that the MLE of the tail probability T_i(s) for every value of s up to the cut-off point is guaranteed to be “close to” the true tail probability.

We then define a slightly modified version of the MLE in which the tail probability of the next unit above the cut-off is modified in an optimistic manner (to be defined).

We show that in conjunction with the greedy allocation algorithm, this minor modification leads to increased exploration, since the next unit beyond the cut-off point always looks at least as good as the cut-off point itself.

Step 2 : We next prove our main Exploitation Lemma (Lemma 4).

This shows that at any time step, if the number of units greedy-allocated to each venue is strictly below the cut-off point for that venue, then the allocation is “epsilon-optimal” (to be defined).

Step 3 : We then prove our main Exploration Lemma (Lemma 5).

This shows that on any time step at which the greedy-allocation is not “epsilon-optimal”, it is possible to lower-bound the probability that the algorithm explores. Anytime we are not in a known state and thus cannot ensure optimal allocation, we are instead assured of exploring.

Step 4 : Finally, we show that on any sufficiently long sequence of time steps (“sufficiently long” is polynomial in some parameters), we must have :

either the algorithm achieves an “epsilon-optimal” solution on a lower-bounded fraction of the sequence,

or the algorithm has explored sufficiently often to learn accurate estimates of the tail distributions out to V units on every venue.

In either case, we can show that with probability 1 − δ, at the end of the sequence, the current algorithm achieves an “epsilon-optimal” solution with a lower-bound probability 1 −ϵ.

Photo by Israel Gil on Unsplash

Step 1 : Review & Adaptation of the Maximum Likelihood Estimator

We will work with a given venue i.

We define by z the true probability that the demand (or the liquidity, or the maximum number of units that can be absorbed) in this venue is exactly s units given that it is at least s :

Let D be the number of direct observations of s units up to time t, and let N be the number of (direct or censored) observations of at least s units on time steps at which more than s units were requested. The quantity N is then the number of times there was an opportunity for a direct observation of s units, whether or not one occurred.

We can then naturally define the empirical probability of a direct observation of s units given that a direct observation of s units was possible, what leads us to the MLE of the tail probability for any s > 0 after t time steps :

MLE of tail probabilities, traditional calculation

Here is what we get when we implement the MLE update in Algorithm 2 :

We’ve reached the point where mathematical theory stops helping us :

There are convergence rates for the MLE to the true underlying distribution in the case that the submission sequence is i.i.d ;

for the MLE to the true underlying distribution in the case that ; There are asymptotic convergence results for non-i.i.d. settings ;

results for settings ; We are in the non-i.i.d. case, since the submitted volumes at one venue are a function of the entire history of allocations and executions across all venues, but we would like a finite sample convergence bound.

That’s not a deadend though, as the authors demonstrate. Indeed, from the formula of the tail probability estimate, they extract theorem 2 hereunder which states that these tail proability estimates also converge with time to the true distributions, for each venue.

Theorem 2 :

(I will not explicit the proof, which calls lots of unpleasant calculus, since it doesn’t help the comprehension. Feel free to read it in the original paper.)

Hurray ! Now that we have a convergence bound of the MLE towards the true distribution, we can try to modify this estimator a bit. Indeed, it seems obvious that, the more data we have, the more accurate our estimator will be, but we would like to guarantee that it will indeed get sufficiently accurate if we provide for the necessary amount of data. I will explain the details hereunder.

Computation of the OptimisticKM

In Algorithm 3 , the value c can intuitively be viewed as a “cut-off point” up to which we are guaranteed to have sufficient data to accurately estimate the tail probabilities using the MLE.

Thus for every quantity s lower than this cut-off point, we simply calculate the tail probability estimates through MLE. It remains to fix the estimates for quantities above that threshold. We decide to set the value of the estimated tail probability one unit above the cut-off point to the MLE of the tail probability at the cut-off point.

This optimistic modification ensures that the greedy algorithm explores (i.e., has a chance of making progress towards increasing at least one cut-off value) on every time step for which the produced allocation is not already optimal.

In particular, suppose that the current greedy solution allocated no more than c_i units to any venue i and exactly c_j units to some venue j. Using the standard maximum likelihood tail probability estimates, this allocation could be suboptimal (there is no way to know if it would have been better to include the next unit from venue j in place of a unit from another venue since we do not have an accurate estimate of the tail probability for this unit), and yet no exploration is taking place.

By optimistically modifying the tail probability estimates of the current amounts increased by 1 for each venue, we ensure that no venue remains unexplored simply because the algorithm unluckily observes a low demand a small number of times.

This c_i is a “cut-off point” up to which, at each step, the MLE are accurate.

Hereunder is a Python implementation of the procedure.

Steps 2 & 3 : Exploitation and Exploration Lemmas

In this part, I will not aim at proving every lemma and theorem mathematically or go deeply into technical details, but rather at giving a general intuition of their meanings.

Lemma 2 :

Lemma 2

This is a trivial consequence of theorem 2 combined with the formula of cut-off points from algorithm 3.

Lemma 3 :

Lemma 3

Lemma 3 shows that it is also possible to achieve additive bounds on the error of tail probability estimates for quantities s much bigger than the cut-off point as long as the tail probability at that cut-off point is sufficiently small.

We are now ready to state our main Exploitation Lemma (Step 2), which formalizes the idea that once a sufficient amount of exploration has occurred, the allocation output by the greedy algorithm will be epsilon-optimal.

Lemma 4 (Exploitation Lemma) :

Lemma 4 : Exploitation Lemma

Note that these bounds are tight in the sense that it is possible to construct examples where the equality is obtained.

Finally, we present the main Exploration Lemma (Step 3), which states that on any time step at which the allocation is not epsilon-optimal, the probability of a “useful” observation is lower bounded.

Lemma 5 (Exploration Lemma) :

Same assumptions. If the allocation is notϵ-optimal, then for some venue i, with probability at leastϵ/(8V ),

Step 4:

Putting it all together, we finally get to the main theorem hereunder :

Theorem 3 :

For any ϵ > 0 and δ > 0, with probability 1 − δ (over the randomness of draws from Q and {P_i}), after running for a time polynomial in K, V, 1/ϵ, and ln(1/δ), Algorithm 2 makes anϵ-optimal allocation on each subsequent time step with probability at least 1 − ϵ.

I will conclude this theory part with a few brief remarks.

Our optimistic tail modifications of the maximum likelihood estimators are relatively mild. In many circumstances we actually expect that the estimate-allocate loop with unmodified MLE would work well .

. It also seems quite likely that variants of the algorithm and analysis could be developed for alternative objective functions, such as minimizing the number of steps of reallocation required to execute all shares, or maximizing the probability of executing all V shares on a single step.

I hope you are not overwhelmed by the flood of mathematics. Don’t worry, we will now drift towards the dark pools.