Introducing Hoeffding’s Inequality for creating Storage-less Decision Trees | by Dr. Robert Kübler | Towards Data Science

Imagine that you have a huge labeled dataset and you want to build a model for a prediction task. This can be Twitter, for example, where you have more Tweets (the features) than you can count including the corresponding number of likes (labels). Now you want to build a model that can predict whether a tweet will be liked more than 100 times or not, trained on all tweets written in the last year, i.e. we want to solve a classification task. A label of 1 means that the tweet has more than 100 likes, 0 otherwise.
You decide to use a Decision Tree or even something smart derived from it, like a Random Forest or Gradient Boosting. But Decision Trees, models based on them, and even other models share the following disadvantage:
You need the training data readily available in memory.
In the best case, the complete training data fits into your local machine’s memory. However, you realize that the tweets dataset is larger than 8–32GB, so you are out of luck. Maybe you have access to a cluster with 512GB of RAM, but this is not large enough, too.
How large is the dataset actually? Let us do a rough estimation. Twitter itself and several other sources (here and here) report that there are around 6,000 Tweets per second. This means around 6,000 * 602 * 24 * 365 = 189,216,000,000‬ tweets per year. Let us assume that each tweet has a size of 140 bytes, one byte for each character in the bag-of-words encoding. We ignore that each tweet might have some metadata attached and that we could also use bigrams, trigrams, etc. This is a whopping 140 * 189,216,000,000‬ / 109= 26,490 GB of tweet data for a single year!
So, using RAM is not an option. And even if you happen to own a hard disk big enough to fit the whole dataset, reading the data from it would make the training awfully slow, as you can see here (Figure 3).
Well, what to do?
Pedro Domingos and Geoff Hulten from the Department of Computer Science & Engineering of the University of Washington introduced a variant of Decision Trees — called Hoeffding Trees [1] — that can be used in a streaming fashion. This means that we have to parse the large training dataset only once and build the tree along the way.
We don’t even have to store the samples used: We can grab them directly from Twitter (or any large database), process them by increasing some counters and forget about them again. Why this is possible can be explained using Hoeffding’s Inequality, giving the Hoeffding Trees their name.
The high-level idea is that we do not have to look at all the samples, but only at a sufficiently large random subset at each splitting point in the Decision Tree algorithm. How large this subset has to be is the subject of the next section.
I am writing this article since Domingos’ and Hulten’s paper is quite technical (and therefore, also precise) and I want to present a high-level, easy to understand view on why the authors’ method works.
Also, if you do not want to deal with the math in the next sections, check out the last section for some code at least! There, I am using the package scikit-multiflow to use Hoeffding Trees.
Let us examine what Hoeffding’s Inequality says and how we can utilize it to solve the storage problem.
Wassily Hoeffding, a Finnish statistician and one of the founders of nonparametric statistics, invented (building upon ideas of Herman Chernoff) discovered an inequality [2] quantifying the following statement:
The sum (and also mean) of bounded random variables is tightly concentrated around its expected value.
Take a fair coin (probability of seeing heads = 0.5) that we flip 1000 times, for example. Define random variables X1, ..., X1000 with
Then the number of heads — let us call it X — is just the sum of the Xi’s. We already know that we can expect 500 times heads since
Now Hoeffding’s Inequality got us covered: We also know that X will not deviate much from 500 with high probability, see the illustration below. We will see what “much” and “with high probability” mean in a second.
Let X1, ..., Xn be independent Bernoulli random variables, i.e. each of them takes the values 0 or 1. Let X = X1 + ... + Xn be their sum. Then
Note: A more general formula holds, where the Xi’s can be bounded between any real numbers a and b and don’t even have to be stochastically independent. But we work with the version presented here.
Intuitively, the inequality makes sense: The larger t gets, i.e. the larger we allow the error to become, the more the probability of landing inside the interval or length 2t increases.
But the real interesting thing is the following: The right side of the equation only includes the amount of random variables n and the allowed error t. The most important thing, the probability of the Xi’s being equal to one, is nowhere to be found on the right-hand side of the inequality. The probabilities can be known or unknown to us, they can be the same for all i or not, it just does not matter. This is what makes this inequality so versatile and great to work with.
Other inequalities with this property are the easier Markov’s Inequality and Chebyshev’s Inequality, but the lower bounds on the right-hand side they give us are much worse. To be fair, these two inequalities do not need that our random variables are bounded. Hoeffding’s Inequality, however, uses the fact that the Xi’s are bounded between 0 and 1.
Taking the inequality, replacing t with nt and dividing in the inequality inside P(...) by n yields the Hoeffding Inequality for the mean:
where
Let us see the inequalities in action now. At first, we will talk about a very basic coin example. Then we will go to the second very important example of estimating shares, that we will need to understand why Hoeffding Trees work.
Let’s say that we want to compute the probability of getting less than or equal 450 or greater than or equal 550 heads. Since E(X)=500, we can set t=50 and end up with
This means that the probability of deviating less than 50 from the expected value is very high. The number of heads is extremely concentrated around 500.
Assume that there is a large pool of N balls, an unknown share p of them being white and the rest being black. A natural question might be: How large is the share p of white balls? If the pool size N is large enough (think of several billion), picking up every ball and checking it is not an option.
Of course, the natural thing to do is to uniformly sample a few — let’s say n — balls with replacement and check them. If w of them are white, we would like to conclude that p≈w/n. But how large should our subsample be? What is a good size for n? Is 10 ok? 100? 1000?
Well, it depends on how much deviation from p you are fine with and how much confidence you need to be within this deviation. You can set two parameters:
an upper bound for the error on p, let’s call it t again
a lower bound for the probability of being within the interval centered around p with length 2t, let’s call is 1-ɛ for a small ɛ>0.
This means that t and ɛ are fixed, and we want to find a number n of examples, such that with probability at least 1-ɛ the fraction w/n does not differ from p by more than t.
We shall formalize this now. Let us define n random variables X1, ..., Xn with
Since we randomly draw balls with replacement, the probability of Xi being 1 is exactly p, the unknown share. This is also exactly the expected value of Xi.
Let X̅=w/n be the mean of these Bernoulli variables again. Then we also have
and Hoeffding’s Inequality gives us
Now we want the right-hand side to be larger than 1-ɛ, which gives us a lower bound on X̅=w/n deviating from p by not more than t with probability at least 1-ɛ. Thus, we have to solve the inequality
for n. Using some elementary operations, we get
If we tolerate an absolute error of at most 1% from p with probability at least 99%, we have to set t=0.01 and ɛ=0.01, giving us a required subsample size of at least
Let’s try it out in Python. We consider a pool of 30,000,000 balls, 10,000,000 of them being white (labeled 1) and 20,000,000 being black (labeled 0). So the real share of white balls is 0.3333... which we pretend we don’t know.
import numpy as npnp.random.seed(0)# Create a pool with 30 million entries, 10 million being a one (white ball) and 20 million being a zero (black ball).# This means that we have a share of 1/3=0.3333... white balls. pool = 200000000 * [0] + 100000000 * [1]# Set Hoeffding parameters.t = 0.01eps = 0.01hoeffding_amount = int(np.ceil(np.log(2 / eps) / (2 * t ** 2)))subsample = np.random.choice(pool, size=hoeffding_amount, replace=True)print(subsample.mean())# Result: 0.32975992752529065
Looks good! The error is only about 0.0035 < 0.01 = t. But the probability for this to happen was at least 99% anyway, so why are we even celebrating? ;)
Note that instead of sampling, saving, and then taking the mean of the 26,492 samples, we can merely sample, increase a counter whenever we have drawn a white ball (a one) and then forget about the sample again. This way we only have to keep track of the counter and the total number of balls we looked at, making this a very memory-efficient memory algorithm (logarithmic in our subsample size n).
In total, we can say that a subsample of size around 26,500 is enough for determining the share of white balls with high confidence.
In the last example, we have seen how to compute the share of white balls in a gigantic pool of black and white balls without checking the entire pool and without a large error with good probability.
We will use this knowledge to train a Decision Tree without the need of having a large amount of training data available in memory, the problem we wanted to solve in the first place.
But first, we have to repeat how a Decision Tree is built the usual way.
I will not go into much detail here how the (better to say: any) Decision Tree algorithm works. There are many great videos and articles about that on the web and here on Medium.
But if the complete training dataset fits into our memory, usually this is what happens in the first step in the algorithm:
We calculate a measure of impurity I1 of the labels of the complete dataset, e.g. the Shannon Entropy for classification tasks, which we will also use now.
Then we take a feature f and a cut point c. Using these, we partition the complete dataset into two disjoint subsets:
One with all the samples where feature f has values smaller (or equal) than c.
The other one with all the samples where feature f has values smaller strictly greater than c.
The impurity of these two sets is measured again and combined into a weighted average, giving a single impurity measure I2 for both sets. Then the Information Gain is calculated as G=I1-I2.
Consider the following example of one split:
This is done for all features and all possible cut points and the one with the largest Information Gain G is chosen as the first split in the tree. We repeat this recursively for all leaves in the tree now until some stopping criterion is met (the impurity of a node is smaller than a threshold, the tree has reached a maximum depth, there are too few samples in a node, ...).
Now, we have to simulate the behavior using fewer samples. As we have seen for the normal Decision Tree:
Only the share p of samples with label 1 matter to compute the impurity!
Fortunately, we can approximate this share p and hence the impurity using a subsample of the original training set, as discussed in the section about Hoeffding’s Inequality. Labels with label 1 are the white balls, samples with label 0 are the black balls, in the language of the ball example.
Considering a single split, we can approximate I1 using only about 26,500 samples. However, for estimating I2, we need 26,500 in each child node. If we are lucky and the samples get split evenly into the two nodes, 2*26,500=53,000 samples are enough to start with. Otherwise, we might need more, but as long as we need less than several millions or billions of samples, we’re better off than before. And even if we need a million: since we can stream them into the tree and keep track of some counters, we will not run into memory issues.
This way we can safely train our model, even on every tweet made on Twitter. Happy large-scale training!
If you want to know how this is done in detail, please read the paper [1]. The authors describe their algorithm in pseudo-code with all the counters necessary and they also give proofs on how much the Hoeffding Tree built via streaming the data, and the corresponding normal Decision Tree will deviate.
We have seen that training Decision Trees on extremely large training sets is infeasible. Overcoming this problem has taken subsampling the right amount of training samples and build the tree with incomplete, yet quite accurate information. This subsampling is justified by Hoeffding’s Inequality, giving these special Decision Trees also their names: Hoeffding Trees.
Moreover, we do not even have to store the subsamples, we just have to keep track of how many samples with label one (white balls) we have seen while scanning the training data, which is an easy and effective way to reduce memory complexity even further.
We have only seen vanilla Hoeffding Trees for classification tasks in this article. There also exist algorithms for Regression and even methods to counter the so-called concept shift, i.e. when the training distribution changes over time.
Luckily, these algorithms are all implemented by the developers of scikit-multiflow for Python! Let us do a quick test.
First, do a fast
pip install scikit-multiflow
and then let us compare fitting a Hoeffding Tree on the complete training set (=enough RAM available) vs. passing in each sample one by one. More elaborate tests like this can also be found in [1].
from sklearn.datasets import make_classificationfrom sklearn.model_selection import train_test_splitfrom skmultiflow.trees import HoeffdingTreeimport matplotlib.pyplot as pltres = []# Create a dataset.X, y = make_classification(10000, random_state=123)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)# Define a tree for fitting the complete dataset and one for streaming.ht_complete = HoeffdingTree()ht_partial = HoeffdingTree()# Fit the complete dataset.ht_complete.fit(X_train, y_train)ht_complete_score = ht_complete.score(X_test, y_test)print(f'Score when fitting at once: {ht_complete_score}')# Streaming samples one after another.timer = Falsej = 0for i in range(len(X_train)):    ht_partial.partial_fit(X_train[i].reshape(1, -1), np.array([y_train[i]]))    res.append(ht_partial.score(X_test, y_test))    print(f'Score when streaming after {i} samples: {res[-1]}')    if res[-1] >= ht_complete_score - 0.01:        print(f'(Almost) full score reached! Continue for another {20 - j} samples.')        timer = True    if timer:        j += 1        if j == 20:            break# Plot the scores after each sample.plt.figure(figsize=(12, 6))plt.plot([0, i], [ht_complete_score, ht_complete_score], '--', label='Hoeffding Tree built at once')plt.plot(res, label='Incrementally built Hoeffding Tree')plt.xlabel('Number of Samples', fontsize=15)plt.ylabel('Accuracy', fontsize=15)plt.title('Fitting a Hoeffding Tree at once (enough Memory available) vs fitting it via Streaming', fontsize=20)plt.legend()
The resulting graphic might look like this:
[1] P. Domingos and G. Hulten, Mining High-Speed Data Streams (2000), Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
[2] W. Hoeffding, Probability inequalities for sums of bounded random variables (1962), Journal of the American Statistical Association. 58 (301)
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!