Introduction to Streaming Algorithms
By: Dr. Robert Kübler
Tags: Streaming, Big Data, Algorithms, Efficiency, Data Science

Now I hear some of you say:

Why would I care about network stuff? I‘m a Machine Learning guy, duh! — you

Well, there is another famous example from the Machine Learning world: Gradient Descent!

If we deal with a small enough data set, it can fit into the (GPU) RAM completely, and we can use Batch Gradient Descent, i.e. put the complete data in the memory at once and process it. However, most of the time our working memory is too small, making it necessary to use the Stochastic Gradient Descent or the Mini-Batch Gradient Descent, which are examples of so-called Streaming Algorithms. Another example is the Hoeffding Tree Algorithm, which I described here.

In this article, I want to show you a few examples of Streaming Algorithms, including Python implementations that you can use! Apart from making you aware of the problem, which I have already done. ;)

Intuition

With Streaming Algorithms, I refer to algorithms that are able to process an extremely large, maybe even unbounded, data set and compute some desired output using only a constant amount of RAM.

If the data set is unbounded, we call it a data stream. In this case, if we stop processing the data stream at some position n, we expect our algorithm to have a solution corresponding to the data seen up to this point.

In the following, just imagine that we either have an enormous data set on our hard disk that we want to process without loading it into our RAM at once (because we can’t) or that there is a source that outputs a data stream, for example, incoming tweets on Twitter. Both cases are handled the same way.

I will phrase the upcoming examples in the language of large data sets since then we know that they are finite, and I don’t have to mention all the time that we stop reading a data stream.

We further assume that we can pass over the data exactly once.

A gentle Start

Let us get familiar with how we can design Streaming Algorithms using two simple examples.

Finding the Minimum

Imagine that there is an extremely large list of numbers, too large for your RAM. You want to find out the minimum of this list. In Python, classically you solve it like this:

print(min(my_list))

But this assumes that my_list is in the RAM already. So, how can we approach this in another way? Maybe you have found a solution already:

Just read the data set number after number and update the minimum, whenever you find a smaller number.

To be more precise: You read the first element and declare it the minimum. Then you read the second element, and if it is smaller than the current minimum (first element), declare it the minimum. Else, do nothing.

Then you read the third element, and if it is smaller than the current minimum, declare it the minimum. Else, do nothing.

Then you read the fourth element, and if it is smaller than the current minimum, declare it the minimum. Else, do nothing.

Ok, I stop it, you know where this is going. This basically works, because

Using this formula, you can easily show via induction that the algorithm is correct. After reading the first element, the result is correct since a₁<∞ and hence min(∞, a₁)=a₁. The induction step is exactly the formula (think about it!). But enough of this, let us get back on track.