Deduplication at Scale. At Amplitude, we ingest events at a…
By: Gurminder Singh
Tags: Big Data, Distributed Systems, Scalability, Probability

Alternate Approach — Bloom Filters

A Bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom. It is designed to tell you, rapidly and memory-efficiently, whether an element is a member of a set. It allows elements to be added to the set, but they can not be removed.

Since it is a probabilistic data structure — it tells us that the element either definitely is not in the set or may be in the set. In other words, false negatives are not possible but false positives are possible.

Bloom Filter — Overview

An empty Bloom filter is a bit array of m bits, all set to 0. It is associated with k different hash functions. Typically, k is a constant, much smaller than m. Also, m is proportional to the number of elements to be added to the bloom filter. The precise choice of k and the constant of proportionality of m are determined by the intended false positive rate of the filter.

To add an element, we feed it to each of the k hash functions to get k array positions. We set the bits at all these positions to 1.

To query for an element (test whether it is in the set), we feed it to each of the k hash functions to get k array positions. If any of the bits at these positions is 0, the element is definitely not in the set — if it were, then all the bits would have been set to 1 when it was inserted. If all are 1, then either the element is in the set, or the bits have by chance been set to 1 during the insertion of other elements, resulting in a false positive.

The following diagram illustrates a bloom filter with two hash functions (items A and B have been added; we query for items A, C and D):

While risking false positives, Bloom filters have a strong space advantage over other data structures for representing sets (we do not store the data items at all). But, we can manage the false positive rate of the Bloom filter. A larger Bloom filter will have less false positives, and a smaller one more.

So, depending on the use-case, we can appropriately choose the parameters for a bloom filter viz. number of items (N), probability of false positives (P), number of hash functions (K) and size of bloom filter (M). There is a Bloom Filter calculator which is really handy to play with these.

The time needed either to add items or to check whether an item is in the set is a fixed constant, O(k), completely independent of the number of items already in the set.

Also, the size of bloom filter (M) grows linearly as we increase the number of items (N) while we keep the probability of false positives (P) and number of hash functions (K) constant.

Fitting the Solution to the Problem

Bloom filter seems to fit the use-case very well, we want a quick way to get an answer on the presence or absence of specific data item (without even storing the item). We can pick the probability values such that the results remain accurate.

We know our traffic numbers (this gives N), pick reasonable number of hash functions say 20 (this gives K) and we can target for low probability of false positives say 1 in a billion (this gives P), this will give us the size of bloom filter (which is M).

Also, since the size of Bloom filter will grow linearly with the increase in traffic, we will have linear scalability of the solution.

We partition our ingested data into a reasonably big number of partitions. This helps us split the problem into smaller sub-problems.

Let us take an example to play with some numbers e.g.

with 200K events per second overall

events per second overall 1024 partitions for processing of data

partitions for processing of data we need to process ~17 million events per day per partition.

Now, if we round this to 20 million events per day per partition, we will need a bloom filter of size ~108Mb per partition with false positive probability of 1 in billion.

This would mean, we will need ~108Gb (1024 partitions*108Mb) of memory per day.

So, for 7 days of dedupe data, we will need ~756Gg (7*108Gb) of memory.

This can be handled easily with just a few machines (e.g. AWS x1e.2xlarge offers 244Gb memory), with really good performance, good reliability (we picked low false probability numbers), with linear scalability, much lower cost and less operational overhead.

Also, to support rolling window of deduplication, we can create a new bloom filter each day and can drop it as we move out of the window.