Efficient Cardinality Estimation using HLL with Spark and Postgres | by Ankit Prakash Gupta | Towards Data Science

A complete guide that assists you to do end-to-end efficient cardinality estimation using HyperLogLog Algorithm.
The cardinality estimation problem is the problem of finding the number of distinct elements in a chunk of data with the same elements being repeated multiple times. The problem is very generic and has a lot of applications in the real world like active-unique users of a website, unique live sessions and many more.
The problem seems to be child’s play, but we have to keep all the values in a set to keep a track of what all values we have already taken into consideration and this will eventually be a problem when we talk of Millions or Billions of Users/ Sessions. We surely cannot keep and process sets having such big cardinalities in memory.
We can use any of the following approaches for doing the cardinality estimation:
Distinct CountingLinear CountingLogLog AlgorithmHyperLogLog Algorithm
Distinct Counting
Linear Counting
LogLog Algorithm
HyperLogLog Algorithm
Distinct Counting
Brute-force approach of finding the cardinality, what we discussed above about keeping all unique elements in a set and finding out the cardinality. It is linear O(n) in terms of time and space complexities, storing all objects as is.
Linear Counting
This technique uses a hash table with capacity B bits, all bits initialised to 0. As soon as an entry arrives we hash it and that bit’s value is changed to 1. Deduplication is being taken care of by hashing the inputs. This algorithm is also linear in terms of time and space complexities but takes much less space as we are only saving bits and not the exact values. Now, let’s compare the size of the hash table with the size of Unique Elements we can have. Suppose, N unique entries can come in your set:
When N << B, then hash collisions would be less and the count of 1 bit in the hash table would be a good estimate of the cardinality.When N ≈ B, then a significant number of hash collisions would surely occur. But we can estimate the number of collisions by checking how full the hash table is. Using this estimation of the number of collisions we can extrapolate to the approximate cardinality of the set.When N >> B, then every bit in the hash table would be set to 1 and it won’t be possible to calculate the true cardinality.
When N << B, then hash collisions would be less and the count of 1 bit in the hash table would be a good estimate of the cardinality.
When N ≈ B, then a significant number of hash collisions would surely occur. But we can estimate the number of collisions by checking how full the hash table is. Using this estimation of the number of collisions we can extrapolate to the approximate cardinality of the set.
When N >> B, then every bit in the hash table would be set to 1 and it won’t be possible to calculate the true cardinality.
This approach seems to be a good fit when we have knowledge about the maximum possible unique elements we can have priori.
LogLog Algorithm
This algorithm is also based on hashing, it keeps a track of the maximum number of zeroes appearing in the binary representation of the hashed values. Let’s assume that the maximum number of zeroes appearing is k, then the estimated count would be 2^k.
It is based on the idea that one-half of the values would start with 1 and one-fourth with 01 and one-eighth with 001 and so on, the probability of getting the k leading zeroes would be 1/2^(k+1). Now, this estimation would be very unstable and would have a very high variance as we can have the very first hashed value having 5 leading zeroes also.
To make it more stable we take average (arithmetic mean) values of the estimates given by multiple hash functions. Bringing us to another computationally expensive process, hashing. Now, to cater for this issue, LogLog converts the value attained from the already hashed value into 2 values, taking the first m bits, which points to a specific bucket of values and then the rest of the bits as the value to get the maximum number of zeroes.
So, if we assume that our hash function returned 32 bits, and we took the first 14 bits to decide the bucket to which it points, so we will have 2^14 buckets to average the value of k from.
Suppose in each bucket we wanted to store N elements then we will have to store only a single number which is of log(N) magnitude and to store that we need only log(log(N)) number of bits, which gives it its name.
Taking as an example, if we take 5 bits register for each bucket to store the max-value of k for that bucket and 2^14 buckets, which will allow us to have values for each bucket from 0–31, which means that we might have 2^31 elements in each bucket (Using the above probability), which implies that we can approximately keep 2^(31+14) distinct elements at max, by using space (214 * 5) = 81920 bits (10 KBs) only.
It defines the estimated count as follows:
where, α → estimation factor (constant value)  registers(i) → value of the max-leading-zero count for the ith bucket
This is an improvement algorithm on LogLog Algorithm. It uses Harmonic mean in place of arithmetic mean when combining the results of multiple buckets, which caters unusually high values of max-leading-zero counts. Additionally, it provides corrections for two extreme cases when not all of the buckets are occupied and when a hash collision causes underestimation.
Different libraries are implementing the HLL algorithm, we chose to use Aggregate Knowledge’s HLL implementation, because of the following factors:
Has an inter-operable library with Postgres DB (postgresql-hll) as well as same has been used by Swoop’s Spark-Alchemy.
Has an ability to union two HLLs.
Easy to use APIs.
To try out the postgresql-hll extension on Mac Systems, Please use the following steps:
So, the extension is installed in Postgres. Now, we need to run the following query to activate the extension in a particular DB.
To test the above example, Create a table and inserted some data with the following queries:
Then tried the interoperability of the HLLs in Postgres with Java APIs. Created another table using the following query
Then used a Java Program to Create HLLs using a range of numbers from 0 to n, so that we know how many unique entries are there in it, pushed them to Postgres and then aggregated and did a count on them. Since we know the unique entries this way, we can calculate the error percentages for user_id and session_id.
A few things to note here are:
If we want to union two HLLs or merge two HLLs, then we should always hash them with the same seed, or else the same value will also give two different hashed value and will be counted as two.
Standard Error for the HLLs can be given by the following formula and the calculated values are expected to be within the σ, 2σ, 3σ of the actual count for 65%, 95%, 99% of the cases respectively.σ = (1.04)/√(number of buckets)
We used the Java APIs to write the UDFs
Let’s visit the types of the HLLs AgKn has come up with to increase the accuracy of Cardinality Estimation.
EmptyA constant value denotes the empty set.
ExplicitAn explicit, unique, sorted list of hashed integers in the set, which is maintained up to a fixed cardinality [If using the Auto expthresh (-1) that number can be defined by (regwidth * 2^log2m)/64 )]. This is done to increase accuracy at low cardinalities, as we can directly keep the explicit set of hashed inputs as a sorted list of 64-bit integers. We can also tweak the expthresh, to keep more numbers or fewer numbers according to our use-case, but the auto mode supports optimal memory usage.
SparseIts idea is based on Sparse Arrays, in which we keep the values of indexes that have only non-zero values. Similarly, it is a map-based implementation of HLL, which stores only the indices values of non-zero registers in a map, until the number of non-zero registers exceeds a fixed cardinality. So, it actually keeps the registers value of only those buckets whose register is having at least 1 non-0 bit.
FullA fully materialised, list-based implementation of HLL. Explicitly stores the values of every register in a list ordered by register index.
Not, visiting the Schema Layout and how data is arranged in the byte array format, it can be visited here.
This value decides the number of buckets or registers, m is the actual number of registers here, and we can calculate the number of registers by 2(Log2m). Also, this value decides the accuracy of the HLLs. The following expression gives the standard error estimate. Also, one important thing to note is the calculated values are expected to be within the σ, 2σ, 3σ of the actual count for 65%, 95%, 99% of the cases respectively.σ = (1.04)/√(m)
This is the width of the Register/Bucket or the number of bits to be used per register in the HLL Algorithm. Must be at least 1 and at most 8. This parameter, in conjunction with log2m, tunes the maximum cardinality of the set whose cardinality can be estimated. Please refer to the table for the clarification of the HLL sizes and Maximum Cardinalities.
3. ExpThresh
Tunes when the EXPLICIT to SPARSE promotion occurs, based on the set’s cardinality.
If the `EXPLICIT` representation is turned off, the `EMPTY` set is promoted directly to `SPARSE`. Must be -1 {Promote at whatever cutoff makes sense for optimal memory usage. ('auto' mode)}, 0 {Skip EXPLICIT representation in hierarchy. (Promotes directly to Sparse.)}, or 1-18 inclusive {Promote at 2^expthresh - 1 cardinality}.
You can choose the EXPLICIT cutoff such that it will end up taking more memory than a FULL hll representation. This is allowed for those cases where perfect precision and accuracy are required up through some pre-set cardinality range, after which estimates of the cardinality are sufficient.
4. SparseOn
Enables or disables the SPARSE representation. If both the EXPLICIT and SPARSE representations are disabled, an EMPTY set will be promoted directly to a FULL set. If SPARSE is enabled, the promotion from SPARSE to FULL will occur when the internal SPARSE representation’s memory footprint would exceed that of the FULL version. Must be either true or false. True means enabled and false means disabled.
HyperLogLog Algorithm helps solve the cardinality estimation problem up to a certain threshold without overburdening the storage space to keep all the elements.
Additionally, Aggregate knowledge provides great Support for HLL Algorithms. Other libraries which can be tried are Datasketch, Zetasketch.