Winning the Data Compression Game | by Don Cross | Towards Data Science

There are several powerful data compression programs in widespread use. Some famous examples are gzip, bzip2, and pkzip. Recently I started wondering if, given a specific input file to be compressed, I could create an algorithm that outperforms all of these programs.
I decided to make it a game for myself. I picked a non-trivial input file. Then I used gzip, bzip2, and zip compression on it. In each case, I supplied whatever command line options produced the most aggressive data compression possible. Then I selected the smallest byte size of the three programs’ outputs as my target. I set out to write a program that could beat this smallest byte size, given that same input file.
I wanted my test input to be something plausibly useful. I didn’t want to contrive some weird input just to give myself an advantage. I decided on using an alphabetic list of English words. Starting with an ASCII file I found online,1 I removed all lines that contained any character other than lowercase letters a through z. The result was this file: https://github.com/cosinekitty/compressgame/blob/master/words.txt
This words.txt file consists of 636946 bytes, and contains 67527 words, one per line. I used the following commands to compress this file. All of these programs are running on my 64-bit Debian Linux system.
First I started with bzip version 1.0.6:
bzip2 -c --best words.txt > words.bz2
Then I used gzip version 1.6:
gzip -c --best words.txt > words.gz
And finally, I used the Info-ZIP program, version 3.0:
zip -9 words.zip words.txt
Here are the resulting file sizes:
don@spearmint:~/github/compressgame $ ls -l words.*-rw-r--r-- 1 don don 248587 Apr 22 15:05 words.bz2-rw-r--r-- 1 don don 212736 Apr 22 15:05 words.gz-rw-r--r-- 1 don don 636946 Apr 22 14:49 words.txt-rw-r--r-- 1 don don 212876 Apr 22 15:05 words.zip
So in this particular case, gzip is the winner. It produced output of 212736 bytes. That was my target to beat.
Every game needs rules. It makes sense to solidify the rules before playing the game. Here are the rules I chose in advance:
The output of my data compressor is another Python program.
When executed, the generated Python program must print the exact text contained in words.txt.
The program must not reference any external files. It must be completely self-contained and create its output algorithmically.
The winning program’s source code must be smaller than 212736 bytes.
To make it more interesting, I decided the generated Python program must not use any import statements. It must use only built-in parts of the Python language.
I knew I would be experimenting a lot. I needed a test harness that could try any number of algorithms. For each algorithm, the harness should generate the candidate Python program as output, execute the program, confirm it produces the original word list verbatim, and record the size of the program in bytes.
The harness must abort with an error message if any generated program fails to produce correct output. If all programs pass the test, it picks the winner: the output that has the smallest byte size.
My test harness is called squash.py. It expects a list of algorithms in a variable AlgorithmList. Each algorithm is an object that must expose a Name function and a Compress function. Here is the relevant excerpt from squash.py. (Complete project source code is available on GitHub. See the References section at the end of this article.2)
The algorithms I tried all rely on Huffman coding, a technique that represents a finite set of symbols using a variable number of bits. Symbols that occur more frequently are represented by fewer bits; rarer symbols require more bits. The main burden with Huffman coding is that the symbols are not byte-aligned. This requires extra code to pack the bits together as tightly as possible, ignoring byte boundaries.
Huffman coding works by building a binary tree whose leaf nodes represent every distinct symbol present in the input. The binary encoding of each symbol is determined by the path from the root node to that symbol’s leaf node.
I’ll explain the algorithm for building the Huffman tree using a simple example where there are only five symbols: the letters A through E. Suppose we count how many times each of these letters appears in our input and arrive at the following numbers.
A : 351B :  12C :   9D :  13E : 823
If we were to use a fixed number of bits to represent 5 alternative symbols, we would need 3 bits, because 23=8, which is the smallest power of two large enough to represent 5 alternatives. Huffman coding can do a more efficient job by exploiting the fact that we can use fewer bits for the symbols that occur most often.
The Huffman encoder starts by creating 5 leaf nodes, one for each symbol. There is no tree yet. We just have leaf nodes floating separately, not connected to each other yet. The encoder creates a list of the nodes sorted in ascending order of count. So here’s what it looks like when we start:
Then the Huffman encoder goes into a loop. In each iteration, the encoder removes the first two nodes in the sorted list and creates a new parent node for them. The parent node’s count becomes the sum of the two child nodes’ counts. The parent node is inserted back into the list of nodes at whatever position keeps the nodes sorted by count.
So in our example, the encoder removes the C:9 node and the B:12 node, creates a parent node whose count is 9+12=21, and inserts that new node between D:13 and A:351, to preserve the sorted order. The result looks like this:
Note that the newly created node is an internal node, not a leaf node, so it does not represent a single symbol. It, therefore, has a count but no symbol attached to it.
Because each iteration removes two nodes from the list and replaces them with one node, the list gets one node shorter each time. The algorithm keeps looping until only one node remains. That node is the root node of the final tree. In our example, here is what the finished tree looks like:
At this point, we no longer need the frequency counts. Also, it is helpful to label the branches in the tree with the bits 0 and 1 to make the encoding clear. Every left branch receives a 0, and every right branch receives a 1. Let’s also make the leaf nodes and the internal nodes look different. The tree now looks like this:
We use this tree to build a binary representation for the letters A through E. For each letter, start at the root of the tree (marked with a *) and travel down the tree until you reach that symbol. Each time you branch left, append a 0. Each time you branch right, append a 1. This results in the following bit patterns for our five symbols:
A = 01B = 0011C = 0010D = 000E = 1
Just as we wanted, the symbol that appears most often, E, uses the smallest amount of data: just 1 bit! The two least common symbols, B and C, use 4 bits each, which is worse than the 3 bits that a fixed bit length representation would have used. But the good news is there aren’t many B’s and C’s in the input.
So how good is our data compression in this simple example? Let’s compare the naive fixed-bit approach with the Huffman variable-bit approach.
The fixed-bit approach requires us to use 3 bits for every symbol. If we add up the total number of symbols in the input and multiply by 3 bits per symbol, we get:
3*(351+12+9+13+823) = 3624 bits
Now let’s see how many total bits we need in the Huffman representation. There are 351 A symbols, each using 2 bits; 12 B symbols using 4 bits each, and so on. That gives us
2*351 + 4*12 + 4*9 + 3*13 + 1*823 = 1648 bits
We divide 1648 by 3624 to find that the compression ratio is about 45%. So in this case, Huffman encoding shrinks the input to less than half of its original size. Of course, this compression ratio is different for every kind of input. It all depends on the relative frequency counts of the symbols present in each unique input stream.
It’s also important to note that the Huffman tree itself needs to be stored in the compressed output. The decompression algorithm needs to know how to interpret the bits, in order to reconstruct the original data. So this adds a little bit of overhead to the size of the compressed output.
The source code for my Huffman encoder consists of two classes: HuffmanNode and HuffmanEncoder. Both classes are in the source file huffman.py.
A HuffmanNode object represents one of the nodes in the Huffman tree, as explained above. Each node contains a symbol field that holds the original uncompressed symbol, a frequency count, and references to left and right child nodes. The references are set to None for leaf nodes. For internal nodes, left and right refer to other HuffmanNode objects. Here is the code for HuffmanNode:
The HuffmanEncoder class supports the algorithm explained above. It contains a Tally method to be called by a compressor algorithm to count up how many times each symbol appears in the input. It also contains a Compile method that generates the Huffman tree. It implements the algorithm described above for building the tree from a sorted list of nodes. Compile returns the root node of the tree, which gives the caller the entire tree.
Once a compressor algorithm has tallied up all the symbols and called Compile to get the Huffman tree, it needs a way to convert each symbol in the input to its Huffman representation as a variable-length bit pattern. To do so, the compressor calls the root node’s MakeEncoding method. This function creates a dictionary whose keys include every symbol found in the input.
In this dictionary, the value associated with each symbol is a tuple (data, nbits), where data is an integer holding the bits representing the path from the root of the tree to that symbol, and nbits is the number of significant low-order bits in that integer. For example, if the bit pattern were 0111, data would be 7 and nbits would be 4, yielding the tuple (7,4).
Now we have a good foundation for converting raw input text into Huffman encoded bit streams. But how do I represent these variable bit streams in the generated Python source code?
I made my challenge more difficult by requiring the output to be Python source code. Not only does the program have to contain its own decompression logic, but source code by its very nature must contain only printable characters. Arbitrary binary data would cause a garbled mess in Python code.
To get around this problem, I encode the binary data using Base64. In this encoding, each consecutive block of 6 bits is replaced with one of 64 characters: A-Z, a-z, 0–9, +, or /. Each of these ASCII characters requires a full 8-bit byte to represent, so I’m losing 2 bits of efficiency for every 6 bits I want to store.
Although there is a built-in base64 module in Python, my encoder has certain special needs, so I wrote my own class called BitBuffer. It has the following features:
It has an Append function that accepts a tuple (data, nbits). This meshes perfectly with the tuples stored in the dictionary returned by HuffmanNode.MakeEncoding.
The Append function converts to Base64 on the fly. For every batch of 6 bits, it buffers another Base64 character. It remembers any leftover bits and holds them until more bits arrive to fill out more 6-bit chunks.
Append prevents text lines from becoming too long by wrapping with a newline character after every 80 Base64 characters.
There is another method Format that flushes any residual bits as a final Base64 character. It then converts the entire Base64 buffer to a UTF-8 string that can be written to the output source code. It returns that string to the caller.
Here is the BitBuffer class.
Now I had my infrastructure in place. The next step was to try out various compression algorithms that could generate bit streams to feed into the BitBuffer class, and create the corresponding decompression algorithms to extract the original data back from the compressed Base64 image.
In my first attempt, I knew I would not beat gzip, but I had to start somewhere. All I did was use a Huffman encoding on every character in the input. There are 27 distinct characters that appear in the input: a-z and the newline character (represented as '\n' in Python). My first attempt processes the input in two passes. In the first pass, it counts up how many times each of the 27 characters appears. It uses this information to build a Huffman tree.
Once it has the Huffman tree, it makes a second pass calling a private function _Encode to create a compressed Base64 representation of the word list.
I wrote this first compressor as a module named cmp_letters.py. Here is its source code:
How well did this form of compression work? It did make the data smaller, but as I suspected, it didn’t beat gzip. The total size of the output file letters.py is 460531 bytes for a compression ratio of 460531/636946 = 72%. Here is what letters.py looks like, edited for brevity. (You can obtain the entire thing by cloning my GitHub repository in the References section at the end of this article2 and running the squash.py program.)
There are some features of my first attempt that are shared by all three of my decompressors. The first line of executable code creates a variable Char. It is a representation of the Huffman tree using a tuple. The tuple contains two elements. Each of those elements is another tuple with two elements, and so on, all the way down until you reach leaf nodes that are individual characters.
To understand how the Huffman tree in Char is used, take a look at the function Huffman lower down in letters.py. The Huffman function uses the BitReader class to extract bits one at a time from the Base64 encoding. It travels down the binary tree one bit at a time, selecting either the left branch (the first element inside the tuple) or the right branch (the second element), depending on whether the next bit in the Base64 data is a 0 or a 1. Eventually, it reaches an element that is not a tuple. That element is the leaf node of the tree, consisting of the original, uncompressed character. The Huffman function returns that character to the caller.
I knew I could do better than a 72% compression ratio. My next idea was to exploit the fact that the words appear in sorted order. That means there are usually some letters in common between adjacent words in the list. For example, consider this excerpt from the words.txt file:
decadedecadencedecadencydecadentdecagondecahedron
The boldfaced letters show sections of each word that match the front of the previous word in the list. I reasoned that I could create symbols that represent these integer counts. Then the words above, after decade, can be represented more concisely as
6nce8y7t4gon4hedron
So in my second compressor/decompressor attempt, I create two separate Huffman trees, one for prefix counts, the other for letters that appear in the suffixes. This shows that the Huffman encoder can handle integers or characters as input symbols. The HuffmanEncoder class doesn’t care what the data type is, so long as each symbol allows itself to be used as a dictionary key.
The compressor source file is named cmp_prefix.py. Here is its code:
Here is the corresponding generated decompressor, again edited for brevity:
Replacing the common word prefixes with integer counts really helped. It brought the output size down to 220603 bytes. I was getting excited now, because I was close to gzip’s score of 212736 bytes! I just needed to squeeze a little more efficiency.
My next idea was that I could exploit the fact that certain letters are more likely to occur in pairs than others in English. For example, if you see a q in an English word, the next letter is almost certain to be a u. Even though u is a fairly common English letter, it is much more likely to appear right after a q. In general, the current letter strongly influences the probabilities of what letter comes next.
So my third attempt was based on the prefix optimization of my second attempt, only instead of using a single Huffman tree for letter frequencies, it uses 27 Huffman trees: one for each of the 26 possible preceding letters, plus one more for the first letter in a word (when there is no preceding letter).
This approach means I have to consume extra space in the output to store an extra 26 Huffman trees in the generated output.
Repeat is the Huffman tree for the number of characters at the front of each word that matches the previous word. Tail is the tree for decoding the number of characters in the rest of each word after the prefix. And Char is now a list of Huffman trees. Char[0] is the tree for the first character in a word, Char[1] is for any character that follows the letter a, Char[2] is for any character that follows the letter b, and so on.
This is where I finally beat gzip. The source file pairs.py is 202641 bytes long, or 10095 bytes smaller than words.gz. Victory!
Clearly, my approach in no way dethrones gzip as a top-tier general purpose data compression algorithm. What I did only works this well because I tailored it to my human understanding of the given input file. My algorithm won’t even handle inputs that have characters other than lowercase letters and newlines. It will not perform well if the inputs are not sorted alphabetically.
But this exercise does demonstrate something useful. In cases where you have a fixed data set that you want to compress, it is sometimes possible to craft a custom compression algorithm that performs better than well-known general purpose algorithms.
If you are up for a fun challenge, perhaps you will continue where I left off. Using the same rules, can you create another algorithm that beats mine for the same words.txt input? If so, I would enjoy hearing from you and learning what you did.
My test input is based on a word list I found at: http://www-personal.umich.edu/~jlawler/wordlist.htmlComplete source code for my custom data compressor:https://github.com/cosinekitty/compressgame
My test input is based on a word list I found at: http://www-personal.umich.edu/~jlawler/wordlist.html
Complete source code for my custom data compressor:https://github.com/cosinekitty/compressgame