Creating Custom Character Encoding to Save Space
By: Oliver Jackman
Tags: Encoding, Data Storage, Algorithms, JavaScript, Mathematics

A thought experiment

Let’s say we want to store the list of oauth providers and we represent that as a shortcode identifier and the name of the provider:

A simple representation of a short list of OAuth providers

We know that the code of each provider is only ever going to be from a limited set of characters. Assuming we use ASCII or a code page where the English alphabet is used and hence each character is represented as a single byte, then we know that all of the above codes would take up 5 bytes of space in a file. In fact, if we use the 5 character limit and alpha-characters only as a rule on our data, is there anything we can do to save space when writing this data to a file or over the wire?

Considering we will only ever look to use a combination of 26 different characters, we could attempt to encode the alphabet with a different numbering system and therefore reduce the space used to store the data. Instead of the characters A-Z being a number represented by a single byte (8 bits = 255), we can map each letter to a different number and therefore assume the code is actually a number in a different base. e.g.

A single number is given to each letter of the English alphabet

With this numbering system, we are creating a base-26 numbering system. The number 0 is represented as A or AAAAA , the number 25 is represented by Z or AAAAZ and therefore larger numbers like 54321 could be represented as ADCJH .

If we take our first code AMZON we can see that this would be broken down as follows:

A M Z O N

0 * 26^4 + 12 * 26^3 + 25 * 26^2 + 14 * 26^1 + 13 * 26^0

Encoding all codes to a number in the same way we end up with the following:

Encoded codes

We can see from the table that at most there are only about 3 bytes of information. We can confirm this in 2 ways. Firstly, the largest number in our system (using max 5 chars) would be represented by the code ZZZZZ . The value of this is 11881375, which is 10110101 01001011 10011111 in binary, and will use 24 bits.

Secondly, we are using a base-26 numbering system. Therefore we can use mathematics to find out our largest number:

num = 26 ^ [num characters] − 1 = 26 ^ 5 − 1 = 11881375

Seeing as the maximum number that can be represented by 3 bytes is 2 ^ 24 — 1 = 16777215 then we are well within these limits.

So we’ve taken a 5 byte string code and stored it as a 3 byte integer. A space saving of 40%!!!!!

Side Note: The well known Base 64 encoding does not save space. It’s encoding mechanism is completely different, requires padding and will actually increase the data size.

Mathematics

In mathematical terms, we know that the largest number that can be represented in binary can be represented by:

Whatever character encoding mapping we use can be converted to a number that can be represented by

Therefore, our constraint is as simple as

Being clever, and with some simple logarithm rules, we can find the maximum number of characters that can be stored for a given character mapping.

As an example, if we were to take a dictionary size of 40 characters and we wanted to fit this into an Int32 (4 bytes) we would calculate that the maximum number of characters that can be in our code will be 6:

8 * 4 / log2(40) = 32 / 5.3219 = 6.0128

Implement Encoding/Decoding in Javascript

A simple overview of an encoding algorithm (to provide a number) would be as follows

1. Split the code into an array of each character

2. Iterate over the array and map to the relevant number in the mapping

3. Reduce the array into a single number by multiplying each number by a decreasing power of 26.

A simple implementation of this would be:

const encode = (code) => {

return code

.split("") // split into an array of each character

.map(c => c.charCodeAt(0) - 65) // map to number in mapping

.reduce((a,b,i)=>a + b * Math.pow(26, code.length - 1 - i),0)

The decoding of a number, back to a code string would be a little more involved. The algorithm would be as follows

1. Start with a remainder value equal to the number provided

2. Iterate from 4 to 0 (i)

a. For each i divide the remainder by 26^i and floor to the nearest integer (n)

b. Add the character corresponding to the n + 65 to the result string

c. Take (26 ^n) away from remainder

The implementation would then look like this:

const decode = (num) => {

let rem = 0 + Number(num);

let str = ""; for (let i = 4; i >= 0; i--) {

let n = Math.floor(rem / Math.pow(26, i));

str += String.fromCharCode(n + 65);

rem = rem - n * Math.pow(26, i);

}

return str

}

Here’s a codesandbox demo showing a full implementation of this working with debugging values.