Erasure Coding For The Masses
By: Vishesh Khemani
Tags: Getting Started, Editors Pick, Data Science, Software Engineering, Mathematics

How Can You Use Linear Algebra To Erasure Code?

If you want to erasure code your data, you have to dive into the “More Math…” part in the subtitle of this article. To make the mathematical concepts more accessible, you’ll build up an example [3, 2] erasure code to encode a melody written in the 7-letter musical alphabet from A to G. The small size of the alphabet makes it easier to illustrate the math involved without getting bogged down in large numbers. We’ll encode the melody for the line “Happy birthday to you”, which is “CCDCFE”.

If you want to [N, K] erasure-code your data, you start by processing the source data stream in fixed-size blocks of data. Each data-block consists of K equal-sized chunks. Figure 4 illustrates how this chunking and blocking works for the example melody data.

Figure 4 (image by author)

Since each chunk has a fixed size, you can define a numerical representation for the chunks. Then, each data-block specifies a K-component vector in a K-dimensional vector space. Figure 5 shows the numerical representation and data-block vectors for the example.

Figure 5 (image by author)

Erasure coding is a linear transformation from each K-component data-block vector to its corresponding N-component code-block vector. The code-block chunks are striped across the N encoded shards i.e. each code-chunk is appended to the corresponding shard. The linear transformation of a data-block into a code-block can be represented by an N by K matrix.

Will any N by K encoder matrix work? Of course not. The matrix must have one crucial property. If you want to reconstruct the K chunks of a data-block from any K of N chunks of the corresponding code-block, you need every K by K submatrix of the encoder matrix to have an inverse matrix. Each such inverse matrix is the decoder matrix for the corresponding K chunks of a code-block.

How can you construct an N by K matrix such that every K by K submatrix has an inverse? Recall that a square matrix has an inverse if and only if its determinant is non-zero. So the encoder must have a non-zero determinant for every K by K submatrix. One way to achieve this is to use a Cauchy matrix (https://en.wikipedia.org/wiki/Cauchy_matrix). How? Choose a set of N distinct numbers {x₁, x₂, …, xₙ} and a non-overlapping set of K distinct numbers {y₁, y₂, …, yₖ}. Then, the matrix element in the ith row and kth column is 1/(xᵢ-yⱼ). The determinant of the K by K square submatrix with the K rows {r₁, r₂, …, rₖ} selected is non-zero and is given by the following:

We covered a lot of ground above. Let’s illustrate all of that in our example of the “Happy birthday” melody. Figure 6 constructs a 3 by 2 Cauchy encoder matrix for the [3, 2] erasure code used in the example.

Figure 6 (image by author)

Figure 7 shows how the vectors for the 3 data-blocks (“CC”, “DC”, “FE”) are encoded into 3 code-blocks (each with 3 chunks) and how the chunks in each code-block are striped across the encoded shards.

Figure 7 (image by author)

Figure 8 shows all the decoder matrices for the example, for each of the 3 ways in which 2 shards can be picked out of 3.

Figure 8 (image by author)

Figure 9 shows how each pair of shards decodes into the source data using the corresponding decoder matrix.

Figure 9 (image by author)

There you have it. You now know how to use Linear Algebra to erasure code your data. To recap the main ideas, an [N, K] erasure code maps a K-chunk data-block in a K-dimensional vector space to an N-chunk code-block in a K-dimensional subspace of an N-dimensional vector space. The code-chunks in a code-block are striped across the code shards. The map is specially constructed, using a Cauchy matrix, to be reversible for any K-chunk subset of the N-chunk code-blocks, allowing the source data to be decoded from any K code shards.