An In-Depth Introduction to Sparse Matrix
By: Edward Cui
Tags: Scipy, Sparse Matrix, Data Structures, Python, Recommender Systems

A comprehensive comparison of various sparse matrix formats

Recently, as I dig deeper into recommender systems, I found that the sparse matrix is the best data structure to represent the commonly used user-item matrix. In this review blog post, I will introduce scipy.sparse in-depth, and share some lessons I have learned so far.

Motivation to use sparse matrix

In recommender systems, we have the data on a rating of a user on a specific item. For example, on Amazon, a buyer may have a 4.5-star rating on the book Gregory Mankiw’s Principles of Economics. Given there are millions or even billions of users, and hundreds of thousands of book titles on Amazon, it is not possible for every user to give a rating on every single book. Often times, we will have most of the users rating on several books, and most of the books will only have some tens to hundreds of users to rate it. If we imagine this as a matrix where the rows represent the user ids, whereas the columns represent the book id (i.e. item), then we have a very large matrix, but only with a small percentage of it filled in. Most of the entries in this matrix will be empty. This matrix, with most of the values missing or empty, will be called a sparse matrix. The task of a recommender system is often to predict the ratings on these missing entries in order to make recommendations to users on new items.

Example user-item matrix encountered in recommender systems

We can, in theory, fill all these missing values with zeros, and use numpy.ndarray to pandas.DataFrame to represent it. But every 0 we store in our memory or disk is probably a waste of memory or storage space. Also, if we fill these missing values with 0, then we wouldn't be able to distinguish cases where a user literally rate a book with 0 stars (although most rating systems probably do not allow it, perhaps also consider other types of data with potentially 0 values distinguished from missing values). Therefore, it probably makes more sense to store only the entries of the matrix where it really has some data. Sparse matrix data structures are designed exactly for this purpose.

Scipy’s sparse matrix class

scipy.sparse has 7 different sparse matrix structures, each has its pros and cons in its use in practice. All 7 types are inter-convertible from each other. Some are good for slicing the matrix and do arithmetic operations on, while others are better used for constructing the data matrix from other data structures. We will analyze each one of them.

Compressed Sparse Column Matrix ( csc_matrix)

csc_matrix is good for column-wise slicing and fast arithmetics including + , - , * , / , power ** and matrix multiplication @ . csc_matrix is not efficient when adding a new element to the data, nor it is the best choice for constructing a new sparse matrix.

csc_matrix stores 3 pieces of information regarding the sparse matrix, the data itself as an array ( data ), the indices of the rows ( indices ), and the indirect register pointer ( indptr ) with respect to the indices . As an example, suppose we have the following sparse csc matrix

Then its csc representation is

csc_matrix example

data is simply a column-wise flattened version of the matrix

is simply a column-wise flattened version of the matrix indices is the row indices for the corresponding elements of the data , e.g., the first element in the data is a 1, and it is located in the row index 1(second row); the second element in the data is a 2, and the row index is 3(fourth row), etc...

is the row indices for the corresponding elements of the , e.g., the first element in the data is a 1, and it is located in the row index 1(second row); the second element in the data is a 2, and the row index is 3(fourth row), etc... indptr is a pointer for data and indices , an array of length col + 1, with max of its element as the length of the data or indices . It is representing the range of column indices that each element in the data belongs to. In the above example, we have

Column 0 has data and indices between indices A.indptr[0]=0 and A.indptr[0+1]=1 , i.e. A.data[0:1]=1 and A.indices[0:1]=1 Column 1 has data and indices between indices A.indptr[1]=1 and A.indptr[1+1]=2 , i.e. A.data[1:2]=2 and A.indices[1:2]=3 Column 2 has data and indices between indices A.indptr[2]=2 and A.indptr[2+1]=4 , i.e. A.data[2:4]=[1,2] and A.indices[2:4]=[2,4] Column 3 has data and indices between indices A.indptr[3]=4 and A.indptr[3+1]=6 , i.e. A.data[4:6]=[2,1] and A.indices[4:6]=[0,4] Column 4 has data and indices between indices A.indptr[4]=6 and A.indptr[4+1]=7 , i.e. A.data[6:7]=2 and A.indices[6:7]=2

Compressed Sparse Row Matrix (csr_matrix)

csr_matrix is good for row-wise slicing and like csc_matrix , fast for arithmetic operations including + , - , * , / , power ** and matrix multiplication @ . csr_matrix is also not efficient when adding a new element to the data.

The data structure of csr matrix is similar to that of csc matrix, except that the indices indicates the indices of the columns. For the same example matrix, the csr representation is

where

data is a row-wise flattened array

is a row-wise flattened array indices are indices of the columns

are indices of the columns indptr is a pointer for data and indices , an array of length row + 1, with the max of its element as the length of the data or indices .

csr_matrix example

Block Sparse Row matrix (bsr_matrix)

bsr_matrix has all the advantages of csr_matrix . In addition, when data is dense at certain regions of the matrix, arithmetic operations can be even more efficient than csr_matrix . To efficiently store the sparse matrix as bsr format, it is required that the block size are all the same, and are divisible by the number of rows and columns, i.e. if the block size is (R, C) and matrix size is (M, N), then it is required that M % R = 0 and N % C = 0.

The bsr sparse matrix representation has the following structure:

data : a K × R × C matrix, where K is the number of blocks, and R and C are dimensions of each block

: a K × R × C matrix, where K is the number of blocks, and R and C are dimensions of each block indices : length K array, column indices of the blocks

: length K array, column indices of the blocks indptr : length K+1 array, pointers of block data and indices

For the following example matrix, the bsr representation is

bsr_matrix example

Coordinate Sparse Matrix (coo_matrix)

coo_matrix is the best and fastest format for constructing a new sparse matrix using large arrays and row/column indices. Once the sparse matrix is constructed, one can easily cast it from coo to other sparse matrix formats.

A feature of coo_matrix format when constructing a sparse matrix is that it allows repeated rows and columns. When converting to other sparse matrix formats such as csc_matrix or csr_matrix , the duplicated (row, col) entries will be summed together. This is especially helpful when constructing finite element matrices.

To construct a sparse matrix without summing the duplicated entries, one can use dok_matrix .

For the following example matrix, the coo_matrix structure is

coo_matrix example

Dictionary of Keys Sparse Matrix (dok_matrix)

dok_matrix is the best format to use when we try to update/replace the elements in a sparse matrix. It is basically a dictionary of the format {(row, col): data, ...} .

keys() : this method will return the (row, col) tuples which index the data

: this method will return the (row, col) tuples which index the data values() : this method will return the data stored

For the following matrix, the dok_matrix 's equivalent dict structure is

dok_matrix example

Row-based List of List Sparse Matrix (lil_matrix)

lil_matrix is also a good format to construct a sparse matrix, but only if the matrix does not involve a lot of data. For larger data construction, consider using coo_matrix . lil_matrix stores the following two pieces of information about the sparse matrix

rows : lists of column indices for each row.

: lists of column indices for each row. data : lists of list of row elements.

For the following example matrix, when converting to lil sparse matrix, the structures are

lil_matrix example

lil_matrix offers a very simple and flexible interface to slice data. A key advantage of using lil format to slice the matrix is that it allows row and column indices to be broadcasted.

Suppose we have an array of row indices rows and column indices cols , to slice out a subset of the sparse matrix for every rows and cols , we can do

Diagonal Storage Sparse Matrix (dia_matrix)

dia_matrix is the best format to use when constructing diagonal matrices, or off-diagonal matrices. It stores 2 pieces of information about the sparse matrix.

offsets : an array of diagonal offsets. 0 means at the main diagonal of the matrix; a negative value means below the main diagonal; a positive value means above the main diagonal. The array is usually sorted ascendingly (e.g. below main diagonal, main diagonal, above main diagonal)

: an array of diagonal offsets. 0 means at the main diagonal of the matrix; a negative value means below the main diagonal; a positive value means above the main diagonal. The array is usually sorted ascendingly (e.g. below main diagonal, main diagonal, above main diagonal) data : K × D matrix, where K is the number of diagonal terms, and D is the size fo the main diagonal. Note that if the diagonal is at the off-diagonal (i.e. not at the main diagonal), then

If this is diagonal is below the main diagonal (offset < 0), then trailing zeros will be added to the array to match to the size of the main diagonal array If this is diagonal is above the main diagonal (offset > 0), then leading zeros will be added to the array to match to the size of the main diagonal array These zeros values can actually be any other values, but it won’t be part of the matrix. In the scipy.sparse.dia_matrix document example, the dia matrix was constructed using 3 identical length vectors, placed at the diagonals [-1, 0, 1] . Although, when accessing the data via A.data , it was shown that there were no zeros filled right after the initial construction, casting first to coo_matrix then back to dia_matrix and access data with A.data again will show these leading and lagging zeros.

For the following example matrix, when converting to lil sparse matrix, the structures are

dia_matrix example

Summary

The following table summaries the ways to construct and update the 7 sparse matrices. For dok_matrix , lil_matrix , and dia_matrix , I have included some in-object methods to update the matrix, which are not found in scipy ’s documentation.

Construction and Update of 7 Sparse Matrix Formats and Inter-conversion

The following table makes a comparison of the performance of several types of operations for each sparse matrix. In general, csc_matrix , csr_matrix , and bsr_matrix are great for arithmetic operations, while others are better for matrix construction.

Comparison of all 7 sparse matrix data structures in a table

Sparse Matrix Broadcasting

There are many cases where we want to add, subtract, or normalize over columns, rows, or the entire matrix, without affecting the missing data. In this section, we are going to consider several cases of sparse matrix broadcasting.

Scalar Broadcasting

Unlike numpy , simply adding a scalar on the sparse matrix with mat_csc + 1 will give us the error

NotImplementedError: adding a nonzero scalar to a sparse matrix is not supported

Fortunately, the actual data of the matrix is stored in .data attributes as a numpy.ndarray . Therefore, we can simply modify the .data attributes,

mat_csc.data += 1

Row-wise broadcasting

Suppose we have the csr sparse matrix mat_csr of size $M \times N$ and a row vector array_r of length $M$. We want to add (or subtract or element-wise multiplication or subtraction) each row of the sparse matrix by the row vector. In numpy , we can simply do

mat + array_r[np.newaxis, :]

With Sparse matrix, we again can access the .data attribute. We can also access the column indices of each data point via the .indices attribute, then the equivalent sparse matrix row-wise broadcasting is then

mat_csr.data += np.take(array_r, mat_csr.indices)

For other types of sparse matrices, first, convert to csr_matrix with tocsr() , make the broadcasting calculation, and then convert it back to the original format.

This can be implemented as

Column-wise broadcasting

Similarly, for column-wise broadcasting, suppose we have csc matrix mat_csc of size M × N and a vector array_c of size N. If we want to broadcast the vector to all the columns of the sparse matrix, we can do

mat_csc.data += np.take(array_c, mat_csc.indices)

Again, for other types of sparse matrices, first, convert to csc_matrix with tocsc() , make the broadcasting calculation, and then convert it back to the original format.

This can be implemented as

Sparse multiplication

In matrix factorization problems, we often need to compute a loss function after the prediction by multiplying the two factorized dense matrices. But since the loss function should only be calculated where the data is available on the large sparse matrix, it is only necessary to make the multiplication at a small subset of combinations of rows, instead of the full matrix multiplication.

Suppose we have two dense matrices U (N × K) and W (M × K) and suppose N and M can be very large, but K is very small. We have a user-item rating sparse matrix R, and we are trying to use the product

to predict R. Since R is sparse, it is better that predicted R is also sparse to avoid unnecessary computations.

A trick is to use coo_matrix to determine which combinations of rows and columns of R needs to be predicted. Then slice out only the necessary rows from the two dense matrices (for necessary rows of U, they would be the rows of R; for necessary rows of W, they would be the columns of R). After selecting only the necessary rows, we can do dot products on these sliced set of row vectors. The following snippets implement this.

Sparse Matrix in Other Python Libraries

In addition to scipy 's sparse matrix implementations, other Python libraries also have sparse data structure implemented, including but not limited to:

pandas : implemented as types SparseArray and SparseDataFrame , similar to coo, and interfaces with scipy.sparse.coo_matrix very well

conversion from pandas to scipy.sparse.coo_matrix with df.sparse.to_coo() conversion to pandas from scipy.sparse.coo_matrix with pd.Series.sparse.from_coo()