Wouldn't it be more efficient to use the dia_matrix format? Both in terms of memory usage and in terms of the final multiplication.