Can we construct the sparse matrix directly here?