Speed up (and test) sparse matrix-vector multiplication

Fixes #4707