When using sparse matrices, use sparse vectors for gradients only.