Note that removing this `toarray` makes `asarray(b)` produce an object array and fail below.
The tests should probably check all of the following permutations:

(i) b ndarray vector
(ii) b sparse vector
(iii) b ndarray non-vector
(iv) b sparse non-vector