Only convert array if needed

This is a fix for https://github.com/scipy/scipy/issues/4203
`np.asarray` would convert a `np.matrix` to a `np.ndarray`, and it would do unspeakable things to a `scipy.sparse` array.