Only convert array if needed

This is a fix for https://github.com/scipy/scipy/issues/4203
asarray(..., dtype=dtype) is better.
The arguments are always numpy arrays, so no helper function is needed.