Only convert array if needed

This is a fix for https://github.com/scipy/scipy/issues/4203
I'll investigate more on what data types are used in this case and update accordingly. Note that this pull request is a response to this issue: https://github.com/scipy/scipy/issues/4203