Only convert array if needed

This is a fix for https://github.com/scipy/scipy/issues/4203
> a bigger picture

I think the bigger picture is that scipy has no automated detection of performance regressions.