Only convert array if needed
Merge pull request #4283 from larsmans/backport-gh-4281

BUG: sparse: backport of performance regression fix to 0.14.1