Backport of performance regression fix to 0.14.1

This backports the `scipy.sparse` performance regression fix from gh-4281 to 0.14.1.

Ping @pv.