Backport of performance regression fix to 0.14.1

This backports the `scipy.sparse` performance regression fix from gh-4281 (see issue gh-4203) to 0.14.1.

Ping @pv.