Revert "Merge pull request #4421 from meltingwax/meltingwax/4382"

Caused SciPy tests to fail when built with this NumPy.
Merge pull request #4451 from meltingwax/meltingwax/revert-4421

BUG: Revert PR #4421: causes SciPy tests to fail