Revert "Merge pull request #4421 from meltingwax/meltingwax/4382"

Caused SciPy tests to fail when built with this NumPy.