BUG: Revert PR #4421: causes SciPy tests to fail

Reverts #4421. SciPy tests fail when built with NumPy after the merge.