BUG: Revert PR #4382: broken and causes SciPy tests to fail

SciPy tests fail when built with NumPy after this commit.