Revert "BLD: special: fix build when system doesn't have isfinite macro"

This fix broke windows builds.

This reverts commit 091e6fc93d90a9c63ecdbab2bbbee1673902eb1f.
BUG: fix build on RH5.
Merge pull request #2833 from cournape/fix_windows_build

BUG: special: Fix windows build