Revert "BLD: special: fix build when system doesn't have isfinite macro"

This fix broke windows builds.

This reverts commit 091e6fc93d90a9c63ecdbab2bbbee1673902eb1f.
(cherry picked from commit e783ab9cf93d7dd52d1e4d773a846c2ad944c13c)
BUG: fix build on RH5.
(cherry picked from commit a3dd45fea2bbe0e417bd4748703532a9fc2db1e3)