BLD: special: fix build when system doesn't have isfinite macro

(cherry picked from commit 091e6fc93d90a9c63ecdbab2bbbee1673902eb1f)