lib: remove and restructure calls to isNaN()

In #7840, @bnoordhuis pointed out a simple optimization surrounding a call to `isNaN()`. This commit implements that optimization. It also removes two unnecessary calls to `isNaN()` that are already covered by calls to `isFinite()`. Closes #7840.

@trevnorris

As a side note, there appear to be two tests that are failing in master.