lib: remove and restructure calls to isNaN()

Switch condition order to check for `null` before calling `isNaN()`. Also remove two unnecessary calls to `isNaN()` that are already covered by calls to `isFinite()`. This commit targets v0.10, as opposed to #7891, which targets master (suggested by @bnoordhuis). Closes #7840.