[ia32/x64] Smaller instruction to check NaN

substract 1 and test for overflow

BUG=
R=jkummerow@chromium.org

Review URL: https://codereview.chromium.org/202083002

Patch from Weiliang Lin <weiliang.lin@intel.com>.

git-svn-id: https://v8.googlecode.com/svn/branches/bleeding_edge@20185 ce2b1a6d-e550-0410-aec6-3dcde31c8c00