test: fix floating point NaN tests on mips

MIPS machines use a slightly different format for NaNs (still perfectly
valid though). This patch adjusts the buffer testcases to allow for
this.

See https://en.wikipedia.org/wiki/NaN#Encoding for some more info.

Based on patch applied to debian by Jérémy Lal <kapouer@melix.org>

Reviewed-By: Fedor Indutny <fedor@indutny.com>
PR-URL: https://github.com/joyent/node/pull/8793