test: fix floating point NaN tests on mips

MIPS machines use a slightly different format for NaNs (still perfectly valid though). This patch adjusts the buffer test cases to allow for this.

See https://en.wikipedia.org/wiki/NaN#Encoding for some more info.

This affects master and 0.12. It probably affects 0.10 as well but I didn't manage to build 0.10 on MIPS so I couldn't test it.