Fix NaN warnings and failures in test_serialize

don't use empty, which can create a bunch of NaNs,
causing equality tests to fail.
Merge pull request #6071 from minrk/serialize-nan

Fix NaN warnings and failures in test_serialize