Fix NaN warnings and failures in test_serialize

Don't use empty, which can create a bunch of NaNs, causing equality tests to fail.