Add stack overflow check for serialization to test/misc.lua