Fix 'r' is used uninitialized when compiled

When line:529 "if" condition is true, "r" is uninitialized and returned, causing compile error under LLVM

engine/serialize.c:529:6: error: variable 'r' is used uninitialized whenever 'if'
condition is true [-Werror,-Wsometimes-uninitialized]
if (exp_xsum != act_xsum)