fix two small bugs in encodeHash

I was usually testing with p==sp which caused the zeroTest to fail every time since shifting by 32 is effectively a no-op. This patch includes a check for this case.

Also the order of operations on the return value wasn't producing the intended result.

There is still a bug when reading the cardinality of a sparse set but that's less of an issue since it works fine after the conversion to normal.