clean up tests a bit

don't use same buffer for encode and decode
add a test for decoding an ueven amount of hex characters

this test will fail
fix so we can decode a hex string with uneven amount of characters

shown broken in test in last commit
refactor to make hex and modhex use the same code
_decode() takes destination size as third parameter