harness: fix lots of -Wincompatible-pointer-types and -Wpointer-sign warnings
2.3307: add binary, allow hex and octal with binary

write and accept \xNN JSON encoding with binary only,
also accept octal \0NNN JSON with binary
test fixes for 5.6, update README