HEX_ERRNO_SIZE is no longer the correct upper limit for format_hex_number_sigsafe
Lightly refactor and test format_hex_number_sigsafe

Better tests for upper bounds, and for failing cases.

Also, change the function's interface to take a buffer length rather
than a maximum length, and then NUL-terminate: functions that don't
NUL-terminate are trouble waiting to happen.