Separate the extraction of hex values.

This way we can call it twice: first to count the size of output (and
sanity check the extraction indices), and the second time to actually
extract.  Otherwise with bad extraction code we will smash the stack.

Furthermore, this makes for cleaner code since the large #ifdef forest
(about differing long double formats, and different uv/nv/quad sizes)
is moved within the helper function.

Furthermore: IEEE 754 128-bit extraction was extracting wrong:
we need only the mantissa bits, and in the right nybble order.

Last but not least, handle the 'implicit bit' of IEEE 754 quadruple
and double-double (but which is explicitly *not* in the 80-bit format).
(If we handled the standard 64-bit format by explicit byte extraction,
instead of frexp+ldexp the hidden bit would be needed there, too.)