Fix parsing of hex and octal integer numbers

The code for detecting hex/octal was correct, but the "is_hex" and
"is_octal" flags were not set properly. Also, added some additional
asertions in the code to check for mutually exclusive options (e.g.
a floating point number cannot be hex, and so on).
Fix hex number constant in tests