Error handling of binary/hexadecimal floating literals

See #4804 for details.