Common: Fix AsciiToHex returning true on overflow values

We should be checking `errno` against `ERANGE`.