Use isxdigit.

Thanks to Benjamin Kramer for the suggestion.