preproc: rewrite digit value parsing/checks

This avoids some unnecessary calculations and fixes a bug where
isxdigit() was called on values not representing an unsigned char or
EOF.