Better fix - just check that the upper and lower nibbles are >= 0xA.

An unsigned 8-bit integer is always <= 0xFF; that was the cause of the
warning.

To see whether a byte, when represented as hex, has letters for its
upper and lower nibbles, just check whether both nibbles are >= 0xA.
Cast the extracted nibbles to make sure there's no sign-extension.

Change-Id: If4c7717a5d2fe341c02e9309ee6b89973a6ac292
Reviewed-on: https://code.wireshark.org/review/4739
Reviewed-by: Guy Harris <guy@alum.mit.edu>
(cherry picked from commit f6bc6f957a13471fecb4ce0a31ca09f30cdde44f)
Reviewed-on: https://code.wireshark.org/review/4740