MathLib:isHex(): fix detection of missing suffixes (U,L).

Implement a finite state machine to parse hex numbers. Added missing test cases.