uat: tighten up dec/hex uat field validity checking

Specifically:
- Use the proper code for testing strtol() result;
- Also: Values greater than 32-bits treated as an error
   (on LP64 architectures);

Change-Id: I56e8e734fbb9a22dbd9ed4112e24327ffd7ee3c0
Reviewed-on: https://code.wireshark.org/review/5394
Reviewed-by: Bill Meier <wmeier@newsguy.com>
Petri-Dish: Bill Meier <wmeier@newsguy.com>
Tested-by: Petri Dish Buildbot <buildbot-no-reply@wireshark.org>
Reviewed-by: Alexis La Goutte <alexis.lagoutte@gmail.com>