lib: fix conversion of hex char to int and add test

Signed-off-by: Alexis Laferrière <alexis.laf@xymus.net>