Properly use strtoll for hex constants

Since we can't rely on strtod supporting hex constants, we obviously
need to use strtoll instead of using strtod.