Return error when a bit string indicates an invalid amount of bits left

Reviewed-by: Matt Caswell <matt@openssl.org>