Fix crash when passing invalid hex strings to cipher.update or decipher.update

Fixes issue #5723.