crypto/rsa/rsa_chk.c: harmonize error codes.

Reviewed-by: Dr. Stephen Henson <steve@openssl.org>