Fixed unexpected non-RSA verification failures.

This fix ensures that the RSA verification is only used for RSA signatures.

joyent/node#4771