Bail out early if we can't get a BIGNUM

Review: @JoelKatz @rec