math/big: Allow non-prime modulus for ModInverse

The inverse is defined whenever the element and the
modulus are relatively prime.  The code already handles
this situation, but the spec does not.

Test that it does indeed work.

Fixes issue 8875

LGTM=agl
R=agl
CC=golang-codereviews
https://codereview.appspot.com/155010043