isprime: fix potential mulmod overflow exhibited by isprime(Uint64)