Fix modulo performance

Thanks coyotte for hinting.
Note that the branch is remove by both GCC and Clang.