OFStateManager: change gentable checksum operation

Add/subtract has advantages over XOR in that two adds of the same value don't
cancel each other out, and the high bits are better utilized.

128 bit arithmetic is done by manually checking for overflow/underflow. This
could be optimized by using a 128 bit type if compiling with GCC.