expt: avoid undefined integer overflow in fixnum implementation

Also, extend fixnum fast path to work more often in 64-bit mode.

More review would be appreciated to help ensure that the revised
implementation avoids undefined behavior in C.