Less math in the tight loop

There's probably more to be gained here