Don't compute decay_factor ** count when not needed anymore

Otherwise we eventually get an OverflowError. This fixes issue #479.