Avoid divide by 1 in fft for performance