Speed up copy_to_fft_buffer a bit