TST: reduce memory usage of fft test

tests should not use more than 500 MB of memory, this one uses more than
a GB. Also it didn't join its threads.
Merge pull request #4719 from juliantaylor/fft-test

TST: reduce memory usage of fft test