Backward pass of inverse FFT is sometimes incorrect on GPU
Labels: high priority, module: cuda, triaged, module: third_party, module: correctness (silent), module: fft

### 🐛 Describe the bug

On GPU, if you compute FFT of a tensor followed by inverse FFT of the tensor, the backward pass is (sometimes) incorrect. This only happens when the tensor does not have a gradient on it already, and when the inverse FFT is the last operation in a sequence before computing the backward pass.

I believe the problem is that the gradient of the input tensor is being initialized with the value of the tensor itself, instead of with zeros.

Here's a minimal example to reproduce it:

```Python
import torch

device = 'cuda'

x = torch.randn(32, dtype=torch.cfloat, device=device, requires_grad=True)
dout = torch.zeros(32, dtype=torch.cfloat, device=device)

# compute iFFT(FFT(x))
out = torch.fft.ifft(torch.fft.fft(x))
out.backward(dout, retain_graph=True)

print('Gradient of iFFT(FFT(x)) should be FFT(iFFT(dout))')
dx = torch.fft.fft(torch.fft.ifft(dout))

print('Difference between x.grad and what it should be. This should be zero!')
print((x.grad - dx).abs().max())

print('Difference between x.grad and x. This should be non-zero.')
print((x.grad - x).abs().max())
```

Expected output:
```
Gradient of iFFT(FFT(x)) should be FFT(iFFT(dout))
Difference between x.grad and what it should be. This should be zero!
tensor(4.7777e-07, device='cuda:0')
Difference between x.grad and x. This should be non-zero.
tensor(1.7879, device='cuda:0', grad_fn=<MaxBackward1>)
```

Actual output:
```
Gradient of iFFT(FFT(x)) should be FFT(iFFT(dout))
Difference between x.grad and what it should be. This should be zero!
tensor(1.7879, device='cuda:0')
Difference between x.grad and x. This should be non-zero.
tensor(4.7777e-07, device='cuda:0', grad_fn=<MaxBackward1>)
```

This bug goes away if the last operation is not an iFFT, or if `x.grad` is initialized before the first call to FFT.

For example, adding these lines before to call to `out = torch.fft...` fixes it:
```
x.backward(dout, retain_graph=True)
x.grad.data.zero_()
```

Switching the order of FFT/iFFT, or adding some other operation afterwards also both fix the bug:
* `out = torch.fft.fft(torch.fft.ifft(x))` is correct
* `out = torch.fft.ifft(torch.fft.fft(x)) * 2` is correct

This bug does not happen on CPU, so I suspect something is broken in the backward pass in C++/CUDA for the inverse FFT, in the case where the gradient on the input tensor is not initialized.

### Versions

I'm using NVIDIA PyTorch Docker image, version 23.05.

```
Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-4.19.0-24-cloud-amd64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          12
On-line CPU(s) list:             0-11
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
Stepping:                        7
BogoMIPS:                        4400.44
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       192 KiB (6 instances)
L1i cache:                       192 KiB (6 instances)
L2 cache:                        6 MiB (6 instances)
L3 cache:                        38.5 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-11
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Mitigation; Enhanced IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT Host state unknown

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] pytorch-fast-transformers==0.4.0
[pip3] pytorch-lightning==2.0.1.post0
[pip3] pytorch-quantization==2.1.2
[pip3] pytorch-ranger==0.1.1
[pip3] torch==2.0.0
[pip3] torch-optimizer==0.3.0
[pip3] torch-tensorrt==1.4.0.dev0
[pip3] torchdata==0.6.0
[pip3] torchmetrics==0.11.3
[pip3] torchtext==0.15.1
[pip3] torchvision==0.15.1
[pip3] triton==2.0.0.dev20221202
[conda] Could not collect
```

cc @ezyang @gchanan @zou3519 @kadeng @ptrblck @mruberry @peterbell10