Repository: pytorch/pytorch

Issue #63152: torch.fft.rfft bug
Created: 2021-08-12T16:55:26Z
🐛 Bug
When performing zero padding followed by rfft on cuda, incorrect results can happen on the second operation when padded length is even for longer sequences.  This behavior is observed on cuda only, not cpu.
To Reproduce
Steps to reproduce the behavior:
import torch
import torch.fft as fft
import numpy as np
from scipy.fft import rfft


if __name__ == '__main__':
    dtype = torch.float64
    for device in ['cuda', 'cpu']:
        print('\ndevice={}'.format(device))
        Ly = 1600
        y = torch.randn(1, 1, Ly, dtype=dtype, device=device)
        for Lx in [160, 161, 1600, 1601, 16000, 16001]:
            x = torch.randn(1, 1, Lx, dtype=dtype, device=device)
            x = torch.nn.functional.pad(x, [0, Ly-1])
            yy = torch.nn.functional.pad(y, [0, Lx-1])
            X = fft.rfft(x)
            Y = fft.rfft(yy)
            X_s = rfft(x[0, 0, :].cpu().numpy())
            Y_s = rfft(yy[0, 0, :].cpu().numpy())
            dif_x = np.max(np.abs(X_s - X[0, 0, :].cpu().numpy()))
            dif_y = np.max(np.abs(Y_s - Y[0, 0, :].cpu().numpy()))
            print('Lx={}: dif_x={}, dif_y={}'.format(Lx, dif_x, dif_y))


Output:
device=cuda
Lx=160: dif_x=6.080392413311918e-12, dif_y=1.9477723502559983e-11
Lx=161: dif_x=1.4273168140722167e-14, dif_y=3.972054645195637e-14
Lx=1600: dif_x=8.206403321718826e-14, dif_y=113.69819800575286
Lx=1601: dif_x=5.0242958677880804e-14, dif_y=4.832213032818602e-14
Lx=16000: dif_x=3.0511999048007806e-13, dif_y=121.9687795227097
Lx=16001: dif_x=2.3198975494145925e-13, dif_y=5.54951617168815e-14

device=cpu
Lx=160: dif_x=2.416106516409301e-14, dif_y=7.105427357601002e-14
Lx=161: dif_x=1.1916163935586911e-14, dif_y=4.266955553143359e-14
Lx=1600: dif_x=7.105982447430966e-14, dif_y=9.500188217416325e-14
Lx=1601: dif_x=4.4938668397781776e-14, dif_y=4.278033777464534e-14
Lx=16000: dif_x=2.5579538487363607e-13, dif_y=7.8482010346102e-14
Lx=16001: dif_x=2.744571917763205e-13, dif_y=7.791709393054201e-14


Expected behavior
dif_y for Lx=1601 and 16001 for cuda run should be around 1e-14.
Environment
Collecting environment information...
PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.10
Python version: 3.7.6 (default, Jan  8 2020, 19:59:22)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.19.56-1.el7.x86_64-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
Nvidia driver version: 418.40.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] pytorch-ignite==0.2.0
[pip3] pytorch-lightning==1.1.1
[pip3] torch==1.9.0+cu111
[pip3] torchaudio==0.9.0
[pip3] torchinfo==1.5.3
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.10.0+cu111
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] pytorch-ignite            0.2.0                    pypi_0    pypi
[conda] pytorch-lightning         1.1.1                    pypi_0    pypi
[conda] torch                     1.9.0+cu111              pypi_0    pypi
[conda] torchaudio                0.9.0                    pypi_0    pypi
[conda] torchinfo                 1.5.3                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.10.0+cu111             pypi_0    pypi
cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @ngimel @mruberry @peterbell10 @walterddr

Duplicate issue #67140: cufft cache makes rfft with cuda tensors to return incorrect results in subsequent calls. 
Created: 2021-10-23T09:48:47Z
🐛 Bug
rfft returns incorrect results in subsequent calls in CUDA. This seems to be related to the cufft cache.
Clearing the cache with torch.backends.cuda.cufft_plan_cache.clear() makes the calculation to be performed correctly
.
To Reproduce
Steps to reproduce the behavior:

Run fft with a cuda tensor twice
The second (and all subsequent results) will be different fromt the first (correct) result


import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
from torch.fft import rfft, irfft
device = 'cuda'
pdf = torch.from_numpy(
    norm.pdf(np.linspace(-10, 10, 1000).reshape(-1,1))
).to(device)
n_iter = 50
n_full = pdf.shape[0] * n_iter - n_iter + 1
fft0 = rfft(pdf, axis=0, n=n_full)
fft1 = rfft(pdf, axis=0, n=n_full)
plt.plot(np.abs(fft0.cpu()[:, 0]), label='fft0')
plt.plot(np.abs(fft1.cpu()[:, 0]), label='fft1')
plt.legend()
plt.title(f'On {device}')

Expected behavior
Both resutls should be equal. When device='cpu' the result is the expected one.
Environment
PyTorch version: 1.9.1
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A
OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A
Python version: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19042-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 970
Nvidia driver version: 496.13
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.9.1
[pip3] torchaudio==0.9.1
[pip3] torchvision==0.2.2
[conda] _pytorch_select           0.1                       cpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               11.1.1               heb2d755_9    conda-forge
[conda] libmklml                  2019.0.5             haa95532_0
[conda] mkl                       2019.4                      245
[conda] mkl-service               2.3.0            py38h196d8e1_0
[conda] mkl_fft                   1.3.0            py38h46781fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] numpy                     1.19.2           py38hadc3359_0
[conda] numpy-base                1.19.2           py38ha3acd2a_0
[conda] pytorch                   1.9.1           py3.8_cuda11.1_cudnn8_0    pytorch
[conda] torchaudio                0.9.1                      py38    pytorch
[conda] torchvision               0.2.2                      py_3    pytorch
Additional context
On GPU:

On CPU:

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @ngimel

Closing comment (2021-11-01T17:11:50Z): Closing as duplicate of #63152