Torch.rfft vs scipy.fftpack.fft

username: JuanFMontesinos
cooked: <p>Hi,<br>
I was wondering why torch rfft doesn’t match the one of scipy:</p>
<pre><code class="lang-auto">import torch
import numpy as np

from scipy.fftpack import fft


@torch.no_grad()
def _fix_shape(x, n, axis):
    """ Internal auxiliary function for _raw_fft, _raw_fftnd."""
    s = list(x.shape)

    index = [slice(None)] * len(s)
    index[axis] = slice(0, s[axis])
    s[axis] = n
    z = torch.zeros(s, dtype=x.dtype, device=x.device)
    z[tuple(index)] = x
    return z


N_FFT = 512
cuda = torch.rand(1, 44000, device='cuda:0')
tensor = cuda.cpu()
numpy = tensor.clone().numpy()
cuda_fft = torch.rfft(cuda, signal_ndim=1, onesided=False)[0, :, 0]

tensor_fft = torch.rfft(tensor, signal_ndim=1, onesided=False)[0, :, 0].numpy()
numpy_fft = np.real(fft(numpy, axis=1, n=None))[0]
print(f'Tensor type:{tensor_fft.dtype}\n'
      f'Numpy array type: {numpy_fft.dtype}'
      )
print(f'CPU-CUDA: {np.abs(tensor_fft - cuda_fft).sum()}')
print(f'CPU-NP: {np.abs(tensor_fft - numpy_fft).sum()}')
print(f'CUDA-NP: {np.abs(cuda_fft - numpy_fft).sum()}')
</code></pre>
<p>The error is not extremely big but big enough not to allow to use them indifferently.</p>
<pre><code class="lang-auto">Tensor type:float32
Numpy array type: float32
0.32703477144241333
</code></pre>

username: JuanFMontesinos
cooked: <p>In fact not even cpu and gpu versions match</p>
<pre><code class="lang-auto">Tensor type:float32
Numpy array type: float32
Numpy array type: torch.float32
CPU-CUDA: 0.5086007714271545
CPU-NP: 0.3293250799179077
CUDA-NP: 0.37688499689102173
</code></pre>

username: tom
cooked: <p>Yeah, but if you use mean instead of sum, you’ll get 1e-5ish for float32 and 1e-14ish for float64, so it’s different implementations matching up to numerical accuracy.</p>

username: JuanFMontesinos
cooked: <p>Hmmm, Soo the thing is I was porting MIR eval library to pytorch and I don’t really know if a tol 1e-5 is good enough as the final result diverges.</p>
<p>Shouldn’t it be way smaller, like 1e-10 or 1e-15?</p>

username: tom
cooked: <p>No! fp32 has as <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format#Converting_from_single-precision_binary_to_decimal" rel="noopener nofollow ugc">23 bit significand, so the lowest bit is about 2^(-23)</a>~10^(-7) in magnitude. This is roughly the relative precision of the representation. If you now compute with that, you lose some accuracy, and 1e-5ish is relatively reasonable. It also matches my experience in general.</p>

username: JuanFMontesinos
cooked: <p>Thank you <a class="mention" href="/u/tom">@tom</a>. It’s always nice to read you.<br>
So I’ll keep looking for bugs ^^</p>

username: tom
cooked: <p>Same! I was about to add a PS that it’s good to see you here but then didn’t want to make it sound as if you were gone. It’s an honor that you might have had a question that my modest knowledge of matters helps answer. <img src="https://discuss.pytorch.org/images/emoji/apple/bowing_man.png?v=9" title=":bowing_man:" class="emoji" alt=":bowing_man:"></p>

username: JuanFMontesinos
cooked: <p>Hahhaha don’t worry, I’ll be here for a couple of years at least. It’s just that questions are too complicated for me. I will have to read your blogpost in depth to follow the rythm.</p>