Numerical issue in torch.matmul

Xin-yang-Liu: <h3><a name="this-is-a-copy-from-the-github-issuehttpsgithubcompytorchpytorchissues60234-i-raised-1" class="anchor" href="#this-is-a-copy-from-the-github-issuehttpsgithubcompytorchpytorchissues60234-i-raised-1"></a>This is a copy from the Github <a href="https://github.com/pytorch/pytorch/issues/60234" rel="noopener nofollow ugc">issue</a> I raised.</h3>
<hr>

<p>I tried to utilize the boardcast feature in torch.matmul, but here are some numerical errors when certain types of sparse matrix is involved. To be clear, I did not use any sparse matrix format but most elements in the matrix <code>lapa_c</code> are zero (see the sample code I provide below).</p>
<h2><a name="to-reproduce-2" class="anchor" href="#to-reproduce-2"></a>To Reproduce</h2>

<pre data-code-wrap="Python"><code class="lang-Python">
lapa_c=Build_CoefficientMatrix([1/90,-3/20,3/2,-49/18],64,periodic=True)

x=torch.rand(2,64,1)
out1=torch.matmul(lapa_c,x)
out2=torch.matmul(lapa_c,x[0,:,0])
print(out1[0,0,0].item())
print(out2[0].item())
</code></pre>
<p>The <code>Build_CoefficientMatrix</code> is used to build a 64*64 (with some non-zero elements in the bottom left and upper right corner when <code>periodic=True</code>)</p>
<pre data-code-wrap="Python"><code class="lang-Python">def Build_CoefficientMatrix(c:list,meshx:int,periodic:bool=False):
    '''
    c is list of FD coefficient
    e.g. for 1st derivative with 2nd accuracy central difference:
    c=[-0.5,0] 
    '''
    if 2*len(c)-1&gt;=meshx: raise ValueError
    acc = len(c)   
    
    tmp=[]
    c.reverse()
    for i in range(acc):
        x = torch.cat((torch.cat((torch.zeros((i,meshx-i)),
                                    c[i]*torch.eye(meshx-i)),dim=0),
                                    torch.zeros((meshx,i))
                                    ),dim=1)
        tmp.append(x)
    re=tmp[0]
    for k in tmp[1:]:
        re+=k+k.T

    if periodic:
        re[:acc,-acc:]=re[acc:2*acc,:acc]
        re[-acc:,:acc]=re[:acc,acc:2*acc]
    return re
</code></pre>
<h2><a name="expected-behavior-3" class="anchor" href="#expected-behavior-3"></a>Expected behavior</h2>
<p>The two printed outputs are supposed to be the same, but actually they are not.</p>
<h2><a name="if-you-change-4" class="anchor" href="#if-you-change-4"></a>if you change:</h2>
<ul>
<li>the batch size of <code>x</code> back to <code>1</code>(<code>x=torch.rand(1,64,1)</code>)</li>
<li>the <code>periodic</code> parameter in <code>Build_CoefficientMatrix</code> to <code>False</code></li>
<li>the <code>c</code> parameter in <code>Build_CoefficientMatrix</code> to a list with length no greater than 3</li>
</ul>
<p>the two results match each other.</p>
<p>According to <strong><a href="https://github.com/albanD" rel="noopener nofollow ugc">albanD</a></strong> in this pytorch/issues/58128.  <em>New users can only put 2 links…</em></p>
<blockquote>
<p>when you change the input size, the low level compute libraries that we use might choose a different algorithm which can lead to small difference of the order of 1e-6 for each single op (as per the floating point standard). It’s not something we can change.</p>
</blockquote>
<p>But for me, I need to iteratively use torch.matmul for hundreds of times (e.g. in reinforcement learning, a single episode contains hundreds of steps), and the accumulated error will be unacceptable. I just wonder if it is possible to add an option to force the low level compute libs use the  same algorithms (though it may not be the most appropriate one).</p>
<h2><a name="environment-5" class="anchor" href="#environment-5"></a>Environment</h2>
<p>PyTorch version: 1.8.1<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.2 LTS (x86_64)<br>
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0<br>
Clang version: Could not collect<br>
CMake version: version 3.16.3<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Python platform: Linux-5.8.0-55-generic-x86_64-with-glibc2.10<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: GeForce RTX 3070<br>
Nvidia driver version: 460.80<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.2<br>
[pip3] numpydoc==1.1.0<br>
[pip3] torch==1.8.1<br>
[pip3] torch-cluster==1.5.8<br>
[pip3] torch-geometric==1.6.3<br>
[pip3] torch-scatter==2.0.5<br>
[pip3] torch-sparse==0.6.8<br>
[pip3] torch-spline-conv==1.2.0<br>
[pip3] torchaudio==0.8.0a0+e4e171a<br>
[pip3] torchvision==0.9.1<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               11.1.1               h6406543_8    conda-forge<br>
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
[conda] mkl                       2020.4             h726a3e6_304    conda-forge<br>
[conda] mkl-service               2.3.0            py38h1e0a361_2    conda-forge<br>
[conda] mkl_fft                   1.3.0            py38h5c078b8_1    conda-forge<br>
[conda] mkl_random                1.2.0            py38hc5bc63f_1    conda-forge<br>
[conda] numpy                     1.19.2           py38h54aff64_0<br>
[conda] numpy-base                1.19.2           py38hfa32c7d_0<br>
[conda] numpydoc                  1.1.0                      py_1    conda-forge<br>
[conda] pytorch                   1.8.1           py3.8_cuda11.1_cudnn8.0.5_0    pytorch<br>
[conda] torch-cluster             1.5.8                    pypi_0    pypi<br>
[conda] torch-geometric           1.6.3                    pypi_0    pypi<br>
[conda] torch-scatter             2.0.5                    pypi_0    pypi<br>
[conda] torch-sparse              0.6.8                    pypi_0    pypi<br>
[conda] torch-spline-conv         1.2.0                    pypi_0    pypi<br>
[conda] torchaudio                0.8.1                      py38    pytorch<br>
[conda] torchvision               0.9.1                py38_cu111    pytorch</p>