Weird outputs for F.conv2d

111479: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03fa7def1d940d38bb2b7a0900cbf56e2ab2ab0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/03fa7def1d940d38bb2b7a0900cbf56e2ab2ab0b" title="weired_conv2d"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03fa7def1d940d38bb2b7a0900cbf56e2ab2ab0b.png" alt="weired_conv2d" data-base62-sha1="zc6KAXdcPgkllx2DCT1dM94pLl" width="690" height="322" data-dominant-color="282828"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">weired_conv2d</span><span class="informations">997×466 12.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please refer to the attached image.</p>
<p>I had three non-negative tensors of float32 type, gamma: kernel for conv2d, beta: bias for conv2d, and v: an input.</p>
<p>When convolving seperately for each instance of minibatch v, I got normal outputs of non-negative values.</p>
<p>However, convolution over the entire v produced the weird outputs with a negative minimum.</p>
<p>Why did this happen?</p>
<p>Please help me!</p>
<p>Thanks.</p>
<p>========================================================<br>
<strong>Reproducing:</strong></p>
<p>Link for the data: <a href="https://drive.google.com/file/d/1jPmQMQFDGlY-7zsdZdWCJb-0TCIdh8df/view?usp=sharing" rel="noopener nofollow ugc">weird_conv2d.pt</a></p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

# loading
params = torch.load('./weird_conv2d.pt')
v = params['v']
gamma = params['gamma']
beta = params['beta']

# check the minimums
print('v', v.min().item())
print('gamma', gamma.min().item())
print('beta', beta .min().item())
''' outputs:
v 1.7985612998927536e-14 919835639808.0
gamma 0.0 0.007155213970690966
beta 9.999999974752427e-07 1.0796115398406982
'''

# strange outputs
print(F.conv2d(v, gamma, beta).min().item())
''' outputs:
-51.17139434814453
'''
for i in range(v.shape[0]):
  print(F.conv2d(v[i].unsqueeze(0), gamma, beta).min().item())
''' outputs:
1.6701818822184578e-05
1.5259909559972584e-05
3.263943290221505e-05
3.084764102823101e-05
0.008375695906579494
2.105729618051555e-05
2.6672159947338514e-05
2.3620612410013564e-05
'''
</code></pre>
<p>========================================================<br>
<strong>My environments:</strong></p>
<p>PyTorch version: 1.7.1+cu110<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.0<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 18.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: Tesla V100-SXM2-32GB<br>
GPU 1: Tesla V100-SXM2-32GB<br>
GPU 2: Tesla V100-SXM2-32GB<br>
GPU 3: Tesla V100-SXM2-32GB</p>
<p>Nvidia driver version: 450.51.06<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.20.1<br>
[pip3] pytorch-msssim==0.2.0<br>
[pip3] torch==1.7.1+cu110<br>
[pip3] torchaudio==0.7.2<br>
[pip3] torchvision==0.8.2+cu110<br>
[conda] numpy                     1.20.1                   pypi_0    pypi<br>
[conda] pytorch-msssim            0.2.0                    pypi_0    pypi<br>
[conda] torch                     1.7.1+cu110              pypi_0    pypi<br>
[conda] torchaudio                0.7.2                    pypi_0    pypi<br>
[conda] torchvision               0.8.2+cu110              pypi_0    pypi</p>

ptrblck: <p>Could you post your setup via: <code>python -m torch.utils.collect_env</code> as well as an executable code snippet to reproduce this issue?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

111479: <p>Please check the updated post.<br>
Thanks.</p>

ptrblck: <p>Thanks! I could reproduce this issue and guess an internal <code>matmul</code> kernel might run into an overflow.<br>
As a workaround, use the CUDA10.2 binaries, which ship with cudnn7.6.5.32 or alternatively you could also build PyTorch from source using cudnn8.1.<br>
I’ve forwarded the issue to the cudnn team, which will be looking into it. Since the latest cudnn release works fine, this issue might have been already fixed.</p>