Negative values with non-negative conv2d inputs

masteryoda: <p>Hi,</p>
<p>I’m trying to do conv2d with <strong>non-negative values</strong>, but getting <strong>negative values</strong> in the output.</p>
<p>z<code>1 = F.conv2d((input * input), sigma_square, None, self.stride, self.padding, self.dilation, self.groups)</code></p>
<p>The input pixels are x*x, and the weights are the variance which is non negative by definition (I also made sure it’s non-negative here) .</p>
<p>My guess it relates to  <strong>numerical error</strong> because the values are very close to zero (10e-5), but I’m not sure.</p>
<p>Any idea how I can solve it? (I have sqrt(z1) after the conv2d, so it can’t be negative).</p>

ptrblck: <p>Could you try to create random tensors, which would reproduce the issue, so that we could take a look at it, please?</p>

masteryoda: <p>I reproduced the case:</p>
<blockquote>
<p><a href="https://github.com/guyber9/repo" class="inline-onebox" rel="noopener nofollow ugc">GitHub - guyber9/repo</a></p>
</blockquote>
<p>You can find in the link 2 files:</p>
<ol>
<li>my_tensors.pt - which includes ‘x’ and ‘w’ tensors for the convolution</li>
<li>main.py - read the tensor files and doing the conv2d.</li>
</ol>
<p>Just run main.py and you’ll see the input is all positive but conv2d results include negative values.</p>

ptrblck: <p>Thanks for the code snippet and the inputs. Since I cannot reproduce the issue, we would need more information about your setup (PyTorch version, GPU, CUDA, cudnn etc.).<br>
Output:</p>
<pre><code class="lang-python">x is negative: tensor(False, device='cuda:0')
w is negative: tensor(False, device='cuda:0')
z (= Wx) is negative: tensor(False, device='cuda:0')
v is negative: tensor(False, device='cuda:0')
v isnan: tensor(False, device='cuda:0')
</code></pre>

masteryoda: <p>PyTorch version: 1.9.0+cu102<br>
NVIDIA GeForce RTX 2080 Ti<br>
CUDA Version 10.1.105</p>
<p>I’ve found something interesting. If I’m adding:</p>
<blockquote>
<p>torch.backends.cudnn.deterministic = True</p>
</blockquote>
<p>It solves the issue.</p>
<p>But when running with:</p>
<blockquote>
<p>cudnn.benchmark = True</p>
</blockquote>
<p>It’s bringing back the problem (even torch.backends.cudnn.deterministic = True).</p>

zfzhang: <p>Hi, I’m encountering the exact same problem.</p>
<p>My environment is:<br>
PyTorch 1.9.1+cu102<br>
GeForce 2080 Ti<br>
CUDA 10.2<br>
Win7</p>
<p>Your advice seems to work, and it is possibly related to FFT &amp; Winograd implementations in cudnn.</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/30934">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/30934" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/30934" target="_blank" rel="noopener nofollow ugc">Spurious negative output in convolution of positive tensors</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-12-07" data-time="16:11:34" data-timezone="UTC">04:11PM - 07 Dec 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/ivannz" target="_blank" rel="noopener nofollow ugc">
          <img alt="ivannz" src="https://avatars.githubusercontent.com/u/2227377?v=4" class="onebox-avatar-inline" width="20" height="20">
          ivannz
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: docs
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: convolution
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">The gist of the issue is that at one point in my network in my project I feed po<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">sitive float32 inputs `X` and `W` into a 1d (or 2d) convolution, but unexpectedly obtain explicitly negative outputs.

## Details
The X input is nonnegative about 1e-2 with many explicit 0 (after an elementwise square, or abs), and the W input is positive but with quite large range of [1e–8, 1e+8] (after an exp). I take `F.conv2d(X, W, None)` and **expect nonnegative** outputs (finite or infinite), but actually get **significantly negative** values. The downstream computations rely on the result of this operation to be at least nonnegative.

I tried `torch.backends.cudnn.deterministic = True`, or fp64, or clamping the `W` value and each time the problem seemed to had gone away. In my project I ended up clamping the result of the convolution to nonnegative range, and it sufficed.

Having observed this oddity got me worried, since in my understanding of floating point arithmetic the loss of precision and round off could yield imprecise results, but not **flip the sign** of the overall result. I read in the docs, that non-deterministic cudnn algorithms might induce this behaviour and yield inconsistent results, but I did not expect that atomic additions (according to cudnn reference) could spuriously flip the sign bit of a 32bit float.

I understand that Github issues are not a Q&amp;A support forum, but I think that this inconsistency, might of interest, and would very much appreciate any discussion of guarantees of single precision operations.

## Minimally reproducing example
I came up with a minimal example of this odd behaviour. My observations are as follows (based on this snippet, and on the real data within the network in my project):
* negative outputs of various magnitude occur **only if** deterministic is *False*, and data is *fp32*, and the input `X` is a moderately large-dim tensor.
* `X` and `W` of dims `...x12x12` and `...x4x4`, respectively, give a lot of very small fp32 numbers, some of which are negative
* `X` and `W` of dims `...x11x11` and `...x3x3`, respectively, produce very rare negative tiny fp32, but mostly expected zeros

These suggest that some optimizations within cudnn might be responsible.

The following code constructs explicitly nonnegative `X` and `W`: `X` all zeros except for a single **1** at the *centre*, `W` is a 4x4 kernel with constant **1e-3**. I attach the pickled  [result.gz](https://github.com/pytorch/pytorch/files/3935238/result.gz)

```python
import gzip
import torch
import torch.nn.functional as F

# change these
torch.backends.cudnn.deterministic = False
devtype = dict(device=torch.device("cuda:0"), dtype=torch.float32)

# x -- zeros with a spike
x = torch.zeros(1024, 1, 12, 12).to(**devtype)  
x[..., 5, 5] = 1.

# small nonnegative values
w = (torch.ones(1, 1, 4, 4) * 1e-3).to(**devtype)
assert x.ge(0).all() and w.ge(0).all()

# conv1d/2d
result = F.conv2d(x, w, None)
with gzip.open("result.gz", "wb") as fout:
    torch.save(dict(x=x, w=w, result=result), fout)

assert result.ge(0).all()
```

To unpack the attached `.gz` pickled result.
```python
import gzip
import torch

with gzip.open("result.gz", "rb") as fin:
    pack = torch.load(fin)

pack["result]
```

Here is a link to a [colab notebook](https://colab.research.google.com/drive/136n8oKDitchxEkXMhV7AJeYfFUayiS4X), where it can be seen, that the output is spurious even for `X` is `25 x 1 x 12 x 12`. Especially striking is that `24 x 1 x 12 x 12` appears to be correct.

## Specs and versions
Below are very abridged specs:

`Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz, 256GB RAM`, `4x GeForce GTX 1080 Ti` with `Driver Version: 390.77` on `Linux Mint 18.3 Sylvia (GNU/Linux 4.10.0-38-generic x86_64)` and conda `python 3.7.4` packages:
* `pytorch                   1.1.0           py3.7_cuda9.0.176_cudnn7.5.1_0    pytorch`
* `cudatoolkit               9.0                  h13b8566_0`


cc @jlin27 @mruberry</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you!</p>