About a fifth of the gradients are lost if using .cuda() not inplace

username: ElToto
cooked: <p>I think I found a pytorch bug but I’m not entirely certain.<br>
Before I’m writing an issue I’ll post it here.</p>
<p>I have uploaded the sample code to google colab and the output with the error is visible in the output field. colab link here <a href="https://colab.research.google.com/drive/1UC7mL2FS4Gz5HIQuSeQaSnWFynQmIHzn#scrollTo=V4MjwHAljuTN&amp;uniqifier=2" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a><br>
<strong>Dont forget to activate the gpu in notebook settings.</strong></p>
<p>I am generating 4 RGB random images and put them to my gpu.</p>
<pre><code>images = torch.rand(1,3,4,widht,height)
images = images.cuda() # this causes the error
images = images.requires_grad_(True)
</code></pre>
<p>after that I feed the images into a Maxpool3d and use a dummy loss so I can calculate<br>
the gradients of my input images.</p>
<pre><code>x = nn.MaxPool3d(kernel_size=(1, 2, 2), stride=(1, 2, 2), padding=(0, 0, 0))(images)
criterion = nn.MSELoss()
label = torch.ones_like(x)
loss = criterion(x, label)
loss.backward()
</code></pre>
<p>If I now analyze the gradients of the images</p>
<pre><code> grads_images = images.grad
</code></pre>
<p>I noticed that about a third of the gradients are very close to zero although there is no<br>
reason for that.</p>
<p>To make this problem clear I am visualizing the gradients.<br>
I take only one image, use the absolute and scale them between 0 and 1. Then I add<br>
the gradients of the red, green and blue channel together so i get a<br>
numpy array with size [widht, height]</p>
<p>The values in this array can now be visualized with a colormap of plt.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ece2124bb47f987f5ace48361de04fe8f65de412.png" alt="gradients_gone" data-base62-sha1="xNz2pzJj3CmsmN2ntI7ofqjZ9tw" width="360" height="251"></p>
<p>You can see the right fifth of the gradients are ~0</p>
<p>But If I change one line</p>
<pre><code> images = torch.rand(1,3,4,widht,height) 
 images.cuda() # this fixes the gradient error
</code></pre>
<p>The gradients are no longer ~0 in the right fifht of the image.<br>
(I’m a new user so I can’t paste more than one image so the new image is here <a href="https://imgur.com/a/ruuQQr4" rel="noopener nofollow ugc">https://imgur.com/a/ruuQQr4</a> and also in the colab)</p>
<p>If you don’t want to use colab, the full code is here:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
"""
I create 4 random images with batchsize 1 and channelsize 3 (rgb)
Then I use a 3d Maxpool
Then I want to visualize the gradients of our images with a target image
that consists of only ones.

The gradients dissapear if I overwrite my input images with images.cuda()
"""
height = 100 # width of our image
width = 150 # height of our image
images = torch.rand(1,3,4,height,width) # create 4 images with this height and width
images = images.cuda() # This creates the error
"""IF YOU REPLACE THE ABOVE LINE WITH images.cuda() IT WORKS AS INTENDED"""
images = images.requires_grad_(True)

pooled_imgs = nn.MaxPool3d(kernel_size=(1, 2, 2), stride=(1, 2, 2), padding=(0, 0, 0))(images)

# create a dummy target (same size as output with ones) and calculate the mse
criterion = nn.MSELoss()
label = torch.ones_like(pooled_imgs)
loss = criterion(pooled_imgs, label)
loss.backward()

# scale gradients between 0 and 1
pooled_absolute = torch.abs(images.grad)
max_grad_value = torch.max(pooled_absolute)
scaled_grads = pooled_absolute / max_grad_value # scale grads between 0 and 1
scaled_grads = scaled_grads.cpu().numpy()

# scaled grads has shape [Batchsize, Channels(RGB),amnt_images, width, height]
grad_pic = scaled_grads[0, :,0, :, :] # use batch 0 and the first image in our depth
# grad pic has no shape [3,height,width]

grad_pic = np.transpose(grad_pic, (1, 2, 0)) # shape now [height, width, 3]
# I want to add all gradients of the 3 channels (red, green blue)
# so I can use a plt colormap
grad_one_channel = np.ones([height, width])
grad_one_channel[:,:] = grad_pic[:,:,0]+grad_pic[:,:,1]+grad_pic[:,:,2]

plt.imshow(grad_one_channel, cmap='viridis')
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Thanks for the colab link!<br>
Running with a brand new env there, I get the proper result for each cell though. Can you try again on your side?</p>
<p>Also <code>image.cuda()</code> does nothing! You can remove this line and it will have the exact same behavior. (.cuda() cannot be done inplace).</p>

username: ElToto
cooked: <p>It seems that colab switched to torch 1.6 at the time between my question and your reply.<br>
At first glance I am also no longer able to reproduce the error with torch 1.6 in the colab.</p>
<p>I am still able to reproduce the bug with torch 1.4 with copy pasting the first cell to my local machine (deleting %matplotlib inline and adding as last line plt.show())<br>
I don’t have torch 1.5 installed on my local machine but the code worked yesterday in google colab which was using torch 1.5.</p>
<p>I can’t use</p>
<blockquote>
<p>!pip install torch==1.5.0 torchvision==0.6.0</p>
</blockquote>
<p>in colab because torch then complains about</p>
<blockquote>
<p>The NVIDIA driver on your system is too old (found version 10010). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver.</p>
</blockquote>
<p>Are you aware of any fixes in torch 1.6 that might have fixed this error?</p>
<p><strong>EDIT</strong> I think I found it myself:<br>
In the torch 1.6 bugfixes there is this line<br>
and I think this fits perfectly for my error</p>
<blockquote>
<p>nn.MaxPool3d: fixed incorrect CUDA backward results for non-square output (<span class="hashtag-raw">#36820</span>)</p>
</blockquote>
<aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/pytorch/pull/36820">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/pull/36820" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/pytorch/pull/36820" target="_blank" rel="noopener nofollow ugc">Fix MaxPool3d CUDA backward incorrect results for non-square output</a>
      </h4>

    <div class="branches">
      <code>pytorch:master</code> ← <code>linziyi96:master</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-17" data-time="19:02:01" data-timezone="UTC">07:02PM - 17 Apr 20 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/linziyi96" target="_blank" rel="noopener nofollow ugc">
            <img alt="linziyi96" src="https://avatars.githubusercontent.com/u/53928811?v=4" class="onebox-avatar-inline" width="20" height="20">
            linziyi96
          </a>
        </div>

        <div class="lines" title="2 commits changed 2 files with 11 additions and 1 deletions">
          <a href="https://github.com/pytorch/pytorch/pull/36820/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+11</span>
            <span class="removed">-1</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">In the CUDA version of max_pool3d backward, function  `max_pool3d_with_indices_b<span class="show-more-container"><a href="https://github.com/pytorch/pytorch/pull/36820" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">ackward_out_frame` is defined with args as `..., oheight, owidth, ...` but called with `..., owidth, oheight, ...`. As a result gradients are not fully calculated along the longer dimension due to insufficient grid size.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: albanD
cooked: <p>Hi,</p>
<p>Yes there were some maxpool fixes in 1.6. The one you linked looks like the one.<br>
Hopefully you can just use 1.6 without issues now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>