Loss.backward() failure due to contiguous issue

xuesongwang: <p>Hi,<br>
I found this error when I backprop the loss<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c7c195910ea7bd9357417b17e241010b114ef73.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c7c195910ea7bd9357417b17e241010b114ef73" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c7c195910ea7bd9357417b17e241010b114ef73.png" alt="image" data-base62-sha1="ftHrL2Ovvyima4sD9WWkH4Vj2Kf" width="690" height="214" data-dominant-color="313438"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1369×426 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I suppose it indicates that ones needs to be contiguous, I have modified my codes into these:</p>
<p>self.sigma = nn.Parameter(np.log(init_length_scale) *torch.ones(self.in_channels), requires_grad=True).contiguous()</p>
<p>density = torch.ones(batch_size, n_in, 1).to(self.device).contiguous()</p>
<p>Still they didn’t work. Could somebody please tell me if I have put contiguous() at the wrong places? or I misunderstand the debug info. How should I locate such problems btw?</p>
<p>Many thanks in advance</p>

albanD: <p>Hi,</p>
<p>Can you enable <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly">anomaly detection</a> to see which forward op is responsible for this please?</p>

xuesongwang: <p>Oh, that’s really useful!!! The bugs started to make some sense. It might be convolution channel mismatching for skipped layers. Thank you sooo much <a class="mention" href="/u/alband">@albanD</a>, otherwise I think I’ll just stuck around checking unrelated codes and making no progress at all.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0ecc6070aa6ce504f860ff73df4ae188f03714a.png" data-download-href="https://discuss.pytorch.org/uploads/default/f0ecc6070aa6ce504f860ff73df4ae188f03714a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0ecc6070aa6ce504f860ff73df4ae188f03714a.png" alt="image" data-base62-sha1="ynjSqBlI262OSyFz8UgT0ZtyhGW" width="690" height="411" data-dominant-color="303437"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1300×775 59.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de54fe8f9080e48fcb25951ce8840eef95408cd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/de54fe8f9080e48fcb25951ce8840eef95408cd3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de54fe8f9080e48fcb25951ce8840eef95408cd3.png" alt="image" data-base62-sha1="vIQ64VfhZpL6mQ3tRExlpbC2BmX" width="690" height="176" data-dominant-color="2F3134"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1328×339 18.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Can you print the full info about the convolution inputs/weight/bias? This is most likely a bug on our side I’m afraid.</p>
<pre><code class="lang-python">print("Info for t.")
print("size: ", t.size())
print("stride: ", t.stride())
print("offset: ", t.storage_offset())
print("device: ", t.device)
print("layout: ", t.layout)
</code></pre>
<p>Thanks!</p>

xuesongwang: <p>Hi,<br>
this is the Unet structure I adopted, the forward prop works fine though,</p>
<pre><code class="lang-python">def pad_concat(t1, t2):
    """Concat the activations of two layer channel-wise by padding the layer
    with fewer points with zeros.

    Args:
        t1 (tensor): Activations from first layers of shape `(batch, c1, n1, m1)`.
        t2 (tensor): Activations from second layers of shape `(batch,c2, n2, m2)`.

    Returns:
        tensor: Concatenated activations of both layers of shape
            `(batch, c1 + c2, max(n1, n2), max(m1, m2))`.
    """
    if t1.shape[2] &gt; t2.shape[2]:
        padding = t1.shape[2] - t2.shape[2]
        if padding % 2 == 0:  # Even difference
            t2 = F.pad(t2, (0, 0, int(padding / 2), int(padding / 2)), 'reflect').contiguous()
        else:  # Odd difference
            t2 = F.pad(t2, (0, 0, int((padding - 1) / 2), int((padding + 1) / 2)),
                       'reflect').contiguous()
    elif t2.shape[2] &gt; t1.shape[2]:
        padding = t2.shape[2] - t1.shape[2]
        if padding % 2 == 0:  # Even difference
            t1 = F.pad(t1, (0, 0, int(padding / 2), int(padding / 2)), 'reflect').contiguous()
        else:  # Odd difference
            t1 = F.pad(t1, (0, 0, int((padding - 1) / 2), int((padding + 1) / 2)),
                       'reflect').contiguous()

    # another dimension
    if t1.shape[3] &gt; t2.shape[3]:
        padding = t1.shape[3] - t2.shape[3]
        if padding % 2 == 0:  # Even difference
            t2 = F.pad(t2, (int(padding / 2), int(padding / 2), 0, 0), 'reflect').contiguous()
        else:  # Odd difference
            t2 = F.pad(t2, (int((padding - 1) / 2), int((padding + 1) / 2), 0, 0),
                       'reflect').contiguous()
    elif t2.shape[3] &gt; t1.shape[3]:
        padding = t2.shape[3] - t1.shape[3]
        if padding % 2 == 0:  # Even difference
            t1 = F.pad(t1, (int(padding / 2), int(padding / 2), 0, 0), 'reflect').contiguous()
        else:  # Odd difference
            t1 = F.pad(t1, (int((padding - 1) / 2), int((padding + 1) / 2), 0, 0),
                       'reflect').contiguous()

    return torch.cat([t1, t2], dim=1)


class UNet(nn.Module):
    """Large convolutional architecture from 1d experiments in the paper.
    This is a 12-layer residual network with skip connections implemented by
    concatenation.

    Args:
        in_channels (int, optional): Number of channels on the input to
            network. Defaults to 8.
    """

    def __init__(self, in_channels=8):
        super(UNet, self).__init__()
        self.activation = nn.ReLU()
        self.in_channels = in_channels
        self.out_channels = 16
        self.num_halving_layers = 6

        self.l1 = nn.Conv2d(in_channels=self.in_channels,
                            out_channels=self.in_channels,
                            kernel_size=5, stride=2, padding=2)
        self.l2 = nn.Conv2d(in_channels=self.in_channels,
                            out_channels=2 * self.in_channels,
                            kernel_size=5, stride=2, padding=2)
        self.l3 = nn.Conv2d(in_channels=2 * self.in_channels,
                            out_channels=2 * self.in_channels,
                            kernel_size=5, stride=2, padding=2)
        self.l4 = nn.Conv2d(in_channels=2 * self.in_channels,
                            out_channels=4 * self.in_channels,
                            kernel_size=5, stride=2, padding=2)
        self.l5 = nn.Conv2d(in_channels=4 * self.in_channels,
                            out_channels=8 * self.in_channels,
                            kernel_size=5, stride=2, padding=2)

        for layer in [self.l1, self.l2, self.l3, self.l4, self.l5]:
            init_layer_weights(layer)

        self.l6 = nn.ConvTranspose2d(in_channels=8 * self.in_channels,
                                     out_channels=4 * self.in_channels,
                                     kernel_size=5, stride=2, padding=2,
                                     output_padding=1)
        self.l7 = nn.ConvTranspose2d(in_channels=8 * self.in_channels,
                                     out_channels=2 * self.in_channels,
                                     kernel_size=5, stride=2, padding=2,
                                     output_padding=1)
        self.l8 = nn.ConvTranspose2d(in_channels=4 * self.in_channels,
                                      out_channels=2 * self.in_channels,
                                      kernel_size=5, stride=2, padding=2,
                                      output_padding=1)
        self.l9 = nn.ConvTranspose2d(in_channels=4 * self.in_channels,
                                      out_channels=self.in_channels,
                                      kernel_size=5, stride=2, padding=2,
                                      output_padding=1)
        self.l10 = nn.ConvTranspose2d(in_channels=2 * self.in_channels,
                                      out_channels=self.in_channels,
                                      kernel_size=5, stride=2, padding=2,
                                      output_padding=1)

        for layer in [self.l6, self.l7, self.l8, self.l9, self.l10]:
            init_layer_weights(layer)

    def forward(self, x):
        """Forward pass through the convolutional structure.

        Args:
            x (tensor): Inputs of shape `(batch, n_in, in_channels)`.

        Returns:
            tensor: Outputs of shape `(batch, n_out, out_channels)`.
        """
        h1 = self.activation(self.l1(x))
        self.print_conv_info(x, self.l1)

        h2 = self.activation(self.l2(h1))
        self.print_conv_info(h1, self.l2)

        h3 = self.activation(self.l3(h2))
        self.print_conv_info(h2, self.l3)

        h4 = self.activation(self.l4(h3))
        self.print_conv_info(h3, self.l4)

        h5 = self.activation(self.l5(h4))
        self.print_conv_info(h4, self.l5)

        h6 = self.activation(self.l6(h5))
        self.print_conv_info(h5, self.l6)
        h6 = pad_concat(h4, h6)

        h7 = self.activation(self.l7(h6))
        self.print_conv_info(h6, self.l7)
        h7 = pad_concat(h3, h7)

        h8 = self.activation(self.l8(h7))
        self.print_conv_info(h7, self.l8)
        h8 = pad_concat(h2, h8)

        h9 = self.activation(self.l9(h8))
        self.print_conv_info(h8, self.l9)
        h9 = pad_concat(h1, h9)

        h10 = self.activation(self.l10(h9))
        self.print_conv_info(h9, self.l10)
        output = pad_concat(x, h10)

        return output

    def print_conv_info(self, input, layer):
        print("Info for %s."%layer)
        print("stride:", layer.stride)
        print("input size: ", input.size())
        print("input offset: ", input.storage_offset())
        print("input device: ", input.device)
        print("input layout:", input.layout)
</code></pre>
<p>and the results are the followings. hope it’s not too complicated</p>
<p>Info for Conv2d(8, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 8, 192, 63])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for Conv2d(8, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 8, 96, 32])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for Conv2d(16, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 16, 48, 16])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for Conv2d(16, 32, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 16, 24, 8])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for Conv2d(32, 64, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 32, 12, 4])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for ConvTranspose2d(64, 32, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 64, 6, 2])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for ConvTranspose2d(64, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 64, 12, 4])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for ConvTranspose2d(32, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 32, 24, 8])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for ConvTranspose2d(32, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 32, 48, 16])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>
<p>Info for ConvTranspose2d(16, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (2, 2)<br>
input size:  torch.Size([64, 16, 96, 32])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>

xuesongwang: <p>BTW, I suspect it is related to the padding operation, I am working on padding input x first in order to make its width and height even numbers and see if I will succeed.</p>
<hr>
<p>NO, it’s not <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:">  I have made sure that only concatenation is called without padding in the pad_concat function, and directly output h10 without concatenating with x to avoid size mismatching.</p>
<p>Still the bug remains the same. I am wondering whether it is caused by modules before and after, since the final bug says: “RuntimeError: ones needs to be contiguous”. and I did have learnable parameters like this</p>
<pre><code class="lang-python">self.sigma = nn.Parameter(np.log(init_length_scale) * torch.ones(self.in_channels).contiguous(), requires_grad=True)
</code></pre>
<p>Honestly, I am not sure what this “ones” means</p>

albanD: <p>The stride I was looking for there is the Tensor stride, not the layer’s stride <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> This is what makes a Tensor “non-contiguous” that’s why it is important.</p>

xuesongwang: <p>Oh I see,  here’s the result, thanks again</p>
<p>Info for Conv2d(8, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (96768, 12096, 63, 1)<br>
input size:  torch.Size([64, 8, 192, 63])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for Conv2d(8, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (24576, 3072, 32, 1)<br>
input size:  torch.Size([64, 8, 96, 32])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for Conv2d(16, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (12288, 768, 16, 1)<br>
input size:  torch.Size([64, 16, 48, 16])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for Conv2d(16, 32, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (3072, 192, 8, 1)<br>
input size:  torch.Size([64, 16, 24, 8])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for Conv2d(32, 64, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2)).<br>
stride: (1536, 48, 4, 1)<br>
input size:  torch.Size([64, 32, 12, 4])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for ConvTranspose2d(64, 32, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (768, 12, 2, 1)<br>
input size:  torch.Size([64, 64, 6, 2])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for ConvTranspose2d(64, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (3072, 48, 4, 1)<br>
input size:  torch.Size([64, 64, 12, 4])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for ConvTranspose2d(32, 16, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (6144, 192, 8, 1)<br>
input size:  torch.Size([64, 32, 24, 8])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for ConvTranspose2d(32, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (24576, 768, 16, 1)<br>
input size:  torch.Size([64, 32, 48, 16])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided<br>
Info for ConvTranspose2d(16, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), output_padding=(1, 1)).<br>
stride: (49152, 3072, 32, 1)<br>
input size:  torch.Size([64, 16, 96, 32])<br>
input offset:  0<br>
input device:  cuda:7<br>
input layout: torch.strided</p>

alexeib: <p>i am having hte same error when trying to use deconvolutions with the following arch:</p>
<p>ModuleList(<br>
(0): Sequential(<br>
(0): ConvTranspose1d(512, 512, kernel_size=(2,), stride=(2,), output_padding=(1,))<br>
)<br>
(1): Sequential(<br>
(0): ConvTranspose1d(512, 512, kernel_size=(2,), stride=(2,), output_padding=(1,))<br>
)<br>
(2): Sequential(<br>
(0): ConvTranspose1d(512, 512, kernel_size=(3,), stride=(2,), output_padding=(1,))<br>
)<br>
(3): Sequential(<br>
(0): ConvTranspose1d(512, 512, kernel_size=(3,), stride=(2,), output_padding=(1,))<br>
)<br>
…</p>
<p>must be the output_padding ?</p>

alexeib: <p>its definitely the output padding since if i set it to 0 everything works</p>

xuesongwang: <p>yes!!! <a class="mention" href="/u/alexeib">@alexeib</a> it worked for me too. I set out_padding = 0 and use F.pad to align the size.  Thanks a lot for the suggestion. But still I will be waiting for some possible explanations before I settle with the solution.</p>

alexeib: <p>i created an issue here:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/44949" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/44949" target="_blank" rel="nofollow noopener">conv_transpose1d with output padding backpropagation does not work</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-09-18" data-time="13:56:57" data-timezone="UTC">01:56PM - 18 Sep 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/alexeib" target="_blank" rel="nofollow noopener">
          <img alt="alexeib" src="https://avatars3.githubusercontent.com/u/3497946?v=4" class="onebox-avatar-inline" width="20" height="20">
          alexeib
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">see the discussion here:
https://discuss.pytorch.org/t/loss-backward-failure-due-to-contiguous-issue/96416/9
in a nutshell, having several stacked conv_transpose1d layers with stride &gt; 1 and output padding makes it not...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>maybe you can provide a repro there for pytorch team to look at?</p>

albanD: <p>Can you run your code with <code>TORCH_SHOW_CPP_STACKTRACES=1</code> set to see which exact conv function is responsible? Looking at the code or convtranspose doesn’t really show anything suspicious.</p>
<p><a class="mention" href="/u/alexeib">@alexeib</a> we usually wait to find out what is the issue and make sure it is a bug in pytorch before opening an issue. To avoid opening “empty” issues that don’t contain concrete information.</p>

xuesongwang: <p>Hi there, here’s the result with <code>TORCH_SHOW_CPP_STACKTRACES=1 </code> before python main.py, and I think it started from l10,<br>
( h10 = self.activation(self.l10(h9)))<br>
which the first layer of convolution if you do loss back propagation,</p>
<pre><code class="lang-auto">Warning: Traceback of forward call that caused the error:
  File "NP_PROV_train.py", line 55, in &lt;module&gt;
    mean, var = convcnp(x_context.to(device), y_context.to(device), x_target.to(device))
  File "/home/xuesongwang/venv_torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xuesongwang/PycharmProject/Fluctuation_Resistant_Neural_Process/module/NP_PROV.py", line 508, in forward
    h = self.rho(h)
  File "/home/xuesongwang/venv_torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xuesongwang/PycharmProject/Fluctuation_Resistant_Neural_Process/module/NP_PROV.py", line 246, in forward
    h10 = self.activation(self.l10(h9))
  File "/home/xuesongwang/venv_torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xuesongwang/venv_torch/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 778, in forward
    output_padding, self.groups, self.dilation)
 (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57)
  0%|                                                                                                            | 0/200000 [00:00&lt;?, ?it/s]
Traceback (most recent call last):
  File "NP_PROV_train.py", line 58, in &lt;module&gt;
    loss.backward()
  File "/home/xuesongwang/venv_torch/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/xuesongwang/venv_torch/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: ones needs to be contiguous
</code></pre>

urukhai: <p>I don’t know if this is the same problem, but I just got an autograd error trying to run a small Fastai Tabular model.  I’d appreciate advice about how to track down the problem.</p>
<pre><code class="lang-bash">~/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    123         retain_graph = create_graph
    124 
--&gt; 125     Variable._execution_engine.run_backward(
    126         tensors, grad_tensors, retain_graph, create_graph,
    127         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Found dtype Short but expected Float
Exception raised from compute_types at /Users/distiller/project/conda/conda-bld/pytorch_1595629449223/work/aten/src/ATen/native/TensorIterator.cpp:183 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string&lt;char, std::__1::char_traits&lt;char&gt;, std::__1::allocator&lt;char&gt; &gt;) + 169 (0x128c4e199 in libc10.dylib)
frame #1: at::TensorIterator::compute_types(at::TensorIteratorConfig const&amp;) + 3842 (0x121193312 in libtorch_cpu.dylib)
frame #2: at::TensorIterator::build(at::TensorIteratorConfig&amp;) + 618 (0x12119c51a in libtorch_cpu.dylib)
frame #3: at::TensorIterator::TensorIterator(at::TensorIteratorConfig&amp;) + 223 (0x12119c1ff in libtorch_cpu.dylib)
frame #4: at::native::mse_loss_backward_out(at::Tensor&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, long long) + 410 (0x120fe7f7a in libtorch_cpu.dylib)
</code></pre>

xuesongwang: <p>hi <span class="mention">@urukhaim</span>, did you try the <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" rel="nofollow noopener">detect_anomaly</a> suggested by <a class="mention" href="/u/alband">@albanD</a> ? It may help you locate which operation is responsible</p>

alexeib: <p><a class="mention" href="/u/xuesongwang">@xuesongwang</a> is it possible to create a toy example and provide it in the issue above?</p>

xuesongwang: <p>Hi <a class="mention" href="/u/alexeib">@alexeib</a> , I tried to write a demo code using only Unet architecture,  and I assigned random input and output. The out_padding was set to 1.  However, it worked and I got the gradient. Hence, I think it is related to the modules before and after Unet, if that’s the case, then I would probably have to release all my codes <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:">  , sorry about that.</p>

albanD: <p>If that comes from here, if you just run the previous layer and this one, l10, does that trigger the same error? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

xuesongwang: <p>Sorry I am a bit confused here. Let’s say l10 belongs to this UNet structure, before it was a linear module plus one learnable parameter, by “previous layer”, you mean I input my data,  go through all the layers before l10, and directly used the output of l10 to compute loss and back propagation?</p>