"invalid shape dimension <huge negative number>" on tensor masking operation

Algomorph: <p>I’m (mostly) someone-else’s code in combination with <a href="https://github.com/NVlabs/PWC-Net" rel="noopener nofollow ugc">PWC-Net</a> with (their) existing inputs. I’m getting an error in one of the cases (not every case). Here is all the code that I think is relevant:</p>
<pre data-code-wrap="Python"><code class="lang-Python">    print("Shape of mask is:", mask.shape)
    s = (mask &gt; 0.999).shape
    print("Shape of (mask &gt; 0.999) is:", s)
    mask[mask &gt; 0.999] = 1.0  # &lt;-- Error occurs here!
</code></pre>
<p>Here, <code>mask</code> is a PyTorch tensor. The output is:</p>
<blockquote>
<p>Shape of mask is: torch.Size([1, 1, 28, 40])<br>
Shape of (mask &gt; 0.999) is: torch.Size([1, 1, 28, 40])<br>
Traceback (most recent call last):<br>
<br>
File “/home/algomorph/Workbench/NeuralTracking/model/pwcnet.py”, line 41, in Backward<br>
mask[mask &gt; 0.999] = 1.0  # ← Error occurs here!<br>
RuntimeError: invalid shape dimension -1096498240</p>
</blockquote>
<p>The only clue I found online is the <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/InferSize.h" rel="noopener nofollow ugc">C++ header file</a> that raises this error in PyTorch.</p>
<p>Any clues as to what might be happening here?</p>
<p>Aside: I’m using PyTorch 1.7.1 compiled from source, i.e. version ‘1.7.0a0+57bffc3’ [also tested with the official 1.8.1 CUDA 11.1 release for pip]<br>
Another aside: <code>mask</code> is of type torch.cuda.FloatTensor and is located on the <code>cuda:0</code> device.</p>
<p>Here is the <a href="https://srv-store5.gofile.io/download/knPz5d/mask.pt" rel="noopener nofollow ugc"><code>mask</code> tensor</a> serialized via torch.save(…) . When I load it in a separate python shell and try to perform the operation on it, the error does not occur.</p>

Algomorph: <p>Big thanks to Ahmed Taha, who (in another thread / forum) suggested to replace the<br>
<code>mask[mask &gt; 0.999] = 1.0</code><br>
with<br>
<code>mask = torch.where(mask &gt; 0.999, torch.ones_like(mask), mask)</code></p>
<p>This solution didn’t produce any errors so far, so I think it is valid. Why the original code produced an error for some cases remains a mystery to me.</p>

Algomorph: <p>My intuitions is: this is a bug in PyTorch. Consider this code:</p>
<pre><code class="lang-auto">        object_estimate = self.moduleSix(pyramid_first[-1], pyramid_second[-1], None)
        flow6 = object_estimate['flow']

        mask2 = torch.load("mask.pt")
        mask2[mask2 &gt; 0.999] = 1.0 # &lt;-- No error here...

        object_estimate = self.moduleFiv(pyramid_first[-2], pyramid_second[-2], object_estimate)
        flow5 = object_estimate['flow']

        mask2 = torch.load("mask.pt")
        mask2[mask2 &gt; 0.999] = 1.0 # &lt;-- Error here...
</code></pre>
<p><code>mask2</code>, in both cases, is just a copy of the mask tensor I provide &amp; talk about above loaded from disk. <code>moduleFiv</code> and <code>moduleSix</code> are Decoder objects based on <code>torch.nn.Module</code>, which vary only by <code>in_channels</code> counts to their convolutional layers.</p>

ShirAmir: <p><a class="mention" href="/u/algomorph">@Algomorph</a> I too encountered this issue when loading a tensor from the disk and trying to filter it with a mask.<br>
Your solution helped me as well, but it is indeed confusing why…</p>
<p>If anyone has any insights pls let us know.</p>

anonymous_anonymous: <p>I met this error as well… I used the padding mask tensor to filter out non-padding result: tensor[padding_mask], after I changed the padding_mask device to cuda as well, the error disappeared, but the final result seems wrong.</p>

anonymous_anonymous: <p>In another code, I just do the same operation, but no error occurred.</p>

rdesh26: <p>I ran into the same issue with indexing recently, and had to fix it using <code>torch.where()</code>. Is there a relevant issue created on the repo to track this?</p>