Tensor.any() got ] Assertion `input_val >= zero && input_val <= one` failed

username: jinfagang
cooked: <pre><code class="lang-auto"> mask = mask.bool()
            # if len(torch.nonzero(mask)) &lt; 1:
            print(mask.shape)
            if not mask.any():
</code></pre>
<p>For BCE such error I can understand, but here the tracback stops at mask.any(), and keep gives me this:</p>
<pre><code class="lang-auto">/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [8,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [9,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [10,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [11,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [12,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [13,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [14,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [15,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [16,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [17,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [18,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [19,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [20,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [21,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [22,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [23,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [24,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [25,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [26,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [27,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [28,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [29,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [30,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [63,0,0], thread: [31,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [32,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [33,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [34,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [35,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [36,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [37,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [38,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [39,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [40,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [41,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [42,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [43,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [44,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [45,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [46,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [47,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [48,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [49,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [50,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [51,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [52,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [53,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [54,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [55,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [56,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [57,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [58,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [59,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [60,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [61,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [62,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [59,0,0], thread: [63,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [0,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [1,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [2,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [3,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [4,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [5,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [6,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [7,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [8,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [9,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [10,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [11,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [12,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [13,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [14,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [15,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [16,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [17,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [18,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [19,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [20,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [21,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [22,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [23,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [24,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [25,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [26,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [27,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [28,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [29,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [30,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [139,0,0], thread: [31,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [107,0,0], thread: [0,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [107,0,0], thread: [1,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [107,0,0], thread: [2,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [107,0,0], thread: [3,0,0] Assertion `input_val &gt;= zero &amp;&amp; input_val &lt;= one` failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102Traceback (most recent call last):

</code></pre>
<p>trace are:</p>
<pre><code class="lang-auto"> line 412, in forward
    if not mask.any():
RuntimeError: CUDA error: device-side assert triggered

</code></pre>
<p>How could this be? Even my mask is full of zeros or False, it shouldn’t gives me this error? the mask a Tensor of shape like [2, 3, 24, 56]</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so errors (in this case the <code>assert</code> triggered by the loss function) can be reported at a later line of code.<br>
To isolate the offending operation you could rerun the script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> or run it on the CPU.<br>
The latest PyTorch release also gives you this information in the error message.</p>