RuntimeError: CUDA error: device-side assert triggered when i tried TorchVision Object Detection Finetuning Tutorial

ronghui: <p>When i tried TorchVision Object Detection Finetuning Tutorial using my own image. I got following error. I am sure my label does not have negative one.</p>
<p>/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [1,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [2,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [3,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [4,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [5,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [6,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [7,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [8,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [9,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [10,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [11,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [12,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [13,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [14,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [15,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [16,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [17,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [18,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [19,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [20,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [21,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [22,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [23,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [24,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [25,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [26,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [27,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [28,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [29,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [30,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [31,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [32,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [33,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [34,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [35,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [36,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [37,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [38,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [39,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [40,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [41,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [42,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [43,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [44,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [45,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [46,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [47,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [48,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [49,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [50,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [51,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [52,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [53,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [54,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [55,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [56,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [57,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [58,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [59,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [60,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [61,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [62,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [63,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [64,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [65,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [66,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [67,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [68,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [69,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [70,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [71,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [72,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [73,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [74,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [75,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [76,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [77,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [78,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [79,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [80,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [81,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [82,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [83,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [84,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [85,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [86,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [87,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [88,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [89,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [90,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [91,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [92,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [93,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [94,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [95,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [96,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [97,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [98,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [99,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [100,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [101,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [102,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [103,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [104,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [105,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [106,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [107,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [108,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [109,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [110,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [111,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [112,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [113,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [114,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [115,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [116,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [117,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [118,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [119,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [120,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [121,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [122,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [123,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [124,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [125,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [126,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [127,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
Traceback (most recent call last):<br>
File “main1.py”, line 73, in <br>
main()<br>
File “main1.py”, line 64, in main<br>
train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)<br>
File “/home/zhanronghui/project/docAnalysis/engine1.py”, line 39, in train_one_epoch<br>
loss_dict = model(images, targets)<br>
File “/home/zhanronghui/torches/torch18/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/zhanronghui/torches/torch18/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py”, line 97, in forward<br>
proposals, proposal_losses = self.rpn(images, features, targets)<br>
File “/home/zhanronghui/torches/torch18/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/zhanronghui/torches/torch18/lib/python3.8/site-packages/torchvision/models/detection/rpn.py”, line 364, in forward<br>
loss_objectness, loss_rpn_box_reg = self.compute_loss(<br>
File “/home/zhanronghui/torches/torch18/lib/python3.8/site-packages/torchvision/models/detection/rpn.py”, line 296, in compute_loss<br>
sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)<br>
File “/home/zhanronghui/torches/torch18/lib/python3.8/site-packages/torchvision/models/detection/_utils.py”, line 61, in <strong>call</strong><br>
neg_idx_per_image = negative[perm2]<br>
RuntimeError: CUDA error: device-side assert triggered</p>

ptrblck: <p>Could you print the min and max values of <code>perm2</code> as well as the shape of <code>negative</code> as it seems that this operation is failing?</p>

ronghui: <p>print(max(perm1)) is tensor(7, device=‘cuda:3’)<br>
print(min(perm1)) is  tensor(0, device=‘cuda:3’)<br>
print(max(perm2)) is tensor(4771563806306336768, device=‘cuda:3’)<br>
print(min(perm2)) is tensor(4733283209477357568, device=‘cuda:3’)</p>

ptrblck: <p><code>perm2</code> seems to contain quite unusual values and I assume that <code>negative</code>'s size is smaller than these values and will thus raise the error. You would thus have to check why <code>perm2</code> is containing these values and if they are expected.</p>

ronghui: <p>but it runs fine on cpu. I really have no idea what’s going on.</p>

ronghui: <p>maxperm1: tensor(12)<br>
minperm1: tensor(0)<br>
maxperm2: tensor(1982)<br>
minperm2: tensor(10)</p>
<p>when i ran on cpu. i got those value. Everything is fine.</p>

tyhawa: <p>Hi, got a similar CUDA error to ronghui when trying to run the object detection finetuning tutorial. I think also got same error when running the example code on mscoco. See below:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=1    --use_env train.py    --data-path=/home/matt/phd/data/mscoco    --dataset coco --model fasterrcnn_resnet50_fpn --epochs 26    --lr-steps 16 22 --aspect-ratio-group-factor 3
| distributed init (rank 0): env://
Namespace(aspect_ratio_group_factor=3, batch_size=2, data_augmentation='hflip', data_path='/home/matt/phd/data/mscoco', dataset='coco', device='cuda', dist_backend='nccl', dist_url='env://', distributed=True, epochs=26, gpu=0, lr=0.02, lr_gamma=0.1, lr_scheduler='multisteplr', lr_step_size=8, lr_steps=[16, 22], model='fasterrcnn_resnet50_fpn', momentum=0.9, output_dir='.', pretrained=False, print_freq=20, rank=0, resume='', rpn_score_thresh=None, start_epoch=0, sync_bn=False, test_only=False, trainable_backbone_layers=None, weight_decay=0.0001, workers=4, world_size=1)
Loading data
loading annotations into memory...
Done (t=11.17s)
creating index...
index created!
loading annotations into memory...
Done (t=1.98s)
creating index...
index created!
Creating data loaders
Using [0, 0.5, 0.6299605249474366, 0.7937005259840997, 1.0, 1.2599210498948732, 1.5874010519681994, 2.0, inf] as bins for aspect ratio quantization
Count of instances per bin: [  104   982 24236  2332  8225 74466  5763  1158]
Creating model
Start training
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [1,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [2,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [3,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [4,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [5,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [6,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [7,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [8,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [9,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [10,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [11,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [12,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [13,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [14,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [15,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [16,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [17,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [18,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [19,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [20,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [21,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [22,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [23,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [24,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [25,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [26,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [27,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [28,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [29,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [30,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [31,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [32,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [33,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [34,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [35,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [36,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [37,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [38,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [39,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [40,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [41,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [42,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [43,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [44,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [45,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [46,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [47,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [48,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [49,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [50,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [51,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [52,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [53,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [54,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [55,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [56,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [57,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [58,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [59,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [60,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [61,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [62,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [63,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [64,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [65,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [66,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [67,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [68,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [69,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [70,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [71,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [72,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [73,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [74,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [75,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [76,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [77,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [78,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [79,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [80,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [81,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [82,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [83,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [84,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [85,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [86,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [87,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [88,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [89,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [90,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [91,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [92,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [93,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [94,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [95,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [96,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [97,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [98,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [99,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [100,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [101,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [102,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [103,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [104,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [105,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [106,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [107,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [108,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [109,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [110,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [111,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [112,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [113,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [114,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [115,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [116,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [117,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [118,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [119,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [120,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [121,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [122,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [123,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [124,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [125,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [126,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554793803/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [127,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
Traceback (most recent call last):
  File "train.py", line 234, in &lt;module&gt;
    main(args)
  File "train.py", line 207, in main
    train_one_epoch(model, optimizer, data_loader, device, epoch, args.print_freq)
  File "/home/matt/phd/robust_cnn/src/ext/engine.py", line 30, in train_one_epoch
    loss_dict = model(images, targets)
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py", line 97, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 364, in forward
    loss_objectness, loss_rpn_box_reg = self.compute_loss(
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 296, in compute_loss
    sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
  File "/home/matt/anaconda3/envs/torch_env/lib/python3.8/site-packages/torchvision/models/detection/_utils.py", line 45, in __call__
    positive = torch.where(matched_idxs_per_image &gt;= 1)[0]
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>Using retinanet however did not seem to have the same problem with mscoco.</p>

ptrblck: <p>Could both of you check, if you are seeing the same issue using the latest nightly release?</p>

tyhawa: <p>Thank you Piotr, I seem to be able to use fasterrcnn now with <code>pytorch 1.9.0.dev20210513</code>.  Was this a recent bug? I tried to look on github issues for something related, but I didn’t see anything.</p>
<p>Also, while you are online, let me say thanks heaps for you and your team’s support work! Your help has saved my butt so many times (although this is my first time posting in the forum).</p>

ptrblck: <p>I think you might have been hitting an already fixed issue in the way <code>cub/thrust</code> was used, which was fixed <a href="https://github.com/pytorch/pytorch/pull/55292">here</a>. The fix also contains a link to a potentially related error.</p>
<p>Thanks for the kind words! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

rinchenson: <p>Hi tyhawa, i ran into similar problem like you. Could you please let me know the full configuration of your env. I will try to get my env. config exactly same as you. Thank you so much <a class="mention" href="/u/tyhawa">@tyhawa</a> .</p>
<p>you can grab your env configuration by using this command: python3 -m torch.utils.collect_env</p>

rinchenson: <p>never mind guys. I managed to fix it. thank you</p>