CUDA TorchScript error when using TensorRT

ravijoshi: <p>I am getting CUDA TorchScript error when using TensorRT inside PyTorch. Below are the errors:</p>
<pre><code class="lang-auto">  File "/home/ravi/yolact_edge/layers/output_utils.py", line 109, in postprocess
    masks = crop(masks, boxes)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/home/ravi/yolact_edge/layers/box_utils.py", line 316, in fallback_function
    """
    h, w, n = masks.size()
    x1, x2 = sanitize_coordinates(boxes[:, 0], boxes[:, 2], w, padding, cast=False)
             ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    y1, y2 = sanitize_coordinates(boxes[:, 1], boxes[:, 3], h, padding, cast=False)
  File "/home/ravi/yolact_edge/layers/box_utils.py", line 297, in sanitize_coordinates
        _x1 = _x1.long()
        _x2 = _x2.long()
    x1 = torch.min(_x1, _x2)
         ~~~~~~~~~ &lt;--- HERE
    x2 = torch.max(_x1, _x2)
    x1 = torch.clamp(x1-padding, min=0)
RuntimeError: CUDA error: device-side assert triggered

/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: indexSelectSmallIndex: block: [0,0,0], thread: [0,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: indexSelectSmallIndex: block: [0,0,0], thread: [1,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: indexSelectSmallIndex: block: [0,0,0], thread: [2,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: indexSelectSmallIndex: block: [0,0,0], thread: [3,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.

Traceback (most recent call last):
  File "/home/ravi/yolact_edge/layers/output_utils.py", line 100, in postprocess
    masks = proto_data @ masks.t()
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
</code></pre>
<p>The two errors from the above are <code>RuntimeError: CUDA error: device-side assert triggered</code> and <code>RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling 'cublasCreate(handle)'</code></p>
<p>Please note that these errors arrive only when all of the following conditions are met:</p>
<ol>
<li>When TensorRT is used.</li>
<li>After processing a few frames</li>
<li>When the parameter <code>score_threshold</code> is non-zero</li>
</ol>
<p>Below are my environment details:</p>
<pre><code class="lang-auto">* python --version 3.6.9
* tensorrt.__version__ 7.2.2.3
* torch.__version__ 1.7.1
* torchvision.__version__ 0.8.2
* torch.version.cuda: 10.2
</code></pre>
<p>My question is how to debug the above errors which are caused by TensorRT?</p>
<p>PS: The source code is available <a href="https://github.com/haotian-liu/yolact_edge" rel="noopener nofollow ugc">here</a></p>

ptrblck: <p>The error seems to be:</p>
<pre><code class="lang-python">/pytorch/aten/src/ATen/native/cuda/Indexing.cu:605: indexSelectSmallIndex: block: [0,0,0], thread: [0,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
</code></pre>
<p>most likely caused in:</p>
<pre><code class="lang-python">x1, x2 = sanitize_coordinates(boxes[:, 0], boxes[:, 2], w, padding, cast=False)
</code></pre>
<p>so I assume the indexing in <code>boxes</code> fails.<br>
You could rerun the script with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and verify the line of code raising this error.</p>

ravijoshi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you very much for the suggestion.</p>
<p>I managed to make it work and noticed something strange during this process as mentioned below:</p>
<p>TensorRT tensor is showing such error when slicing in NumPy fashion. For example <code>my_tensor[idx]</code> is showing error but <code>torch.index_select(my_tensor, 0, idx)</code> is working fine.</p>