RuntimeError: CUDA error: invalid argument when running tracer.export_caffe2

GilbertTanner: <p>When running <code>tracer.export_caffe2()</code> on an <a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">Detecton2 model</a> inside Google Colab I’m getting:</p>
<pre><code class="lang-auto">[11/25 21:54:49 d2.export.caffe2_export]: Exporting a Caffe2GeneralizedRCNN model via ONNX ... Some warnings from ONNX are expected and are usually not to worry about.
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py:269: UserWarning: `add_node_names' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `add_node_names` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py:269: UserWarning: `do_constant_folding' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `do_constant_folding` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-14-9d750fee609f&gt; in &lt;module&gt;()
      1 # convert and save caffe2 model
----&gt; 2 caffe2_model = tracer.export_caffe2()
      3 caffe2_model.save_protobuf(output)
      4 # draw the caffe2 graph
      5 caffe2_model.save_graph(os.path.join(output, "model.svg"), inputs=first_batch)

20 frames
/usr/local/lib/python3.7/dist-packages/detectron2/export/c10.py in _generate_proposals(self, images, objectness_logits_pred, anchor_deltas_pred, gt_instances)
    201                 angle_bound_hi=180,
    202                 clip_angle_thresh=1.0,  # Default
--&gt; 203                 legacy_plus_one=False,
    204             )
    205             rpn_rois_list.append(rpn_rois)

RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>This error didn’t occur when using Detectron v0.5. I suspect this is connected to the PyTorch and Cuda versions used in Colab but I wasn’t able to fix it yet. Any help is greatly appreciated.</p>
<p>Colab notebook: <a href="https://colab.research.google.com/github/TannerGilbert/Object-Detection-and-Image-Segmentation-with-Detectron2/blob/master/Detectron2_train_and_export_model.ipynb#scrollTo=3O22vkBFj_6u" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

ptrblck: <p>I’m not sure, if you would be running into <a href="https://github.com/facebookresearch/detectron2/commit/9fb31b378d50212652cf383f3da76a923d7cc8e9">this issue</a>, but could check if updating to the latest <code>detectron2</code> release could solve it.</p>

GilbertTanner: <p>Sorry for the exceptionally late reply. Completely forgot about this issue. I just tried using the newest version of Detectron2 using <code>!pip install 'git+https://github.com/facebookresearch/detectron2.git'</code> as suggested by <a class="mention" href="/u/ptrblck">@ptrblck</a> but that didn’t solve the issue. Therefore I just opend <a href="https://github.com/facebookresearch/detectron2/issues/3998" rel="noopener nofollow ugc">an issue</a> on the <a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">Detectron2 repository</a>.</p>