What does fallback_function actually meaning when torch.autograd.profiler.profile called

username: johnnylee
cooked: <p>HI all:<br>
when I compare the inference time cost between libtorch(c++) and torchscript(python)，with LaneGCN network，I found that c++ inference time more slow than python, I have no idea about that.<br>
actually, when compare torchscript with the original ckpt model, torchscript inference sometimes run more slower than ckpt inference, when I use <code>torch.autograd.profiler.profile</code> to debug these slower time point, I got this pictual below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/6846244a304ef666e17524cbbca39b8ee3988671.png" data-download-href="https://discuss.pytorch.org/uploads/default/6846244a304ef666e17524cbbca39b8ee3988671" title="fallback_function"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/6846244a304ef666e17524cbbca39b8ee3988671_2_690x213.png" alt="fallback_function" data-base62-sha1="eSrVXWiILHQKWvhuKmA2fdno7v3" width="690" height="213" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/6846244a304ef666e17524cbbca39b8ee3988671_2_690x213.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/6846244a304ef666e17524cbbca39b8ee3988671_2_1035x319.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/6846244a304ef666e17524cbbca39b8ee3988671_2_1380x426.png 2x" data-dominant-color="E7D1E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fallback_function</span><span class="informations">1760×545 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I search a lot, but don’t know what the fallback_function mean, it cost a lot, does it mean the gpu memory realloc or something else?</p>

username: tevaru
cooked: <p>Hi <a class="mention" href="/u/johnnylee">@johnnylee</a> ,<br>
have found the root cause of your issue and a fix for it? I am observing the same issue when using a torchscript created from a stock pretrained faster rcnn.<br>
Thx, T.</p>

username: tevaru
cooked: <p>A workaround / solution was described in <a href="https://discuss.pytorch.org/t/slow-forward-on-traced-graph-on-cuda-2nd-iteration/118445/7" class="inline-onebox">Slow Forward on traced graph - on cuda, 2nd iteration - #7 by michele-arrival</a> .</p>
<p>EDIT: I corrected the link which I initially intended to post.</p>

username: tom
cooked: <p>Thank you <a class="mention" href="/u/tevaru">@tevaru</a> for linking this.</p>
<p>To also answer the original question: The <code>fallback_function</code> is a function inserted by the PyTorch JIT profiling executor for cases when its optimization assumptions fail. And that probably is the part of the problem with the performance, too…</p>
<p>So as described in the issue disabling the JIT profiling executor / tensor expression fuser one way or the other helps.</p>

username: tevaru
cooked: <p>Thanks <a class="mention" href="/u/tom">@tom</a> for giving this additional answer and context!</p>