Why is torch.jit.script slower?

111480: <p>I compared the performance(speed) of Torchvision’s Squeezenet original model with <code>torch.jit.script(model)</code> which I expected to speed up because Torchscript was asynchronous, but the original model was faster. What’s the reason? Which one did I miss?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41722d8ff4bab290f121fa326bc6a86433b64bbd.png" alt="image" data-base62-sha1="9kXEXZDF9p83VTwX9xMbYgHGSx7" width="590" height="81"></p>
<p>scripted model : 0.2353s<br>
original model : 0.1644s</p>

googlebot: <p>it is not asynchronous (beyond cuda kernel launches, which is not related to jit), just python-less execution mode with optimizations</p>
<p>one thing I’ve seen, is that some jitted operations incorrectly enable requires_grad</p>
<p>but simpler explanation is that you’re not measuring it right - time the THIRD call of compiled model (actually, from your screenshot it seems you’re compiling twice [i.e. two model objects], which is also incorrect). reason is that profiling mode executor creates optimized bytecode on second call.</p>

111480: <p>Thanks for replying.<br>
The time was measured separately between <code>torch.jit.script</code> and the original model.<br>
like this,</p>
<pre><code class="lang-auto">torch.jit.script(models.squeezenet1_0(pretrained=True).cuda().eval())
torch.jit.script(models.squeezenet1_0(pretrained=True).cuda().eval())
#models.squeezenet1_0(pretrained=True).cuda().eval()
#models.squeezenet1_0(pretrained=True).cuda().eval()
</code></pre>
<p>Then result is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee34ec665fdfee6138fb5a5d11cfacf39f13c89c.png" alt="image" data-base62-sha1="xZh1BAAvrLq9EIV67qhmq59l3QM" width="387" height="51"></p>
<pre><code class="lang-auto">#torch.jit.script(models.squeezenet1_0(pretrained=True).cuda().eval())
#torch.jit.script(models.squeezenet1_0(pretrained=True).cuda().eval())
models.squeezenet1_0(pretrained=True).cuda().eval()
models.squeezenet1_0(pretrained=True).cuda().eval()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6db01c642701f7c4be723ce92456dff790ca3ba6.png" alt="image" data-base62-sha1="fElmkpz8S52BrQg9uOX2Q6CMg18" width="320" height="52"></p>
<p>Why exactly is Torchscript slower than pytorch?</p>

googlebot: <p>as I said, do something like:<br>
net = jit.script(models.squeezenet1_0(pretrained=True).cuda().eval())<br>
net(x); net(x)</p>
<p>and only then measure time (e.g. %timeit net(x))</p>

111480: <p>As you say, Torchscript is faster.<br>
Is this the result of different ways of optimizing it?<br>
I want a detailed explanation.</p>

googlebot: <p>In a nutshell,</p>
<ol>
<li>“compilation” analyzes whole functions, with knowledge about variable types - some optimizations are done at this level (e.g. dead code elimination)</li>
<li>python bytecode interpreter is not used to execute generated code - more specialized executor for statically typed code supposedly works faster</li>
<li>fusion optimizations further compile specialized cuda kernels, so e.g. a.mul(b).add(c) is computed in one go</li>
<li>some patterns have specialized optimizations, e.g. conv+batchnorm</li>
</ol>

vince62s: <p>Hi I am hijacking this thread for a similar question.</p>
<p>I ran the code of this link: <a href="https://gist.github.com/ResidentMario/1a4f6473828048990e26d12d58d7a227" class="inline-onebox" rel="noopener nofollow ugc">pytorch-jit.ipynb · GitHub</a></p>
<p>exact same time with and without, not seeing the perf difference as per the link.</p>
<p>any clue?<br>
(running on pytorch 1.12.1 cuda 11.3)</p>

ptrblck: <p>Your notebook seems to show a ~2x speedup so I’m unsure how to interpret your post. Did you see any other results when properly profiling with syncs?</p>

vince62s: <p>this is not “my” notebook. this is someone else’s noebook that I tried on my machine, hence not replicating the speedup.</p>