Speed of First pass is very slow

huoge: <p>Hello everyone. I recently loading a torchscript model in C++, when i use the model to infer, the first pass takes about 20s, while the others take only about 0.5s.</p>
<p>Has anyone ever done any related work or met the same problem?</p>
<p>is there any way to disable the optimization or choose the optimization level or after optimization we can save the model(or the computation graph)?</p>
<p>or is it an Inevitable warm-up process?</p>
<p>I’d appreciate if anybody can help me! Thanks in advance!</p>

James_Reed: <p>Hello huoge,</p>
<p>It’s likely that you’re correct in the assessment that optimization is what’s making the first pass slow. However, 20 seconds seems pretty high and we’d like to understand what exactly is happening here. Do you mind sharing your serialized model file so we can have a look? Also, which version of PyTorch are you using?</p>
<p>We don’t currently have an optimization level option.</p>
<p>James</p>

huoge: <p>Hello  James,</p>
<p>Thanks for replying me, I build the pytorch from source and the version is 1.4.0a0+93db2b8<br>
the model is a modified transformer. I transfer the model to torchscript model by using script_model = torch.jit.script(model). (and definitely some other work for jit-compatible)</p>
<p>by the way, I use the torch.quantization.quantize_dynamic to quantize the model, then the first pass<br>
costs about 12.5s.</p>
<p>this is parts of print(script_model):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c86884e2c7d84b1b529fc8924a2a86abb7cadbe.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c86884e2c7d84b1b529fc8924a2a86abb7cadbe" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c86884e2c7d84b1b529fc8924a2a86abb7cadbe.png" alt="image" data-base62-sha1="44lBqfgoOfnoAUxXDponBKetOVM" width="320" height="500" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">505×787 9.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I only use torch.jit.script(), should I mix the trace and script?</p>
<p>and how can I submit the serialized model file to you? for some reason, I can’t give you the<br>
modifiled transformer, but I can provide the orginal transformer serialized model file for you which first pass costs 32.83s while the others take about 9s(yeah, it’s about 20s again). The origin and modifiled transformer are same net architecture but have different inference functions, so  they have same problem in first pass.</p>

huoge: <p>Can anyone help me with this issue?</p>

Pavel_Platonov: <p>Do you solve a “warm up” scripted model problem?<br>
Simplest way to share this problem is:<br>
<code><br>
import torchvision, torch, time</code></p>
<p>model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)<br>
model = torch.jit.script(model)<br>
model.eval()<br>
x = [torch.randn((3,224,224))]<br>
for i in range(3):<br>
start = time.time()<br>
model(x)<br>
print(‘Time elapsed: {}’.format(time.time()-start))<br>
</p>
<p>Output:<br>
<i> Time elapsed: 38.297527551651<br>
Time elapsed: 6.655704021453857<br>
Time elapsed: 6.651334762573242</i></p>
<p>So, can anybody help with explaining how I can load and run scripted model without this "warm up’?<br>
Thanx.</p>

G.M: <p>Maybe this post helps? <a href="https://discuss.pytorch.org/t/speed-of-custom-rnn-is-super-slow/63209/9" class="inline-onebox">Speed of Custom RNN is SUPER SLOW</a></p>

Pavel_Platonov: <p>Thanks, with torch.jit.optimized_execution(False): really helped.</p>

HMS97: <p> really helpful</p>