Model bundle performance / multi-core inference

sheh: <p>Hello!</p>
<p>I am trying to use Glow in order to increase inference speed for a custom pytorch model based on mobilenet architecture. The model was converted to onnx format and I modified <a href="https://github.com/pytorch/glow/tree/master/examples/bundles/resnet50" rel="nofollow noopener">resnet50 example</a> to get a model bundle for my model.<br>
Inference time for original traced pytorch model is <strong>~27ms</strong> per image. For the compiled bundle inference time is <strong>~800ms</strong>.<br>
Taking in to account that glow uses only 1 core during inference and I have 12-core (24 thread) CPU, the expected performance for multi-core inference using glow is going to be 800 ms / 24 ~ <strong>33 ms</strong> (it is the lowest estimate).</p>
<p>My questions:</p>
<ol>
<li>Performance of the compiled model is worse comparing with the original pytorch model. Is it expected? Have I missed some optimization steps? Should I use another way to get better performance?</li>
<li>How can I run multi-core inference using Glow with CPU backend?</li>
</ol>

jfix: <blockquote>
<ol>
<li>Performance of the compiled model is worse comparing with the original pytorch model. Is it expected? Have I missed some optimization steps? Should I use another way to get better performance?</li>
</ol>
</blockquote>
<p>PyTorch works hard to implement optimized kernels for CPU. In Glow, we haven’t spent much time recently working on optimizing our Conv kernels for our CPU backend. We have focused much more on custom HW AI accelerators such as the NNPI and Habana backends in Glow. IIRC, last we focused on CPU perf a couple years ago it was originally tested on a few specific models of x86 CPUs. So your perf is going to depend on what CPU you’re using.</p>
<blockquote>
<ol start="2">
<li>How can I run multi-core inference using Glow with CPU backend?</li>
</ol>
</blockquote>
<p>We don’t currently support this. See prior discussion here: <a href="https://github.com/pytorch/glow/issues/1749" class="inline-onebox" rel="noopener nofollow ugc">Multi-threaded CPU kernels · Issue #1749 · pytorch/glow · GitHub</a></p>

tkg.rahul: <aside class="quote no-group" data-username="jfix" data-post="2" data-topic="81422">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jfix/48/21510_2.png" class="avatar"> jfix:</div>
<blockquote>
<p>PyTorch works hard to implement optimized kernels for CPU. In Glow, we haven’t spent much time recently working on optimizing our Conv kernels for our CPU backend. We have focused much more on custom HW AI accelerators such as the NNPI and Habana backends in Glow. IIRC, last we focused on CPU perf a couple years ago it was originally tested on a few specific models of x86 CPUs. So your perf is going to depend on what CPU you’re using.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/jfix">@jfix</a> does this mean any evaluation board having any kind of cpu i.e. arm cortex won’t be able to take advantage of glow in inference?</p>

jfix: <p>You can absolutely run on CPU architectures, and performance should still be pretty good, just might not be as good as other frameworks. You could always modify/add to the Glow CPU backend’s libjit with kernels for ops you care about that are better targeted for ARM.</p>
<p>I know NXP has had a lot of success targeting ARM Cortex with Glow: <a href="https://media.nxp.com/news-releases/news-release-details/industrys-first-mcu-based-implementation-glow-neural-network" rel="nofollow noopener">https://media.nxp.com/news-releases/news-release-details/industrys-first-mcu-based-implementation-glow-neural-network</a> – maybe worth posting an issue on Glow’s GH with tagging mciprian13 from NXP to see if you can find out more.</p>

tkg.rahul: <p>Thanks a lot <a class="mention" href="/u/jfix">@jfix</a> for quick response. I will check NXP offerings.<br>
Does glow still uses one cpu core for inference? I can see multi core inference issue is still open on github.</p>

jfix: <p>Yes, we didn’t really invest much in multicore execution. We have focused on accelerators such as NNPI and Habana. I don’t expect it’d be hard to go data parallel and split a batch by the number of cores you have.</p>