ML Inference not scaling with multi threading

username: Anish_Reddy_Ellore
cooked: <p>I’ve created a C++ web server which performs ML inference using Torch, and I’m using a BERT model for prediction. In my experiments, I found that my server is not scaling as expected. For one request, it takes 50 ms, but for 10 parallel requests, it is taking about 500 ms per request. I’ve tested my C++ server with other CPU-intensive workloads, and my results are good, but for ML inference, the server does not scale well. My C++ server creates one thread per connection, and I have 50 cores, so I am expecting my inference server to run 10 requests in parallel. Please find the below CDF for 5, 10, 30, 50 requests.</p>
<p>It would be great if someone can tell me what I’m doing wrong.<br>
1.) Is it even possible to do C++ inference with multi threading that scales well or is there something in ML inference that is stopping me from doing so.<br>
2.) I’m using multi threading to reduce model memory duplication and to avoid model loading times.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e10a1b25ca887628c2746aa5803dd76c775e148.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e10a1b25ca887628c2746aa5803dd76c775e148" title="Screenshot 2023-05-08 at 9.35.20 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e10a1b25ca887628c2746aa5803dd76c775e148_2_585x500.png" alt="Screenshot 2023-05-08 at 9.35.20 PM" data-base62-sha1="4hXXlwmOdim7OrOwpkTj6TcH4jm" width="585" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e10a1b25ca887628c2746aa5803dd76c775e148_2_585x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e10a1b25ca887628c2746aa5803dd76c775e148_2_877x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e10a1b25ca887628c2746aa5803dd76c775e148_2_1170x1000.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-05-08 at 9.35.20 PM</span><span class="informations">1367×1167 88.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: eqy
cooked: <p>I think one potential issue is that numerical CPU backends used by PyTorch such as MKL/oneDNN may already use more than one thread per process by default. Do you see scaling behavior that is closer to what is expected when setting e.g., <code>OMP_NUM_THREADS=1</code> and <code>MKL_NUM_THREADS=1</code>?</p>
<p>See also:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.set_num_threads.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.set_num_threads.html</a></p>

username: Anish_Reddy_Ellore
cooked: <p>I am using torch module in C++, but the link suggests a modification in Python. So, can you please point me to how to do the same in C++? Also, it would be great if you could point me to any production C++ inference system which uses multi-threading. Thanks for taking the time to reply to my query.</p>

username: eqy
cooked: <p>You may want to try:<br>
<code>torch::set_num_threads</code>.</p>
<p>As for a popular inference system there is Triton which e.g., has analogous concurrency support: <a href="https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/optimization.html" rel="noopener nofollow ugc">https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/optimization.html</a></p>

username: Anish_Reddy_Ellore
cooked: <p>I changed num_threads to 1 and my server is scaling well now. Thanks for all your help.</p>