PyTorch on CPU uses only python. Why?

Veter: <p>I’m currently using PyTorch on a AMD CPU Epyc Gen2. I notice that sometimes (with a batch of, for example, 200) it uses several cores, but not more than half.</p>
<p>The next point worries me. When I call the top command (I work on Linux), I see that the processor is busy with the Python process.</p>
<p>I expected that PyTorch would compile the neural network for the architecture and that I would see a compiled C or C++ program in the top of list of processes.</p>
<p>What should I change so that PyTorch compiles the neural network into native processor instructions?</p>
<p>I’m currently using torch.compile and default backend, but it doesn’t have any effect.</p>

KFrank: <p>Hi Veter!</p>
<aside class="quote no-group" data-username="Veter" data-post="1" data-topic="197581" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/5daacb/48.png" class="avatar"> Veter:</div>
<blockquote>
<p>When I call the top command (I work on Linux), I see that the processor is busy with the Python process.</p>
</blockquote>
</aside>
<p>This is to be expected.  When you run python (before you have python<br>
execute a script that happens to run some pytorch stuff), the python<br>
interpreter runs as a process.  When you import pytorch (and maybe<br>
when you actually execute some pytorch stuff), the python interpreter<br>
executes some pytorch python code that dynamically loads some<br>
(compiled) c++ libraries (and potentially some cuda stuff) into that<br>
original python process.  So that single python process shows up in<br>
top, and if those c++ pytorch libraries are dominating the use of the<br>
cpu, that cpu usage is still charged to the python process that you<br>
see in top (because, among other reasons, there is no other process).</p>
<aside class="quote no-group">
<blockquote>
<p>I expected that PyTorch would compile the neural network for the architecture</p>
</blockquote>
</aside>
<p>No, the pytorch c++ (and potentially cuda) code is already compiled <em>as<br>
dynamically-loaded libraries.</em>  The basic idea is that for “real” problems,<br>
the compute time is dominated by applying various c++ / cuda layers to<br>
batches of inputs that flow through the network and the overhead of the<br>
(slow) python code that organizes applying those layers is typically not<br>
particularly large.</p>
<aside class="quote no-group">
<blockquote>
<p>I would see a compiled C or C++ program in the top of list of processes.</p>
</blockquote>
</aside>
<p>Again, pytorch does not spawn additional processes to run the model.<br>
The “fast” cpu / cuda code is packaged as libraries and dynamically<br>
loaded into that single python process.</p>
<p>This is fully analogous to using a “python” package such as numpy.<br>
Numpy also delegates its heavy lifting to compiled c++ code that is<br>
dynamically loaded into the single python process.</p>
<aside class="quote no-group">
<blockquote>
<p>What should I change so that PyTorch compiles the neural network into native processor instructions?</p>
<p>I’m currently using torch.compile and default backend, but it doesn’t have any effect.</p>
</blockquote>
</aside>
<p>Typically, pytorch uses python to string together large compute-intensive<br>
chunks and the python overhead doesn’t matter.  In such cases,<br>
<code>torch.compile</code> doesn’t have much to work with to speed things up.<br>
However, if you write some python loops that call lots of little tensor<br>
operations, then <code>torch.compile</code> (depending on the details) may be able<br>
to speed things up a lot.</p>
<p>As an aside, if your model uses big, chunky cuda operations, your cpu may<br>
be largely free while it waits for the cpu, in which case top will not show<br>
your python process as consuming much cpu (because it isn’t).  However<br>
something like nvidia-smi will show your python process consuming a lot<br>
of gpu time.</p>
<p>Best.</p>
<p>K. Frank</p>

Veter: <p>Thank you, K. Frank!</p>