Autograd profiler shows high usage of "to"

username: steven-lang
cooked: <p>I am profiling to my code in the training loop during a single forward pass like the following:</p>
<pre><code class="lang-python">with torch.autograd.profiler.profile(use_cuda=False) as prof:
    y = model(x)
print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=25))
</code></pre>
<p>The top entry in the printed table is the function <code>to</code> with 32% self CPU total usage and 4500 calls. This is weird to me since nowhere in my model code I call the <code>.to(device)</code> function (only earlier on before the profiling is started).</p>
<p>Here is the output: <a href="https://pastebin.com/raw/Y6Yv3FGe" rel="nofollow noopener">https://pastebin.com/raw/Y6Yv3FGe</a></p>
<p>How can I find out which PyTorch call may internally calls <code>.to(device)</code> without me doing this explicitly?</p>
<p>Cheers</p>
<p>Edit: 
I just found the reason. When my model is already on a cuda device and the forward pass is being called in a profiling context where <code>use_cuda=False</code>, all Tensors are implicitly sent back to the CPU. The above does not happen when my model is not on a cuda device first.
<br>
Edit 2: Nevermind, the <code>to</code> usage is still high when using <code>use_cuda=True</code> and everything is already on the device a priori.</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>use_cuda</code> wil only impact the profiler: it will only allow the profiler to properly time cuda calls or not.</p>
<p>Is your model wrapped in a DataParallel or similar construct?  Could you give more details about your model?</p>

username: steven-lang
cooked: <p>No the model is not wrapped in DataParallel (or anything else).</p>
<p>Which details do you need? I can’t publish the model code (yet) unfortunately, sorry.<br>
It’s a deep model using multiple custom layers performing lots of sum, matmul, exp, einsum and log operations. Nothing fancy per se. We mainly use pytorch to obtain the gradients via autograd.</p>

username: albanD
cooked: <p>The thing is that internally. We try and never call <code>.to</code> under the hook to avoid performance pitfalls. DataParallel is an exception has it needs to move stuff between gpus.</p>
<p>So one forward in your model still calls <code>4500</code> times <code>.to()</code> ?<br>
Note as well that if you do many <code>.float()</code> or <code>.long()</code>, etc, they get converted to <code>.to</code> operators I think.</p>

username: steven-lang
cooked: <p>We call <code>.float()</code> and <code>.long()</code> quite a few times. That explains it. I didn’t that that it calls <code>.to</code> internally (see <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.long" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.long</a>).</p>
<p>Do you happen to know why these calls bother the cpu so much? The tensors they are called upon all reside on a cuda device.</p>

username: albanD
cooked: <p>Ho the doc actually shows their equivalent <code>.to()</code> formulation.</p>
<p>If you set <code>use_cuda=True</code> then every operation will block on the GPU. So you can see how long they take.<br>
These calls make a full copy of the given Tensor every time they’re called. So if you do too many of them, they can slow you down. Especially if your overall task is memory bounded.</p>

username: steven-lang
cooked: <p>That’s good to know. Thanks for your support!</p>

username: steven-lang
cooked: <p>Well, I have to correct my above statement. We neither call <code>.float()</code> nor <code>.long()</code> in the forward pass that is being profiled.</p>
<p>Is there any way to trace, where a <code>.to()</code> call is being done internally?</p>

username: albanD
cooked: <p>I don’t think there is.<br>
You can try to profiler subsets of your model to pinpoint the issue and send the problematic part if it is small enough.</p>