What is aten:to in the profiler

Alexander_Soare: <p>This is a profile of</p>
<pre><code class="lang-auto">with profiler.profile(record_shapes=True) as prof:
    with profiler.record_function("model_train"):
        out = model.forward(data[0].to(DEVICE))
        loss = compute_loss(data[1:], out)
        loss.backward()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50da0564784e92798a60dd167c65c7b6dd0d9df1.png" data-download-href="https://discuss.pytorch.org/uploads/default/50da0564784e92798a60dd167c65c7b6dd0d9df1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50da0564784e92798a60dd167c65c7b6dd0d9df1.png" alt="image" data-base62-sha1="bxfiWLSeoAt3dRWtQm3AMbbNsMp" width="690" height="218" data-dominant-color="E3EBE7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">733×232 8.98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What’s up with the long <code>aten:to</code> block in the middle? At first I thought it was the operation of copying tensors from cpu to gpu. This still makes sense to me, but I don’t understand why the second block is so long.</p>
<p>As part of computing my loss I do a <code>to(DEVICE)</code> on my targets, but that’s supposed to be relatively very small.</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss(ignore_index=tkn.stoi['&lt;pad&gt;'])

def compute_loss(targets, outputs):
        # targets shape [(B, padded_seq_length), (B, 1)]
        # outputs shape (B, S, N+1) S is sequence length of output, N is n_classes
        caption_lengths, sort_ind = targets[1].squeeze(1).sort(dim=0, descending=True)
        decode_lengths = (caption_lengths - 1).tolist()
        predictions = pack_padded_sequence(outputs[sort_ind], decode_lengths, batch_first=True).data
        targets = pack_padded_sequence(targets[0][sort_ind][:, 1:].to(DEVICE), decode_lengths, batch_first=True).data
        loss = criterion(predictions, targets)
        return loss
</code></pre>
<p>Then, I tried returning <code>loss=0</code> without actually computing a loss like this:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss(ignore_index=tkn.stoi['&lt;pad&gt;'])

def compute_loss(targets, outputs):
        # targets shape [(B, padded_seq_length), (B, 1)]
        # outputs shape (B, S, N+1) S is sequence length of output, N is n_classes
        caption_lengths, sort_ind = targets[1].squeeze(1).sort(dim=0, descending=True)
        decode_lengths = (caption_lengths - 1).tolist()
        predictions = pack_padded_sequence(outputs[sort_ind], decode_lengths, batch_first=True).data
        targets = pack_padded_sequence(targets[0][sort_ind][:, 1:].to(DEVICE), decode_lengths, batch_first=True).data
        return nn.Parameter(torch.tensor([0]).float())
</code></pre>
<p>This actually gets rid of the second <code>aten:to</code> block</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76c906676c619821fa1c9646815693a1112090f8.png" alt="image" data-base62-sha1="gWP2cVRzaOJledohIX6gNoY3Xle" width="682" height="152"></p>
<p>So why is the loss taking so long to compute? And was I right in saying that <code>aten:to</code> is the cpu &gt; gpu operation? If so, what does this have to do with calculating the loss?</p>
<p>On a side note, I noticed that as I scale my model size, that second aten:to block takes longer to do (proportionally to the total time of the loop iteration). Why should it, if the loss is calculated independently of model size?</p>

grudloff: <p>Hello <a class="mention" href="/u/alexander_soare">@Alexander_Soare</a>! Did do manage to catch what is the source of that long <code>aten::to</code> block? I am running into a similar issue.</p>

Alexander_Soare: <p>Nope sorry. This problem has been long forgotten.</p>

lellowood: <p>I encountered the same issue, which was ultimately traced back to an operation involving copying a variable across different devices, specifically: tensor.item().</p>

ptrblck: <p>In case other users struggle to isolate the place where an operation or kernel is called from: you can enable Python backtraces in Nsight Systems allowing you to hover over each kernel and showing the backtraces through PyTorch’s C++ backend as well as your Python scripts.</p>