Training is slow

pikaqi: <p>I feel my training is very slow then I checked the gpu and cpu utilization and found the cpu utilization is abnormally high. I thought the code used for calculating loss could be optimize but I don’t understand which operation should I change.</p>
<pre><code>def ctr(self, q, k, tau=1):
    logits = torch.mm(q, k.t())
    labels = torch.LongTensor(range(q.shape[0])).cuda()
    loss = self.criterion(logits/tau, labels)
    return 2*tau*loss
</code></pre>
<p>Also I checked the pytorch profile and found the training is unstable. The time consuming is commonly in microsecond, but scale in millisecond happens about every 8 batches. Is this a problem?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f7abdc700c28fe1479c11d1893443d8d75fdfe9.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f7abdc700c28fe1479c11d1893443d8d75fdfe9" title="ms_us"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f7abdc700c28fe1479c11d1893443d8d75fdfe9_2_690x295.png" alt="ms_us" data-base62-sha1="4utLXT8NctXRBj9DgOXTgmh6QLn" width="690" height="295" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f7abdc700c28fe1479c11d1893443d8d75fdfe9_2_690x295.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f7abdc700c28fe1479c11d1893443d8d75fdfe9_2_1035x442.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f7abdc700c28fe1479c11d1893443d8d75fdfe9.png 2x" data-dominant-color="383838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ms_us</span><span class="informations">1171×502 175 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Kushaj: <p>Create the <code>labels</code> directly on GPU. When you use <code>.cuda()</code>, you are first creating the tensor on CPU and then moving the tensor to GPU which is slow. So instead use this</p>
<pre><code class="lang-auto">labels = torch.tensor(range(q.shape[0]), dtype=torch.int64, device=q.device)
</code></pre>
<p>(There is nothing special about using <code>torch.tensor</code>)</p>

pikaqi: <p>The training is faster slightly.<br>
Another question is why the profile shows I used two ‘to’ operation. I thought I  used ‘to’ only when I create labels.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfb787cae70b43b9f03b4e503eb3935278483950.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cfb787cae70b43b9f03b4e503eb3935278483950" title="two"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb787cae70b43b9f03b4e503eb3935278483950_2_690x462.jpeg" alt="two" data-base62-sha1="tDy30zFe8OwHXeOPWQnIDKqcmju" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb787cae70b43b9f03b4e503eb3935278483950_2_690x462.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cfb787cae70b43b9f03b4e503eb3935278483950_2_1035x693.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfb787cae70b43b9f03b4e503eb3935278483950.jpeg 2x" data-dominant-color="4A4A4B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">two</span><span class="informations">1278×857 248 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Kushaj: <p>I do not know about why two <code>to</code> operations are used? If you want to debug the reason, you can try commenting out lines in <code>ctr</code> and see when the number of calls reduces.</p>