Monitoring loss without synchronisation points

sebquet: <p>Hi,<br>
I am wondering what would be the best way to save the loss of every batch inside a training loop without creating a synchronisation point between GPU and CPU. Because the way I was doing it was to create a list on the CPU and adding every batch loss as a tensor to this list during the training loop (without calling .item or .cpu methods). But I am afraid that the CPU still has to wait for the GPU to finish the backward pass before being able to add an element to its list.</p>
<p>To put things in context, I have a dataloader and I want to use multiprocessing dataloading ( <code>torch.utils.data.DataLoader</code> with num workers &gt;0 ) for the GPU not to wait for data and be used asynchronously with the CPU.</p>
<p>My code looks like this :</p>
<pre><code class="lang-auto">model = model.cuda()
device = torch.device("cuda")
batch_losses = []
for idx, (input, label) in enumerate(tqdm(datasetloader)):

     input = input.to(device)
     optim.zero_grad(set_to_none=True)
     prediction = model(input)
     loss = loss_func(input, prediction) 
     loss.backward()
     optim.step()
     batch_losses.append(loss.detach())
</code></pre>
<p>I guess my first question is : Does appending an element to a list on CPU create a synchronisation point with the GPU? I was not able to use <code>torch.cuda.set_sync_debug_mode</code> with my torch version.<br>
If yes, would the solution be to create a tensor on the GPU before the loop, concatenating detached loss tensors at every iteration with this tensor and transfer it to CPU after the training loop?</p>
<p>Also, does using tqdm create synchronisation point?</p>
<p>Thank you for your help.</p>

ptrblck: <aside class="quote no-group" data-username="sebquet" data-post="1" data-topic="152702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sebquet/48/51908_2.png" class="avatar"> sebquet:</div>
<blockquote>
<p>Does appending an element to a list on CPU create a synchronisation point with the GPU?</p>
</blockquote>
</aside>
<p>Appending a CPUTensor to a <code>list</code> doesn’t create the synchronization point, but moving a CUDATensor to the CPU will.</p>
<aside class="quote no-group" data-username="sebquet" data-post="1" data-topic="152702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sebquet/48/51908_2.png" class="avatar"> sebquet:</div>
<blockquote>
<p>was not able to use <code>torch.cuda.set_sync_debug_mode</code> with my torch version.</p>
</blockquote>
</aside>
<p>You might want to update PyTorch to be able to use this debug utility.</p>
<aside class="quote no-group" data-username="sebquet" data-post="1" data-topic="152702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sebquet/48/51908_2.png" class="avatar"> sebquet:</div>
<blockquote>
<p>If yes, would the solution be to create a tensor on the GPU before the loop, concatenating detached loss tensors at every iteration with this tensor and transfer it to CPU after the training loop?</p>
</blockquote>
</aside>
<p>Your current approach should not create a synchronization point as you are not moving the data to the CPU as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.cuda.set_sync_debug_mode(debug_mode="default")

# setup
model = nn.Linear(1, 1).cuda()
x = torch.randn(10, 1, 1, device='cuda')
y = torch.zeros(1).long().cuda()

torch.cuda.set_sync_debug_mode(debug_mode="error")
losses = []
for i in range(len(x)):
    x_ = x[i]
    out = model(x_)
    loss = F.cross_entropy(out, y)
    losses.append(loss.detach()) # works
    # losses.append(loss.cpu().detach()) # RuntimeError: called a synchronizing CUDA operation
</code></pre>
<aside class="quote no-group" data-username="sebquet" data-post="1" data-topic="152702">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sebquet/48/51908_2.png" class="avatar"> sebquet:</div>
<blockquote>
<p>Also, does using tqdm create synchronisation point?</p>
</blockquote>
</aside>
<p>I don’t know but wouldn’t think so. Once you’ve updated your PyTorch version you could check it with the debug mode.</p>

sebquet: <p>Thank you very much for these invaluable insights!</p>