Applying model() to batches changes loss behavior unexpectedly?

primoz: <p>I am trying to change the following line (because I am running out of memory sometimes when I try to run it):</p>
<pre><code class="lang-auto">_, held_out_features, _ = model(held_out_data)
</code></pre>
<p>This results in the following loss behavior:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/0718c0160a5b8e48d26cb827024411108ca4ca35.png" data-download-href="https://discuss.pytorch.org/uploads/default/0718c0160a5b8e48d26cb827024411108ca4ca35" title="Screen Shot 2022-07-27 at 3.49.02 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/0718c0160a5b8e48d26cb827024411108ca4ca35_2_296x500.png" alt="Screen Shot 2022-07-27 at 3.49.02 PM" data-base62-sha1="10MmUUxRg80EHymsw4k84HivxVr" width="296" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/0718c0160a5b8e48d26cb827024411108ca4ca35_2_296x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/0718c0160a5b8e48d26cb827024411108ca4ca35_2_444x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/0718c0160a5b8e48d26cb827024411108ca4ca35_2_592x1000.png 2x" data-dominant-color="E5E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-07-27 at 3.49.02 PM</span><span class="informations">600×1012 57.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When trying to change this line to something like:</p>
<pre><code class="lang-auto">temp_loader = DataLoader(held_out_data,batch_size = 100)
held_out_features = torch.empty(0, dtype = torch.float, device=device)

for _,data in enumerate(temp_loader):
    held_out_features = torch.cat((held_out_features, model(data)[1]),0)
</code></pre>
<p>I’ve also tried:</p>
<pre><code class="lang-auto">held_out_features = torch.empty(0, dtype = torch.float, device=device, requires_grad=True)

for j in range(0,len(held_out_data),100): 
    data = held_out_data[j:j+100,:,:,:]
    _,temp,_ = model(data)
    held_out_features = torch.cat((held_out_features, temp),0)
</code></pre>
<p>Both of these relieve the memory issue, but then the loss doesn’t decrease steadily (or at all) as before - it hovers around 1.1 throughout mini batches/epochs (I’d post another image but since I’m a new user I can’t).</p>
<p>What is going on here? How can I fix it?</p>

ptrblck: <p>I don’t know how exactly the first code looks like, but the two suggested workarounds would accumulate the computation graphs for the entire dataset and I would assume you would be running out of memory (assuming the first approach trains on each batch).<br>
Could you describe the issue in more detail and explain how the first approach is written?</p>

primoz: <p>The code above appears in a <code>train</code> function. <code>held_out_data</code> is the tensor we would like to feed into <code>model</code>. It contains roughly 900 images. For the purposes of the the project I’m working on, it’s important that we process all of them each training iteration. We make predictions on <code>held_out_data</code> and that’s how we generate loss. When trying the first approach, we see the loss decreasing, but the runtime is very slow and occasionally runs out of memory. When we switch to the 2nd/3rd approach, it doesn’t run out of memory (because we process in groups of 100 instead of all 900 at once), but the loss doesn’t decrease, even though <code>held_out_features</code> should be the same in either case. Let me know if you need further clarification.</p>