Different network output with using batch or not

Terbe_Daniel: <p>I train my model with batch size 128, however if I don’t use batch in the evaluation phase, the network output is wrong.</p>
<p>If the network’s input is in batches:</p>
<pre><code class="lang-auto">crit = nn.MSELoss(reduction='mean')
target = []
netout = []

model.eval() # To handle drop out layers and batch norm
for A, M, label in dataloaders['val']:
    with tr.set_grad_enabled(False): # We don't need gradient computation in eval mode (speed up)
        out = model(A, M)
    
        target.extend(label.data.cpu().numpy())
        netout.extend(out.data.cpu().numpy())

print(tr.Tensor(target).shape, tr.Tensor(netout).shape)
print(f'Loss: {crit(tr.Tensor(target), tr.Tensor(netout))}')
</code></pre>
<p><strong>Result:</strong></p>
<blockquote>
<p>torch.Size([849]) torch.Size([849, 1])<br>
Loss: 1.135872483253479<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d173d070a2b4b90626f7bf552d78256788a9450b.png" data-download-href="https://discuss.pytorch.org/uploads/default/d173d070a2b4b90626f7bf552d78256788a9450b" title="batch_input"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d173d070a2b4b90626f7bf552d78256788a9450b_2_345x183.png" alt="batch_input" data-base62-sha1="tSTVaIuZ8fdCJeQohZcqTlDXIYr" width="345" height="183" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d173d070a2b4b90626f7bf552d78256788a9450b_2_345x183.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d173d070a2b4b90626f7bf552d78256788a9450b_2_517x274.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d173d070a2b4b90626f7bf552d78256788a9450b_2_690x366.png 2x" data-dominant-color="3B465F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">batch_input</span><span class="informations">708×376 82.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>
<p>If I iterate through the single elements, I attain different result:</p>
<pre><code class="lang-auto">crit = nn.MSELoss(reduction='mean')
target = []
netout = []

model.eval() # To handle drop out layers and batch norm
 for A, M, label in testset:
     with tr.set_grad_enabled(False): # We don't need gradient computation in eval mode (speed up)
         out = model(tr.unsqueeze(A, 0), tr.unsqueeze(M, 0))
        
     target.append(label.item())
     netout.append(out.item())
    

print(tr.Tensor(target).shape, tr.Tensor(netout).shape)
print(f'Loss: {crit(tr.Tensor(target), tr.Tensor(netout))}')
</code></pre>
<p><strong>Result</strong></p>
<blockquote>
<p>torch.Size([849]) torch.Size([849])<br>
Loss: 87.88565063476562<br>
<a href="/uploads/short-url/d8EOWbBPkZovUeiMrUWm4aNWUze.png">upload://d8EOWbBPkZovUeiMrUWm4aNWUze.png</a></p>
</blockquote>
<p>But the result should be exactly the same, because the output is deterministic!<br>
What could be the problem? I can overfit the model for a specific batch size, but if I evaluate on single elements, then the result is wrong.</p>

ptrblck: <p>Could you print the shape of <code>tr.Tensor(target)</code> and <code>tr.Tensor(netout)</code> before passing them to the criterion?<br>
Also, which criterion are you using at the moment?</p>

Terbe_Daniel: <p><code>crit = nn.MSELoss(reduction='mean')</code><br>
For batch input:<br>
<em>torch.Size([849]) torch.Size([849, 1])</em></p>
<p>For single input:<br>
<em>torch.Size([849]) torch.Size([849])</em></p>
<p>I edited the post to contain these information.</p>

ptrblck: <p>It seems you might be accidentally broadcasting the inputs to your criterion.<br>
In the latest PyTorch version (<code>1.2.0</code>) you should get a warning.<br>
Make sure to pass the input and target as <code>[batch_size, 1]</code> or <code>[batch_size]</code> (not mixed).</p>

Terbe_Daniel: <p>Yes, but if a correct this (with <code>squeeze</code>), I get a correct loss:<br>
<code>Loss: 0.8491315245628357</code></p>
<p>But the issue remains – the network output differs when batched or single input.</p>

ptrblck: <p>That shouldn’t be the case.<br>
Could you post the model architecture so that we could have a look?</p>

slavavs: <p>I have already spent a lot of time on this problem. My criterion CrossEntropyLoss. The input is [batch, num_class], the target is [batch]. Everything works well on batch training and testing. On a single package does not work. I think that NN sees the data inside the batch. We have to train the model with the size of the batch = 1.</p>

ptrblck: <p>If you have a reproducible code snippet, we could look into it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Terbe_Daniel: <p>I think, I found the bug:</p>
<pre><code class="lang-auto">        _, _, H, W = A.shape
        mask2 = tr.div(mask2*H*W, 2*tr.norm(mask2, p=1))
</code></pre>
<p>Here I calculate the norm overall what is wrong!<br>
I’ll try to correct this – calculate the norm of the mask for each batch…</p>

Terbe_Daniel: <p>This was the problem! Now working with this:</p>
<pre><code class="lang-auto">        B, _, H, W = A.shape
        norm =  2*tr.norm(mask1, p=1, dim=(1,2,3))
        norm = norm.reshape(B, 1, 1, 1)
        mask1 = tr.div(mask1*H*W, norm)
</code></pre>
<p>Thank you for your help!</p>

ptrblck: <p>I’m glad it’s working now! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>