Loss abnormal after using batchnorm1d

username: fatmelon
cooked: <p>Hi, I’m training a model with BCEWithLogitsLoss. The model has a head like this:</p>
<pre><code class="lang-python">self.head = nn.Linear(config.hidden_size, num_labels)
</code></pre>
<p>My loss during this training is about 0.3, when I use the head below:</p>
<pre><code class="lang-python">self.head = nn.Sequential(  #
            nn.Linear(config.hidden_size, 150),
            nn.BatchNorm1d(150),
            nn.Linear(150, num_labels),
        )
</code></pre>
<p>The loss goes to 0.6. Is this a normal situation?</p>

username: ptrblck
cooked: <p>Is just the initial loss higher?<br>
How does the training look like? Are you able to get the same or lower loss with the batchnorm layer or does it stay at a higher level constantly?</p>

username: fatmelon
cooked: <p>The training process is roughly as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f470dfcd2593c77833cac976bb9648f763ba153a.png" data-download-href="https://discuss.pytorch.org/uploads/default/f470dfcd2593c77833cac976bb9648f763ba153a" title="58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f470dfcd2593c77833cac976bb9648f763ba153a_2_690x420.png" alt="58" data-base62-sha1="ySqkKAcJVSxAHnsZ7NARAzpcLG2" width="690" height="420" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f470dfcd2593c77833cac976bb9648f763ba153a_2_690x420.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f470dfcd2593c77833cac976bb9648f763ba153a_2_1035x630.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f470dfcd2593c77833cac976bb9648f763ba153a_2_1380x840.png 2x" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">58</span><span class="informations">2244×1366 474 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f69b87ae30ceb08505159d01277d9bba0939c437.png" data-download-href="https://discuss.pytorch.org/uploads/default/f69b87ae30ceb08505159d01277d9bba0939c437" title="53"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f69b87ae30ceb08505159d01277d9bba0939c437_2_690x372.png" alt="53" data-base62-sha1="zbAG6VGG6M5W0Wu8KAtrtTEToG3" width="690" height="372" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f69b87ae30ceb08505159d01277d9bba0939c437_2_690x372.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f69b87ae30ceb08505159d01277d9bba0939c437_2_1035x558.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f69b87ae30ceb08505159d01277d9bba0939c437_2_1380x744.png 2x" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">53</span><span class="informations">2236×1206 413 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The first one uses BN, the second one does not use BN. The training code is long, I send you a message with it. Thank you.</p>

username: ptrblck
cooked: <p>While the training loss seems to be higher, the validation score seems to be better in the batchnorm model.<br>
In that case, I would stick to it and maybe play around with some hyperparameters.</p>

username: fatmelon
cooked: <p>Yes, the batchnorm model has better validation score, but worse loss. Is it because of the multi-gpu?</p>

username: ptrblck
cooked: <p>It’s hard to tell, where exactly this effect comes from and I have not really any idea, so let’s wait for some experts on this topic <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>