BCE loss stuck at 0.693 in the beginnng of training and then started to decrease, why?

Cyber_punk: <p>I’m using a Transformer encoder with a binary cross entropy loss for CTR prediction. The training batch loss is at around 0.693 constantly for the beginning several thousand steps (batches). I’m using Noam learning rate schedule and a <a href="https://docs.allennlp.org/main/api/training/learning_rate_schedulers/noam/" rel="noopener nofollow ugc">scaling factor</a> for setting the learning rate used by Adam. The first 3000 steps are the warm up steps in Noam:</p>
<blockquote>
<p>lr= factor * ( model_size ** (-0.5) * min(step** (-0.5), step *<br>
warmup_steps ** (-1.5)) )</p>
</blockquote>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa8b0cffba552cb5f8e21abe0fa4eeaf578a8f49.png" alt="image" data-base62-sha1="okH5gkdaD4l7bKBVshZHsg9Avwt" width="569" height="431"></p>
<p>For the learning rate, when setting scaling factor to be 0.5 or 0.6, the period with constant 0.693 is from beginning to 5500th steps. When I use lower lr by setting scaling factor 0.3, the period with constant 0.693 became longer (8000 steps). When I use higher lr by setting scaling factor 1.0, the period with constant 0.693 became shorter (4000 steps).</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3dfe81b97947a45deaaa0bd5b87ac897e9f9cfe1.png" alt="image" data-base62-sha1="8Qqp2soqRwux5cQK6JIXP2KKnFD" width="401" height="326"><br>
Note: the above training batch loss curve is smoothed.</p>
<p>Does this mean I should use a high learning rate (e.g., with scale factor being 1.0)?<br>
Why there is a period with constant 0.693 loss? is it because the learning rate is too small to make any adjustment to the parameters? If so, after the 3000 warmup steps the learning rate is even smaller, why the loss is decreasing?</p>

KFrank: <p>Hi Cyber!</p>
<aside class="quote no-group" data-username="Cyber_punk" data-post="1" data-topic="183958" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/258eb7/48.png" class="avatar"> Cyber_punk:</div>
<blockquote>
<p>I’m using a Transformer encoder with a binary cross entropy loss for CTR prediction. The training batch loss is at around 0.693 constantly for the beginning several thousand steps (batches).</p>
</blockquote>
</aside>
<p>Note that <code>log (1/2) = -0.693147</code>, which is the same as<br>
<code>torch.nn.functional.logsigmoid (torch.tensor (0.0))</code>.</p>
<p>This is telling us that your model is predicting values that are zero (or<br>
that are tightly clustered around zero).  Look at the values your model<br>
is predicting during its “constant period.”</p>
<p>How are you initializing the weights of your model?  You typically want<br>
to initialize your weights randomly so that you don’t get stuck in some<br>
“symmetric” configuration (that, perhaps, predicts zeros).  If you are<br>
using random initialization, you might want to try increasing the scale<br>
(typical size or standard deviation) of those random weights.  Pay<br>
particular attention to the last few layers of your model.  Does your<br>
output layer have a <code>bias</code>?</p>
<p>Do you have <code>ReLU</code>s or other activations that could be starting off<br>
“saturated” at or near zero, especially near the end of your model?</p>
<p>(Using a higher learning rate could certainly get you out of your<br>
“constant period” faster, but it would seem to be a better approach<br>
not to start out in a predict-zero phase in the first place.)</p>
<p>Best.</p>
<p>K. Frank</p>

Cyber_punk: <p>Thanks for your informative analysis!</p>
<p>The input layers are embedding layers initialized with <code>uniform(-0.1, 0.1)</code>:</p>
<pre><code class="lang-auto">init_range = 0.1
self.job_id_embedding_layer.weight.data.uniform_(-init_range, init_range)
</code></pre>
<p>The intermediate layers are a Transformer encoder block. I didn’t change the initialization methods for the weights.</p>
<p>The last few layers are a 3-layer FFNN block:</p>
<pre><code class="lang-auto">self.linear = nn.Sequential(
    nn.Linear(768 * 2, 1024),
    nn.LeakyReLU(),
    # nn.LayerNorm(1024),
    nn.Linear(1024, 512),
    nn.LeakyReLU(),
    # nn.LayerNorm(512),
    nn.Linear(512, 256),
    nn.LeakyReLU(),
    # nn.LayerNorm(256),  # Layer norm
    nn.Linear(256, 1)
)
logits = self.linear(features.view(features.size()[0], -1))
</code></pre>
<p>I’m using the default initialization method for the linear layers as well. It looks it’s Xavier initialization:</p>
<blockquote>
<p>weight – the learnable weights of the module of shape (out_features, in_features). The values are initialized from U( − √(k), √(k)), where k = (1)/( in_features)<br>
bias – the learnable bias of the module of shape (out_features). If bias is True, the values are initialized from U( − √(k), √(k)) where k = (1)/( in_features)<br>
The nn.Linear layers have <code>bias</code> by default. Is it better to not have biases for these linear layers?<br>
I’m using <code>LeakyReLU</code> activation function for these last few linear layers.</p>
</blockquote>
<p>From your comment,<br>
(1) weight initialization for layers especially the last few layers:<br>
Increasing the scale (typical size or standard deviation) of those random weights</p>
<p>(2) bias?</p>
<p>(3) ReLU:<br>
Avoid ReLU because it’s easy to get “turned off” by becoming 0?</p>
<p>Also, Would using <code>nn.LayerNorm</code> in the last few layers be helpful in solving the constant 0.693 loss problem? I wasn’t using them in my last run.</p>
<hr>
<p><em>Update regarding using LayerNorm</em>: adding LayerNorm after the last few linear layers with leakyReLU actually made it worse - the loss stuck at 0.693 forever:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5bfd1885e6d239e7f5d93f06cc3a81290a9b612.png" data-download-href="https://discuss.pytorch.org/uploads/default/b5bfd1885e6d239e7f5d93f06cc3a81290a9b612" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b5bfd1885e6d239e7f5d93f06cc3a81290a9b612_2_690x385.png" alt="image" data-base62-sha1="pVPogp6OKnxdzjfDVoPl5KyOb9E" width="690" height="385" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b5bfd1885e6d239e7f5d93f06cc3a81290a9b612_2_690x385.png, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5bfd1885e6d239e7f5d93f06cc3a81290a9b612.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5bfd1885e6d239e7f5d93f06cc3a81290a9b612.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">933×521 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Cyber!</p>
<aside class="quote no-group quote-modified" data-username="Cyber_punk" data-post="3" data-topic="183958" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/258eb7/48.png" class="avatar"> Cyber_punk:</div>
<blockquote>
<p>The last few layers are a 3-layer FFNN block:</p>
<pre><code class="lang-auto">self.linear = nn.Sequential(
    ...
    nn.Linear(256, 1)
)
logits = self.linear(features.view(features.size()[0], -1))
</code></pre>
</blockquote>
</aside>
<p>If I understand you correctly, the very last layer of your model is<br>
<code>Linear (256, 1)</code>.</p>
<p>I assume that you then pass <code>logits</code> to <code>BCEWithLogitsLoss()</code></p>
<p>(As an aside, if you’re using <code>BCELoss</code>, you should instead be using<br>
<code>BCEWithLogitsLoss</code>.  However, my main comment about the constant<br>
loss would still be true, but just with your <code>input</code> being <code>0.5</code> instead of<br>
<code>0.0</code>.)</p>
<p>Look at the values you get for <code>logits</code> right at the beginning and early on<br>
in the training.  How close are they to zero?</p>
<p>Also, print out your loss to full precision (e.g., <code>print (loss.item()</code>).  How<br>
close is your loss to being exactly <code>log (2)</code>?</p>
<aside class="quote no-group">
<blockquote>
<p>(1) weight initialization for layers especially the last few layers:<br>
Increasing the scale (typical size or standard deviation) of those random weights</p>
</blockquote>
</aside>
<p>With default weight initialization, your weights should be large enough<br>
that your <code>logits</code> should be enough different than <code>0.0</code> that your loss<br>
should not be almost exactly <code>log (2)</code>.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.0'
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt; lin = torch.nn.Linear (256, 1)   # your last layer
&gt;&gt;&gt; lin.bias   # not that close to zero
Parameter containing:
tensor([0.0269], requires_grad=True)
&gt;&gt;&gt; logits = lin (torch.zeros (256))
&gt;&gt;&gt; logits   # not zero even though the input to lin is zero
tensor([0.0269], grad_fn=&lt;AddBackward0&gt;)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss() (logits, torch.ones (1))   # not that close to log (2)
tensor(0.6798, grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward0&gt;)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss() (torch.zeros (1), torch.ones (1))   # input of zero gives log (2)
tensor(0.6931)
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>(2) bias?</p>
</blockquote>
</aside>
<p>In general, you do want to use <code>bias</code> in your <code>Linear</code> layers (and<br>
convolutions, for that matter).</p>
<aside class="quote no-group">
<blockquote>
<p>(3) ReLU:<br>
Avoid ReLU because it’s easy to get “turned off” by becoming 0?</p>
</blockquote>
</aside>
<p>In general, <code>ReLU</code> is a satisfactory activation.  It can get “stuck” in its<br>
“turned off” regime, but this tends not to be a problem in practice.</p>
<p>In any event, you are using <code>LeakyReLU</code>, which avoids this problem.</p>
<aside class="quote no-group">
<blockquote>
<p>Also, Would using <code>nn.LayerNorm</code> in the last few layers be helpful in solving the constant 0.693 loss problem? I wasn’t using them in my last run.</p>
</blockquote>
</aside>
<p>As far as this particular problem goes, <code>LayerNorm</code> is irrelevant.  As noted<br>
above, your final layer has non-zero <code>bias</code>, so the <code>logits</code> you output won’t<br>
be zero.</p>
<p>To track this down, check the actual <code>logits</code> output by your model.  Check<br>
the actual predicted values you input to your loss function.  Print out the<br>
<code>bias</code> of your last layer (your <code>Linear (256, 1)</code>) before you start training.<br>
Plot the deviation of your loss from <code>log (2)</code> as you train.</p>
<p>Please also post the part of your code where you instantiate your loss<br>
criterion and where you pass your predictions to your loss criterion.</p>
<p>Best.</p>
<p>K. Frank</p>

dhruvbird: <p>@ <a href="https://discuss.pytorch.org/u/Cyber_punk">Cyber_punk</a> Do you have a data imbalance in your labels? For example, maybe you have a lot more samples with no click than with click. That may also affect how quickly the model can start learning. In practice however, I’ve seen that models tends to learn to predict the overrepresented class pretty quickly though, so maybe you have a deep network or a very low learning rate (or both), which can exacerbate things.</p>

Cyber_punk: <p>Hi K. Frank,<br>
Yes, the last layer is <code>nn.Linear(256, 1)</code>, and the <code>logits</code> are passed to <code>BCEWithLogitsLoss()</code>.</p>
<p>I did a few runs and for about (sometimes before and sometimes after) 5000 steps the train batch loss was fluctruating around 0.693.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbe66b22c7271839912c86a8e95a990249dc474c.png" data-download-href="https://discuss.pytorch.org/uploads/default/dbe66b22c7271839912c86a8e95a990249dc474c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe66b22c7271839912c86a8e95a990249dc474c_2_690x162.png" alt="image" data-base62-sha1="vnkeAPd2Hl4yE5DEVmbk4ssRFFy" width="690" height="162" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe66b22c7271839912c86a8e95a990249dc474c_2_690x162.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe66b22c7271839912c86a8e95a990249dc474c_2_1035x243.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbe66b22c7271839912c86a8e95a990249dc474c.png 2x" data-dominant-color="FBFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1342×316 11.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The distribution of logits in batches of 64 samples are indeed very close to 0s during these 5000 steps. It was very “narrow” distribution centered around 0. But after the initial 5000 steps, the distribution of logits started to be scattered around in a relatively larger scale. The following is for 3 runs:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8afe79b84e971404fbc1486373d2df1352b0a9f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/8afe79b84e971404fbc1486373d2df1352b0a9f0" title="logits - distribution"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8afe79b84e971404fbc1486373d2df1352b0a9f0_2_690x176.png" alt="logits - distribution" data-base62-sha1="jPB6kt0YeDzFNAU2cqrSp6iKmPe" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8afe79b84e971404fbc1486373d2df1352b0a9f0_2_690x176.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8afe79b84e971404fbc1486373d2df1352b0a9f0_2_1035x264.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8afe79b84e971404fbc1486373d2df1352b0a9f0_2_1380x352.png 2x" data-dominant-color="F9F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">logits - distribution</span><span class="informations">2141×547 98.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>During these initial 5000 steps, the BCE loss is very close to 0.693 or log(2):</p>
<pre><code class="lang-auto">run_id,key,value,step,timestamp
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6891335844993591,1,1689070859610
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6845006346702576,2,1689070859728
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6913905143737793,3,1689070859858
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.693462610244751,4,1689070859981
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6899896264076233,5,1689070860125
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.697765588760376,6,1689070860260
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.7007498145103455,7,1689070860400
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.7083665132522583,8,1689070860508
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6904076933860779,9,1689070860619
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6960662603378296,10,1689070860750
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.7000266909599304,11,1689070860854
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6916379928588867,12,1689070860984
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.7038472294807434,13,1689070861120
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.695785403251648,14,1689070861232
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6908349990844727,15,1689070861331
...
...
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6938340663909912,5298,1689071557099
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6932273507118225,5299,1689071557211
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6899099946022034,5300,1689071557334
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6955896019935608,5301,1689071557467
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6937833428382874,5302,1689071557567
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6919088363647461,5303,1689071557679
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6920832395553589,5304,1689071557783
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6949018239974976,5305,1689071557898
390b7bed44bc4cb39c5c55cfed6cbcde,train_batch_loss,0.6921497583389282,5306,1689071558024
</code></pre>
<p>The bias of last layer (<code>Linear (256, 1)</code>) seems to be pulled towards 0 in the beginning 5000 steps, regardless of the initialized value being positive or negative:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd6cfe45bb26da2b782fc168593a9bb4374691e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd6cfe45bb26da2b782fc168593a9bb4374691e2" title="bias in last linear layer"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6cfe45bb26da2b782fc168593a9bb4374691e2_2_690x170.png" alt="bias in last linear layer" data-base62-sha1="tjhoHtpzWjGmFfvWJwP91ye6JkC" width="690" height="170" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6cfe45bb26da2b782fc168593a9bb4374691e2_2_690x170.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6cfe45bb26da2b782fc168593a9bb4374691e2_2_1035x255.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd6cfe45bb26da2b782fc168593a9bb4374691e2_2_1380x340.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bias in last linear layer</span><span class="informations">2065×509 59.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve also plotted the histgrams of the <code>weight</code>s of the last and the second to the last linear layers. <strong>It looks the distribution of <code>weight</code> of the linear layers before the last linear layer didn’t change much during the first 5000 steps.</strong> If the distribution of <code>weight</code> does not change much, maybe the speicifc values are not changing much then.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fcdb0118d1e37f3b30f0fbf22a0088fc5bdbc03.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5fcdb0118d1e37f3b30f0fbf22a0088fc5bdbc03" title="logits - histgrams"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fcdb0118d1e37f3b30f0fbf22a0088fc5bdbc03_2_611x500.jpeg" alt="logits - histgrams" data-base62-sha1="dFw2HZAZ5jJWG77E6vvb1fZ5IZ5" width="611" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fcdb0118d1e37f3b30f0fbf22a0088fc5bdbc03_2_611x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fcdb0118d1e37f3b30f0fbf22a0088fc5bdbc03_2_916x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fcdb0118d1e37f3b30f0fbf22a0088fc5bdbc03_2_1222x1000.jpeg 2x" data-dominant-color="E5DBDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">logits - histgrams</span><span class="informations">1920×1570 250 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The following is the part related to loss criterion. I used DDP for running the model, but actually I’m running it on my local PC with a single GPU.</p>
<pre><code class="lang-auto">self.ddp_model = DDP(model, device_ids=[gpu_id], find_unused_parameters=True)  # False
...
self.criterion = nn.BCEWithLogitsLoss(reduction='mean')  # batch loss mean
...

for train_batch in self.train_dataloader:

    train_batch_step += 1
    self.optimizer.zero_grad()
    output = self.ddp_model(**get_features(train_batch, self.gpu_id))
    train_batch_loss = self.criterion(output, get_label(train_batch, self.gpu_id))
    train_batch_loss.backward()
    self.optimizer.step()
    self.lr_scheduler.step()
</code></pre>

Cyber_punk: <p>Hi <a class="mention" href="/u/dhruvbird">@dhruvbird</a> , I think I don’t have data imbalance issue. The raw data only has positive class (clicks). I sampled the negative samples based on popularity with exactly 1:1 proportion.</p>
<p>After the negative sampling, I splitted the train and validation sets based on <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html" rel="noopener nofollow ugc">TimeSeriesSplit</a>. The clicking data in validation set are always after the clicking behaviours in training data.</p>
<p>In training, I did shuffling on the training dataloader (<code>shuffle</code> is <code>True</code> by default for <code>DistributedSampler</code>):</p>
<pre><code class="lang-auto">    # Setting num_workers as 4 * num of GPUs
    train_dataloader = DataLoader(
        dataset_dict["train"], batch_size=batch_size, collate_fn=custom_collate_function, pin_memory=True,
        num_workers=num_workers,
        shuffle=False,
        sampler=DistributedSampler(dataset_dict["train"])
    )
    valid_dataloader = DataLoader(
        dataset_dict["valid"], batch_size=batch_size, collate_fn=custom_collate_function, pin_memory=True,
        num_workers=num_workers,
        shuffle=False,
        sampler=DistributedSampler(dataset_dict["valid"], shuffle=False, drop_last=True)  # 504114 % (64 * 4) == 50 samples
    )
</code></pre>
<p>I think the 0.693 loss issue probably has something to do with the last fully connectly block with 4 linear  layers and leakyReLU layers as mentioned by KFrank. Because when I replaced the whole fully connectly block with an inner product operation, there is no more 0.693 loss issue. But there is another problem, so I posted it separately: <a href="https://discuss.pytorch.org/t/does-my-loss-curve-show-the-model-is-overfitting/184039" class="inline-onebox">Does my loss curve show the model is overfitting?</a></p>

KFrank: <p>Hi Cyber!</p>
<aside class="quote no-group quote-modified" data-username="Cyber_punk" data-post="6" data-topic="183958" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/258eb7/48.png" class="avatar"> Cyber_punk:</div>
<blockquote>
<p>I did a few runs and for about (sometimes before and sometimes after) 5000 steps the train batch loss was fluctruating around 0.693.<br>
…<br>
During these initial 5000 steps, the BCE loss is very close to 0.693 or log(2):<br>
…<br>
The bias of last layer (<code>Linear (256, 1)</code>) seems to be pulled towards 0 in the beginning 5000 steps, regardless of the initialized value being positive or negative:</p>
</blockquote>
</aside>
<p>Okay, this makes good sense.  The initial value and early values of your<br>
<code>loss</code> are not “almost exactly” <code>log (2)</code>, but an amount away that is<br>
consistent with the size of the initial value of <code>Linear (256, 1).bias</code>,<br>
which itself is more or less the value of the initial predicted logit.</p>
<p>Note that binary cross entropy penalizes your model more for guessing<br>
wrong than it rewards it for guessing right, so if your model doesn’t (yet)<br>
have a good prediction, it’s best off predicting a logit of zero (which<br>
translates to a probability of one half – i.e., don’t know, so flip a coin).<br>
So – until the rest of your model starts learning how to make a prediction<br>
with some value to it – it makes sense for your final bias to get trained<br>
down to zero.</p>
<p>As to why it takes seemingly so long to start training effectively, I don’t<br>
really know.</p>
<p>If it were me, I would try increasing the initial learning rate until it starts<br>
getting out of the “stuck” phase more expeditiously (or until the initial<br>
training starts to become unstable and diverges).</p>
<p>It’s just me, but when training isn’t “behaving,” I generally like to start<br>
with plain-vanilla <code>SGD</code>, using as large a learning rate as I can without<br>
things diverging.  Then I’ll back off on the learning rate a little, and start<br>
turning on <code>SGD</code>’s <code>momentum</code>.  (Sometimes you have to train for a<br>
shortish warm-up period with a smaller learning so the parameters get<br>
non-wacky before turning up the learning rate or momentum.)</p>
<p>Note, even if <code>Adam</code> ends up training more effectively than <code>SGD</code>, there<br>
is nothing wrong with training with <code>SGD</code> to get out of the “stuck” phase<br>
and then switching over to <code>Adam</code>.</p>
<p>Or, if the “stuck” phase doesn’t really matter in terms of computer time,<br>
you can just stick with your current training scheme and view those first<br>
5000 training iterations as the price of admission.</p>
<p>Best.</p>
<p>K. Frank</p>

kappablanca: <p>I know it is an old post, but for anyone stumbling on this. What is happening here is that the weights of the final linear layer are driven to zero in the first few training steps. This is due to the fact that the transformer was not yet able to pick up on the pattern of the data and your final layer is minimizing the BCE loss by always predicting the mean of your data.</p>
<p>I had this problem myself and I fixed it by adding a loss term preventing the standard deviation to go to zero. I described this here: <a href="https://open.substack.com/pub/yannikkeller/p/solving-vanishing-gradients-from?r=3avwpj&amp;utm_campaign=post&amp;utm_medium=web" rel="noopener nofollow ugc">https://open.substack.com/pub/yannikkeller/p/solving-vanishing-gradients-from?r=3avwpj&amp;utm_campaign=post&amp;utm_medium=web</a></p>