My code needs a very specific random seed not to get stuck at the same loss

username: abdeljalil
cooked: <p>I made a very simple 3-layered fully-connected network for binary classification (using NASA C-MAPSS dataset to classify healthy and faulty turbofan engines). the input is vector of length 26 and the output is a sigmoid activation. The task is pretty easy (a basic logistic regression model gives me 100% test accuracy), I’m porting this code from Keras (where everything worked as expected) but when executing PyTorch code the loss doesn’t change. Executing the exact same code different times, most of the times I get that non-changing loss, sometimes it works and converges within a couple of epochs to 100% test accuracy as expected. I tried using Keras settings (learning rate, Adam parameters, weights initialization) yet the problem persisted. After setting the random seed (without the default weights initialization this time) I get the same result with each run (obviously!), even that the loss doesn’t change with each run but it changes from run to run. I had to try different seeds until I found a seed that actually works and the model works as expected and runs correctly every time (I set the seed to 527, other values may have worked but that’s the only one that I found).<br>
What can be the cause of this behavior?</p>
<p>Here is my code and training process (data processing code is a bit long and I’m sure it’s not the problem):</p>
<pre><code class="lang-auto">class CMAPSSBinaryClassifier(nn.Module):
    def __init__(self):
        super(CMAPSSBinaryClassifier, self).__init__()
        self.fc1 = nn.Linear(26, 16)
        self.fc2 = nn.Linear(16, 4)
        self.fc3 = nn.Linear(4, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        output = torch.sigmoid(self.fc3(x))
        return output

data_path = "/home/abdeljalil/Workspace/Datasets/CMAPSS/"
data_FD01 = CMAPSSDataset(data_path, fd_number=1)
model_FD01 = CMAPSSBinaryClassifier()

#tried Xavier weights initialization scheme
#model_FD01.apply(init_weights)

loader_train, loader_test = data_FD01.construct_binary_classification_data(good_faulty_threshould=30, batch_size=64)
epochs = 100
#tried Adam with a wide range of learning rates
optimizer = optim.Adam(model_FD01.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-7)
#tried SGD with different learning rates too
#optimizer = optim.SGD(model_FD01.parameters(), lr=0.001, momentum=0.9)

model_FD01.train()
criterion = nn.BCELoss()

for epoch in range(epochs+1):
    correct = 0
    for batch_id, (data, target) in enumerate(loader_train):
        optimizer.zero_grad()
        output = model_FD01(data)
        output = output.view_as(target)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        output = torch.round(output)
        correct += output.eq(target).sum().item()

    print('Train epoch: {}\t Accuracy ({:.0f}%)\tLoss: {:.6f}'.format(epoch, 100. * correct/len(loader_train.dataset), loss.item()))

</code></pre>
<p>Here is a sample output (first 6 epochs) when the model doesn’t converge (like 90% of the times):</p>
<pre><code class="lang-auto">Train epoch: 0   Accuracy (50%) Loss: 13.815515
Train epoch: 1   Accuracy (50%) Loss: 13.815515
Train epoch: 2   Accuracy (50%) Loss: 13.815515
Train epoch: 3   Accuracy (50%) Loss: 13.815515
Train epoch: 4   Accuracy (50%) Loss: 13.815515
Train epoch: 5   Accuracy (50%) Loss: 13.815515
</code></pre>
<p>Sometimes the loss changes but stuck at around 0.69:</p>
<pre><code class="lang-auto">Train epoch: 0   Accuracy (50%) Loss: 0.704716
Train epoch: 1   Accuracy (50%) Loss: 0.701211
Train epoch: 2   Accuracy (50%) Loss: 0.698781
Train epoch: 3   Accuracy (50%) Loss: 0.697099
Train epoch: 4   Accuracy (50%) Loss: 0.695932
Train epoch: 5   Accuracy (50%) Loss: 0.695122
</code></pre>
<p>And the rare few times it actually works:</p>
<pre><code class="lang-auto">Train epoch: 0   Accuracy (56%)         Loss: 0.516986
Train epoch: 1   Accuracy (90%)         Loss: 0.318052
Train epoch: 2   Accuracy (100%)        Loss: 0.203251
Train epoch: 3   Accuracy (100%)        Loss: 0.136395
Train epoch: 4   Accuracy (100%)        Loss: 0.096920
Train epoch: 5   Accuracy (100%)        Loss: 0.072752
</code></pre>

username: KFrank
cooked: <p>Hello Abdeljalil!</p>
<aside class="quote no-group quote-modified" data-username="abdeljalil" data-post="1" data-topic="87994" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdeljalil/48/26320_2.png" class="avatar"> abdeljalil:</div>
<blockquote>
<p>Executing the exact same code different times, most of the times I get that non-changing loss, sometimes it works and converges within a couple of epochs to 100% test accuracy as expected.<br>
…<br>
What can be the cause of this behavior?</p>
</blockquote>
</aside>
<p>This particular issue does not sound like a bug in your code,<br>
although I suppose that would be possible.</p>
<p>More likely is that your training procedure is not robust for this<br>
particular network / dataset.</p>
<p>(To clarify, is you loss <em>literally</em> unchanging, or is it just “stuck,” and<br>
not making progress?  Have you tried letting your training run for a<br>
<em>long</em> time to see if it can get off of its plateau?)</p>
<p>First, I would switch to <code>SGD</code> instead of using the <code>Adam</code> optimizer.<br>
(I don’t have much experience with <code>Adam</code>, but the lore seems to<br>
be that when it works, it can converge faster, but that it gets stuck<br>
more easily.)</p>
<p>Second, I would try different learning rates, both significantly larger<br>
and significantly smaller.  Depending on the nature of your plateau<br>
or “getting stuck,” a larger or smaller learning rate can help address<br>
the issue.  If this works, but is suboptimal (e.g., too slow), you could<br>
then try using a learning-rate scheduler.</p>
<p>As for the difference between pytorch and what I assume is<br>
keras / tensorflow, speaking from memory, scale their initial<br>
random weights differently.</p>
<p>However, your training procedure should be robust with respect to<br>
the general scale of you initial weights (and certainly your random<br>
number seed).  So you could try changing the scale of your initial<br>
weights, but even if that were to work, I would say it’s the wrong fix.</p>
<p>In short, I would suggest using <code>SGD</code> and trying various learning<br>
rates.  If something appears to work, make sure it works with most<br>
(all?) initial random number seeds.  If you get something working,<br>
you can then back up and try to fine tune it to get it to run faster.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: abdeljalil
cooked: <p>I just tried both SGD and Adam with a learnign rates from 0.000001 to 0.1 and the problem persisted. I was going to post output in this reply but preferred editing the original post. Please check it again.</p>

username: KFrank
cooked: <p>Hello Abdeljalil!</p>
<aside class="quote no-group" data-username="abdeljalil" data-post="3" data-topic="87994" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdeljalil/48/26320_2.png" class="avatar"> abdeljalil:</div>
<blockquote>
<p>I just tried both SGD and Adam with a learnign rates from 0.000001 to 0.1</p>
</blockquote>
</aside>
<p>It’s probably not the issue, but please also try <code>SGD</code> with no<br>
momentum (<code>momentum = 0.0</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>but preferred editing the original post.</p>
</blockquote>
</aside>
<p>Some comments based on your edited post:</p>
<p>First, get rid of the <code>sigmoid()</code> at the end of your model.  Thus:</p>
<pre data-code-wrap="python"><code class="lang-python">        output = self.fc3(x)
        return output
</code></pre>
<p>And use <code>BCEWithLogitsLoss</code> (rather than <code>BCELoss</code>) as your<br>
<code>criterion</code>.  This is in any event best practice, and it’s possible<br>
that in your case you are getting stuck where <code>sigmoid()</code> saturates<br>
to either <code>0.0</code> or <code>1.0</code>.</p>
<p>(Note, when you convert to <code>BCEWithLogitsLoss</code>, when calculating<br>
your accuracy, you should threshold the output of your model at<br>
<code>0.0</code> rather than <code>0.5</code>.  (Rounding a number that is between 0 and<br>
1 is the same as thresholding it at <code>0.5</code> in the language I’m using.)</p>
<p>Because your accuracy gets “'stuck” at <code>50%</code>, I speculate that<br>
your dataset is exactly balanced – that is, it contains the same<br>
number of “healthy” and “faulty” samples – and that your model<br>
(when stuck) always predicts (close to) 0 or (close to) 1.</p>
<p>What do representative predictions from your model look like?<br>
Is your data in fact balanced?  Do you shuffle your data?  Why<br>
do you get <em>exactly</em> (to two decimal digits) <code>50%</code> accuracy?<br>
(Notice that your accuracy doesn’t move away from <code>50%</code> until<br>
your loss has fallen from about 14 to about 0.5!)</p>
<p>To clarify, what happens when you run your keras / tensorflow<br>
model multiple times with different random seeds.  Is it robust?</p>
<p>As an aside:  Your <code>output = output.view_as(target)</code><br>
looks fishy.  I’m guessing that it’s unnecessary (that is, that<br>
your <code>output</code> and <code>target</code> already have the same shape).</p>
<p>What are the shapes of <code>data</code>, <code>output</code> (before <code>view_as()</code>),<br>
and <code>target</code>?</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: abdeljalil
cooked: <p>I just got rid of <code>torch.sigmoid()</code> and used <code>BCEWithLogitsLoss</code> instead of <code>BCELoss</code> and it just worked fine as expected, there is no stuck loss problem anymore and the model converges within a couple of epochs. Thank you so much for the advice.</p>
<p>From PyTorch documentation:</p>
<blockquote>
<p>This loss combines a <code>Sigmoid</code> layer and the <code>BCELoss</code> in one single class. This version is more numerically stable than using a plain <code>Sigmoid</code> followed by a <code>BCELoss</code> as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.</p>
</blockquote>
<p>But I’m not sure why this really solves my problem.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="87994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Note, when you convert to <code>BCEWithLogitsLoss</code> , when calculating<br>
your accuracy, you should threshold the output of your model at<br>
<code>0.0</code> rather than <code>0.5</code> .</p>
</blockquote>
</aside>
<p>I really don’t know what you mean here exactly, but the model output now is just some integers, example output:</p>
<pre><code class="lang-auto">[ 7., -3., -5., -5., -5.,  4., -5., -5., -5.,  4., -5., 13., -5., -5., -5.,
-4.,  6.,  4.,  3.,  6., -3.,  3., -5.,  3., 11., -4., -4., 11., -5., -6.,
-5.,  4.,  6.,  7.,  5.,  7., -5.,  4.,  9., -5.,  8.,  4., 5.,  9., -4., 
1., 10.,  4., 14., -4., -4., -4.,  5., -5.,  6., 15., 3., -6., -5., -5.,
-5., -5., -4.,  3.]
</code></pre>
<p>So I just did the following:</p>
<pre><code class="lang-auto">output = torch.round(torch.sigmoid(output))
</code></pre>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="87994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Because your accuracy gets “'stuck” at <code>50%</code> , I speculate that<br>
your dataset is exactly balanced</p>
</blockquote>
</aside>
<p>Yes that’s true, it’s because I constructed the dataset myself (extracted the first and last 30 samples from each turbofan engine simulation in C-MAPSS dataset respectively as negative and positive samples) so my dataset is exactly balanced but I never thought that would be the cause of the problem.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="87994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>To clarify, what happens when you run your keras / tensorflow<br>
model multiple times with different random seeds. Is it robust?</p>
</blockquote>
</aside>
<p>Yes running the same code in Keras always yields almost the same results, pretty robust.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="87994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>As an aside: Your <code>output = output.view_as(target)</code><br>
looks fishy. I’m guessing that it’s unnecessary (that is, that<br>
your <code>output</code> and <code>target</code> already have the same shape).</p>
</blockquote>
</aside>
<p>It’s just the way my code generates data, the model output is <code>[64, 1]</code> while the target shape is <code>[64]</code> and yes I should modify the other code instead of doing this trick.</p>

username: KFrank
cooked: <p>Hello Abdeljalil!</p>
<p>Some comments, in line, below:</p>
<aside class="quote no-group quote-modified" data-username="abdeljalil" data-post="5" data-topic="87994" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdeljalil/48/26320_2.png" class="avatar"> abdeljalil:</div>
<blockquote>
<p>I just got rid of <code>torch.sigmoid()</code> and used <code>BCEWithLogitsLoss</code> instead of <code>BCELoss</code> and it just worked fine as expected, there is no stuck loss problem anymore and the model converges within a couple of epochs.<br>
…<br>
But I’m not sure why this really solves my problem.</p>
</blockquote>
</aside>
<p><em>Mathematically,</em> <code>sigmoid()</code> maps <code>[-inf, inf]</code> to <code>[0.0, 1.0]</code>.<br>
But <code>sigmoid()</code> has <code>exp()</code> in it, which “amplifies” numbers<br>
dramatically, so <em>numerically</em> the range for which <code>sigmoid()</code><br>
gives useful results is rather small.</p>
<p>Specifically, <code>sigmoid (x)</code> saturates at <code>1.0</code> already for <code>x = 17</code>,<br>
and saturates at <code>0.0</code> for <code>x = -90</code> (for <code>FloatTensor</code>s). These<br>
are not extreme values and can easily be returned by a <code>Linear</code><br>
layer.</p>
<p>And once the <code>sigmoid()</code> saturates, its gradient becomes zero,<br>
so the optimizer won’t know which direction to go to get you<br>
“unstuck.”</p>
<aside class="quote no-group">
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="87994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Note, when you convert to <code>BCEWithLogitsLoss</code> , when calculating<br>
your accuracy, you should threshold the output of your model at<br>
<code>0.0</code> rather than <code>0.5</code> .</p>
</blockquote>
</aside>
<p>I really don’t know what you mean here exactly, but the model output now is just some integers, example output:</p>
<pre><code class="lang-auto">[ 7., -3., -5., -5., -5.,  4., -5., -5., -5.,  4., -5., 13., -5., -5., -5.,
-4.,  6.,  4.,  3.,  6., -3.,  3., -5.,  3., 11., -4., -4., 11., -5., -6.,
-5.,  4.,  6.,  7.,  5.,  7., -5.,  4.,  9., -5.,  8.,  4., 5.,  9., -4., 
1., 10.,  4., 14., -4., -4., -4.,  5., -5.,  6., 15., 3., -6., -5., -5.,
-5., -5., -4.,  3.]
</code></pre>
</blockquote>
</aside>
<p>These integer values for <code>output</code> surely occur only after you have<br>
calculated <code>output = torch.round (output)</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>So I just did the following:</p>
<pre><code class="lang-auto">output = torch.round(torch.sigmoid(output))
</code></pre>
</blockquote>
</aside>
<p>You can do this and it will give you what you want.  But, to me,<br>
it’s a little more logically straightforward to compare (“threshold”)<br>
your pre-sigmoid outputs with 0.</p>
<p>The rationale is:</p>
<p>An output of a linear layer (a raw-score <em>logit</em>) of <code>0.0</code> is mapped by<br>
<code>sigmoid()</code> to a <em>probability</em>, <code>P</code>, of <code>0.5</code>.  To convert a probability to<br>
a discrete yes-no prediction,  one would typically say <code>P &gt; 0.5</code><br>
means “yes” (and <code>P &lt; 0.5</code> means “no”).</p>
<p>So instead of using <code>sigmoid()</code> to convert to probabilities, you<br>
can just threshold the logits directly: <code>logit &gt; 0.0</code> means “yes”<br>
(and <code>logit &lt; 0.0</code> means “no”).</p>
<aside class="quote no-group">
<blockquote>
<p>Yes running the same code in Keras always yields almost the same results, pretty robust.</p>
</blockquote>
</aside>
<p>Not that it really matters now that you have your model working,<br>
but are you possibly using double-precision in your keras code?</p>
<p>Do you have an explicit <code>sigmoid()</code> in your keras code?  If so,<br>
for what values does the keras <code>sigmoid()</code> saturate?</p>
<p>Best.</p>
<p>K. Frank</p>