What are possibilities that output of linear layer before sigmoid layer return near opposite value?

luan1412167: <p>Hi you guys, pardon me I’m not good at training model. I’m training the model with architecture as following</p>
<pre data-code-wrap="out"><code class="lang-out">        out = self.conv2_dw(out)
        out = self.conv_23(out)
        out = self.conv_3(out)
        out = self.conv_34(out)
        out = self.conv_4(out)
        out = self.conv_45(out)
        out = self.conv_5(out)
        out = self.conv_6_sep(out)
        out = self.conv_6_dw(out)
        out = self.conv_6_flatten(out)
        if self.embedding_size != 512:
            out = self.linear(out)
        out = self.bn(out)
        out = self.drop(out)
        out = self.prob(out)
        print("self.prob", out)
        return out
</code></pre>
<p>Given self.prob = Linear layer, model gradually converged to good result.<br>
I stopped at 25 epochs due to keep training continue I fear that it tend to overfit.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e18d70fa00f97412a1085cd50c061bad98911490.png" data-download-href="https://discuss.pytorch.org/uploads/default/e18d70fa00f97412a1085cd50c061bad98911490" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e18d70fa00f97412a1085cd50c061bad98911490_2_690x476.png" alt="image" data-base62-sha1="wbksYXgnhqVEJKcL7yNK0iWad1e" width="690" height="476" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e18d70fa00f97412a1085cd50c061bad98911490_2_690x476.png, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e18d70fa00f97412a1085cd50c061bad98911490.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e18d70fa00f97412a1085cd50c061bad98911490.png 2x" data-dominant-color="5C6161"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">976×674 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
why when I run test, output at self.prob is near opposite value</p>
<pre><code class="lang-auto">self.prob tensor([[ 4.2354, -4.1672]], device='cuda:0')
sigmoid =&gt;  [[0.9857328  0.01525886]]
</code></pre>
<p>Any answer is highly appreciated. Many thanks</p>

ptrblck: <p>I don’t completely understand the question.</p>
<aside class="quote no-group" data-username="luan1412167" data-post="1" data-topic="95646">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luan1412167/48/28704_2.png" class="avatar"> luan1412167:</div>
<blockquote>
<p>why when I run test, output at self.prob is near opposite value</p>
</blockquote>
</aside>
<p>The output in the test case is opposite to which value?</p>

luan1412167: <p>Thanks for your reply.<br>
It means I’m training a model to classify image is fake or real.<br>
My last layer of model is nn.Linear before I fit it to softmax.<br>
I’m training and looking like the model converged well due to loss, acc decrease gradually.<br>
In test pharse, I print output after linear function as above image.<br>
The output is always ~ U[-k, k] distribution (like above ‘’’self.prob tensor([[ 4.2354, -4.1672]] ‘’’)<br>
I’m new in training model, so If my question is tough to understand, pardon me for that. I’m always highly appreciated your help.</p>

ptrblck: <p>Thanks for the clarification.<br>
If I understand the question correctly, you are wondering why the output values of the last linear take values such as <code>[4.2354, -4.1672]</code>?<br>
The last linear layer outputs logits, which are raw prediction values and are not bounded to a specific range.<br>
The lower the value, the lower the probability and vice versa.<br>
In your case your model is pretty confident that the current sample belongs to class0. You can see the probabilities using <code>softmax</code> (which you’ve already done) and see that the class0 probability would correspond to ~98.6%.</p>

luan1412167: <p>Thanks, I have understood this issue. I am also wondering that if the output ain’t force to U[-k, k] distribution, whether it is better or not ?</p>

ptrblck: <p>The output is not forced to a specific distribution. I’m not sure which use case should be better or worse. Could you explain what you mean?</p>

luan1412167: <p>Look like nn.Linear is initialized by U[-k, k], so its output is always forced to U[-k, k]. I don’t know it is good or not.<br>
However seem as it’s how initialize weights good for model. I only ask about your experiences about how to initialize weights for linear layers.</p>

ptrblck: <p>The initialization isn’t creating a hard bound on the output values, as the range of the input values also determines the output. However, of course it has a certain influence on the output.</p>
<p>I don’t know what the currently most popular weight init method for linear layers is and would try to stick to good performing models and their init methods. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

luan1412167: <p>thanks for your sharing and supports for pytorch community. Keep it up <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>