Outputs from a simple DNN are always the same whatever the input is

username: Raine
cooked: <p>I have built a DNN with only one hidden layer, the following are the parameters:<br>
input_size = 100<br>
hidden_size = 20<br>
output_size = 2<br>
def <strong>init</strong>():<br>
self.linear1 = nn.Linear()<br>
self.linear2 = nn.Linear()<br>
def forward():<br>
x1 = F.leaky_relu()<br>
return F.leaky_relu()<br>
<span class="hashtag">#unimportant</span> codes omitted<br>
loss_function = nn.CrossEntropyLoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.02)</p>
<p>normalized word vectors of size 100 from authoritative github are used as input<br>
My purpose is to identify whether a word is an event. For example, ‘dought’ is an event but ‘dog’ is not.<br>
After training, the 2-dimensional output tensors are almost the same (say,(-0.8,-1.20) and (-0.8,-1.21), (-0.2,-1.01) and (-0.2,-1.02)) even if the activation function and loss function are changed.<br>
Could someone tell me the reason? I tried my best but failed to solve it.</p>

username: ptrblck
cooked: <p>Could you check the <code>weight</code> and <code>bias</code> in both layers?<br>
Sometimes, e.g. when the learning rate is too high, the model just learns the “mean prediction”, i.e. the <code>bias</code> is responsible for most of the prediction, while the <code>weight</code>s and <code>input</code> became more or less useless.<br>
For example when I was playing with a facial keypoint dataset, some models just predicted the “mean position” of the keypoints, regardless of the input image.</p>

username: jpeg729
cooked: <p>This code shouldn’t even run because F.leaky_relu() needs an input.</p>

username: ptrblck
cooked: <p>I thought that’s what “unimportant code omitted” meant <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>

username: Raine
cooked: <p>Thank you! I set the bias to 0 and the problem is solved!!<br>
unfortunately this method did not work~~~~lol</p>

username: yuqli
cooked: <p>Could you please kindly elaborate more on the bias and “mean prediction” part? I’ve seen this explanation multiple times on the Internet but cannot get it. When the learning rate is too high, my understanding is that the model wouldn’t converge? Why would that results in bias responsible for most of the prediction? Thanks!</p>

username: ptrblck
cooked: <p>I’m not sure if there is an underlying mathematical explanation for this effect.<br>
In the past I experienced that basically the bias in the last layer took the mean values of the regression task, so regardless of the input, I always got the average of my targets.<br>
Could be an edge case and I don’t have a proper explanation for it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: yuqli
cooked: <p>That’s probably what happened to my model too, except I did not check that carefully for bias values but I do notice bias getting dominant in terms of scale with regard to weights. The outputs are also indeed the mean.</p>
<p>I solved this by 1) normalizing the input by demean and divide by std and 2) used a smaller learning rate.</p>
<p>Thanks for the prompt reply!</p>

username: Mahsa_Afsharizade
cooked: <p>This problem may be due to the “batch normalization”. When you are evaluating your model, you should disable batch normalization. Use “model.eval()” when you want to evaluate the model (so batch normalization will be disabled) and use “model.train()” again when you want train the model.</p>

username: Hong_Cheng
cooked: <p>This means that if we do the <strong>validation</strong> during <strong>training</strong> with <code>model.eval()</code>, in the <strong>main training loop</strong>, <strong>before</strong> we use<code> optimizer.step()</code>, we should add <code>model.train()</code> ?</p>

username: ptrblck
cooked: <p>You should call <code>model.train()</code> before the forward pass in your training loop.<br>
If you call it before <code>optimizer.step()</code>, the forward pass will have been already executed in <code>eval</code> mode.</p>

username: db111
cooked: <p>Do you just set the bias=false while initialisation or is there any other way to set the bias equal to 0 for the linear layer?</p>

username: ptrblck
cooked: <p>If you set <code>bias=False</code> during the initialization of the layer, the internal <code>.bias</code> parameter will be set to <code>None</code> and will thus not be available, which would be different from setting the value of the <code>bias</code> to zero.<br>
The latter case can be achieved by manipulating this parameter e.g. via:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.linear_layer.bias.fill_(0.)
</code></pre>

username: db111
cooked: <p>Thanks!! I’ll try it out.</p>