Regression Net Architecture Problem

string111: <p>Hey, it is me again <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>
<p>so I am trying to predict two seperate coordinates on an input image, I use CNN layers and a fully connected layer which projects onto 4 output nodes with a sigmoid activation at the end (to get the values between 0 and 1).</p>
<p>My output is tensor of size <code>(batch_size, 4)</code>, where <code>4 --&gt; [x1, y1, x2, y1]</code> are the coordinates of the predicted points.</p>
<p>So I am training and my loss gets stuck already after epoch 1, i increased the learning rate etc. My predicted output always seems to be in close range to <code>[0.5, 0.5, 0.5, 0.5]</code> and I do not get why. The labels are also right.<br>
Is sigmoid the right function for predicting these values?</p>
<p>Btw my loss is <code>MSELoss</code></p>
<p>Hope you can help me!</p>
<p>Raph <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I would remove the sigmoid to be honest, since your model might have a hard time predicting values close to 0 and 1.<br>
IIRC the mse loss has also some disadvantages using a sigmoid or softmax, since the weight updates can stall once your model begins to learn.</p>

string111: <p>Alright! Thanks. I need the outputs to be between 0 and 1 though, because my labeled coordinates are also normalized <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"> <code>x1, y1 = real_x1 / width_image, real_y1 / height_image</code></p>

ptrblck: <p>IIRC you are trying to track mice in a lab setting. Are they often directly on the border?<br>
I would still try to remove the sigmoid from the last layer and try to see, if the predictions get any better.</p>

string111: <p>Yep, the thing is I tracked the mice with the same Model, now I am trying to find their snout and tail on a cropped image where they are usually placed somewhere around the center. All I did was to change the output to 4 instead of 2 nodes. Also my data size (due to manually labeling) is only 2000 images compared to 120.000 before.<br>
Do I need to replace the sigmoid with some other kind of activation?</p>
<p>EDIT:<br>
So I just got rid of the Sigmoid and this happened, does not look really promising <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"> learning rate is 0.01</p>
<pre><code class="lang-auto">Epoch 1/1000
----------
train Loss: 0.007419419793224839, Acc: 0.0
val Loss: 0.0018860326872931587, Acc: 0.0

Epoch 2/1000
----------
train Loss: 0.0018890846087030633, Acc: 0.0
val Loss: 0.0018535365247064166, Acc: 0.0

Epoch 3/1000
----------
train Loss: 0.0018744729794364758, Acc: 0.0
val Loss: 0.0018500339653756883, Acc: 0.0

Epoch 4/1000
----------
train Loss: 0.0018520522586725377, Acc: 0.0
val Loss: 0.0018364218788014517, Acc: 0.0
</code></pre>
<p>EDIT2:<br>
Using <code>Adadelta()</code> optimizer I get weird behaviours like this:</p>
<pre><code class="lang-auto">[[ 1.0000e+00,  4.1560e-39],
 [ 4.1560e-39,  1.0000e+00]] # &lt;-- in one batch all outputs look similiar to this

[[ 0.4993,  0.5002],
 [ 0.4964,  0.5007]] # &lt;-- in next batch similar to this

</code></pre>
<p>the outputs seem to jump from batch to batch between  around <code>[[1, 0], [0, 1]]</code> to around <code>[[0.5, 0.5], [0.5, 0.5]]</code><br>
This I find very weird…and I cannot explain why, maybe you have an idea. I tried messing around with the batch size which gave no improvements at all.</p>

ptrblck: <p>It looks like the predictions jump from <code>[0, 0]</code> to <code>[0.5, 0.5]</code>.<br>
Are you using the same model but with another last layer?<br>
Did your previous model learn the correct regression using a sigmoid at the end?<br>
Did you think about using your pre-trained model and just add an additional “snout/tail” layer?</p>

string111: <p>Yea, the old model worked with sigmoid at the end. Like I said, everything is similiar but instead of projecting onto 2 neurons in the fully connected last layer, i project on 4 neurons in the new model <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:">.<br>
I am using the old model to track the mouse, then I crop the image (where the mouse is around the centre now), I use this cropped image as input for the model I am building atm. Gonna try with the loaded weights, did not think of that thx.</p>

ptrblck: <p>That would be one idea. Another would be to use just one model with two different output layers.<br>
One for the global mouse position, the other one for the snout and tail.<br>
Let me know, how your experiments work out!</p>

string111: <p>Gonna try the two output layer method. Thanks! I won’t be giving up too soon <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>

string111: <p>I ended up using ResNet34 for the Task with the same Hyperparams, worked like a charm, implemented your advises. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Undertale: <p>hi i’m new for the regression task<br>
could you please tell that how to use the resnet34 to do the regression work, with the input is an image and the output is the coordinate<br>
thanks for your apply</p>