Loss not Decreasing while training UNET

username: Jaskeerat
cooked: <p>I have built Unet for image segmentation from scratch and the dataset i am training it on is Cityscapes that is available on their original website. Hyper parameters are:<br>
Batch_size = 16<br>
Learning_rate = 1e-2<br>
Optimizer = Adam<br>
Loss_functions = DiceLoss + CrossEntropyLoss<br>
Original Image and mask were of 1024 x 2048 i transformed them to 128 x 256.</p>
<p>But the main problem is that the model have decreasing loss for first few epochs through dataset but after 2 epochs or in the 2nd epochs it keeps on fluctuating in the range 0.8-1.0 and from that never decreases. I have use torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer , mode=‘min’ , factor=0.1, patience=2). I am thinking that the issue might be with the size of image i am currently using the transform function for image and mask are:</p>
<p>transform_fn = transform.Compose([<br>
transform.Resize((128, 256)),<br>
transform.ToTensor()<br>
# transform.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])<br>
])</p>
<p>target_transform_fn = transform.Compose([<br>
transform.Resize((128, 256), interpolation=Image.NEAREST),<br>
transform.PILToTensor()<br>
])</p>
<p>It will be helpful if anyone can tell me how can i overcome this problem.</p>

username: KFrank
cooked: <p>Hi Jaskeerat!</p>
<aside class="quote no-group quote-modified" data-username="Jaskeerat" data-post="1" data-topic="219167" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaskeerat/48/76675_2.png" class="avatar"> Jaskeerat:</div>
<blockquote>
<p>Batch_size = 16<br>
Learning_rate = 1e-2<br>
Optimizer = Adam<br>
Loss_functions = DiceLoss + CrossEntropyLoss<br>
…<br>
the model have decreasing loss for first few epochs through dataset but after 2 epochs or in the 2nd epochs it keeps on fluctuating in the range 0.8-1.0 and from that never decreases.<br>
…<br>
It will be helpful if anyone can tell me how can i overcome this problem.</p>
</blockquote>
</aside>
<p>Start by seeing if you can overfit.  Try training on a single batch or a small number of<br>
batches (or even on a single image).  You should be able to train to the point that you<br>
get nearly perfect (but useless) predictions.  If you can’t overfit, you have some bug in<br>
your model or training.</p>
<p>You haven’t said how many optimizer steps / epochs you have run.  You ought to be able<br>
to overfit in a reasonable amount of time, but that may still be many – perhaps thousands<br>
or more – of epochs.</p>
<p>If you can overfit, start increasing the size of your training set.  At some point you should<br>
transition from overfitting to “real” training.  But be prepared to train for a potentially large<br>
number of epochs.</p>
<p>I would also recommend that you start with plain-vanilla <code>SGD</code> as your optimizer and with<br>
pure <code>CrossEntropyLoss</code> (no Dice loss) as your loss criterion.  These are simpler and easier<br>
to reason about.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Jaskeerat
cooked: <p>Hi KFrank,<br>
The optimizer i’m using is Adam on learning rate of 0.001 and right now i’m training at 1 Epoch and then analyzing the loss i will adjust learning rate manually. The trend of loss right now is :<br>
Loss at 1th batch is 2.2865967750549316<br>
Loss at 11th batch is 1.6623344421386719<br>
Loss at 21th batch is 1.4776408672332764<br>
Loss at 31th batch is 1.3324888944625854<br>
Loss at 41th batch is 1.3063474893569946<br>
Loss at 51th batch is 1.3136686086654663<br>
Loss at 61th batch is 1.1328212022781372<br>
Loss at 71th batch is 1.2527915239334106<br>
Loss at 81th batch is 1.24680757522583<br>
Loss at 91th batch is 1.1866313219070435<br>
Loss at 101th batch is 1.0283044576644897<br>
Loss at 111th batch is 1.082072138786316<br>
Loss at 121th batch is 1.1612353324890137<br>
Loss at 131th batch is 1.1107628345489502<br>
Loss at 141th batch is 1.1223273277282715<br>
Loss at 151th batch is 1.1529223918914795<br>
Loss at 161th batch is 1.0620512962341309<br>
Loss at 171th batch is 1.0563969612121582<br>
Loss at 181th batch is 1.0337893962860107<br>
Loss at 191th batch is 1.149621844291687<br>
Loss at 201th batch is 1.0832343101501465<br>
Loss at 211th batch is 1.0245585441589355<br>
Loss at 221th batch is 1.0651346445083618<br>
Loss at 231th batch is 0.8998237252235413<br>
Loss at 241th batch is 1.0268874168395996</p>

username: ajayrkumar
cooked: <p>Adding on to the debugging steps suggested by <a class="mention" href="/u/kfrank">@KFrank</a>,<br>
I would also suggest you to re-check your training loop. This is a common issue encountered when the <code>lr.step()</code> is called after completion of each batch, instead of each epoch. If that was the case, before the model could learn something useful, the learning rate is reduced way too much.</p>