Model initialized from scratch is not training at all

bharat0to: <p>Hi,<br>
I am trying to train a cnn to classify dogs, cats. But the network always outputs loss as 0.6931 and accuracy 50%. I have seen  the same issue in this tutorial <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a><br>
I expect the val accuracy to increase slowly in case of scratch_model but why it is not at all increasing?</p>

ptrblck: <p>Could you post your training code so that we can have a look and debug it?<br>
Sometimes some hyperparameters are off, e.g. a too high learning rate was used.</p>

bharat0to: <p>Hi ptrblck<br>
here is the code <a href="https://www.kaggle.com/bharat0/script-dogs-vs-cats/log?scriptVersionId=7399117" rel="nofollow noopener">https://www.kaggle.com/bharat0/script-dogs-vs-cats/log?scriptVersionId=7399117</a></p>
<p>Thank you</p>

bharat0to: <p>Even pretrained is showing same results <a href="https://www.kaggle.com/bharat0/dog-vs-cat-pretrained-resnet?scriptVersionId=7404637" rel="nofollow noopener">https://www.kaggle.com/bharat0/dog-vs-cat-pretrained-resnet?scriptVersionId=7404637</a><br>
I think I made mistake somewhere, no sure where.</p>

ptrblck: <p>Thanks for the code! The second link seems to be dead.<br>
Did the model work in any configuration before?<br>
Your transformations seem to be a bit odd, as you specify some probabilities as 1 for random transformations, making them practically deterministic.</p>
<p>I would suggest to scale down some transformations and the model, and to try fitting a baseline model to the data first.</p>

bharat0to: <pre><code class="lang-auto">
trans.RandomChoice([
        trans.RandomGrayscale(1),
        trans.RandomHorizontalFlip(1),
        trans.RandomVerticalFlip(1),
        trans.RandomAffine(180)
    ]),
</code></pre>
<p>I thought RandomChoice will choose only one among them so I kept probabilities to 1.</p>
<p>I have removed these transformations  and rerun the script but with no improvement.<br>
No the model did not work anytime and that is what I don’t know why.<br>
When I run on single example(giving the same X all the time) the loss sometimes decreases and sometimes it won’t.<br>
Please check the first link is now <a href="https://kaggle.com/bharat0/script-dogs-vs-cats?scriptVersionId=7412317" rel="nofollow noopener">https://kaggle.com/bharat0/script-dogs-vs-cats?scriptVersionId=7412317</a></p>
<p>I am running the second one without transformations. Once it is done I will post the link to it</p>

ptrblck: <p><code>RandomChoice</code> will randomly select one of these transformations.<br>
However if you specify <code>RandomHorizontalFlip</code> with a probability of 1, each image will be flipped.<br>
Try to create a simpler model without all dropout layers and maybe with less layers.<br>
It’s quite hard to debug your training if the model as well as the preprocessing is complicated, since there are a lot of knobs to tune now. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

bharat0to: <p>Sure, I will decrease the model and try.<br>
Preprocessing mainly consists of</p>
<pre><code class="lang-auto">img = Image.open(path_to_image)

y = 1 if 'dog' in path else 0
if self.transform:
     img = self.transform(img)
return img, y
</code></pre>
<p>path is like …/input/train/dog.123.jpg or …/input/train/cat.456.jpg</p>

bharat0to: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have minimized the model and I am thinking that</p>
<ol>
<li>I have used too many dropouts after every layer.</li>
<li>Applying the transformations in the beginning is a bad idea.<br>
We should apply transformations after the net learns something</li>
<li>Learning Rate is most important here<br>
I have used RMSprop optimizer which has default lr 0.01 and with this I the model is not learning.<br>
The model learning with Adam optimizer(default lr=0.001) and RMSprop with lr=0.001 is working.<br>
Then I have increased the model size and lr=0.001 is not working for both optimizer where as lr=0.0001 is working with both optimizers.<br>
I can say that adjusting learning rate made the difference here(more than transforms, dropout).</li>
</ol>
<p>Am I right or am I missing anything?<br>
The code is here: <a href="https://www.kaggle.com/bharat0/dog-vs-cat?scriptVersionId=7421775" rel="nofollow noopener">https://www.kaggle.com/bharat0/dog-vs-cat?scriptVersionId=7421775</a><br>
and thank you very much for the support</p>

ptrblck: <p>Your steps seem logic and apparently the debugging worked.<br>
Congrats! Now if your model is learning, you could try to scale it up a bit, e.g. adding data augmentation, and see if you can get a better accuracy.</p>

bharat0to: <p>Yes, I will train on full data. The code consists of only a sample</p>

monster: <p>Hi, I am currently using pretrained model like FasterRCNN , MaskRCNN ,etc but still getting false positives after fine-tuning…now I am planning to train a model from scratch(without using any pretrained model) but I am not sure it will work or not…so any suggestions ?</p>
<p>Also give me some information about object detection that is when to use pretrained with fine-tuning and model from scratch.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>