Overfitting and hanging

Sylvain_Ard: <p>Hello,<br>
my classification program sometimes hangs at random epoch  (the program doesn’t advance anymore) and tends to overfit. How to fix these two problems.<br>
This is the program : <a href="https://www.dropbox.com/s/ch0ntqvdm312z0k/train.py?dl=0" class="inline-onebox" rel="noopener nofollow ugc">Dropbox - train.py - Simplify your life</a><br>
Thank you for helping me<br>
Best regards</p>

Sylvain_Ard: <p>Noone has an answer please ?</p>

Sylvain_Ard: <p>huh ? nobody please ?</p>

ptrblck: <p>Your description and code snippet isn’t isolating the issue sufficiently as the code is not even executable.<br>
I don’t see any obvious issues in the code so you would need to narrow down where exactly the code is hanging and try to isolate it further to a minimal, executable code snippet.</p>

Sylvain_Ard: <p>ok but could you explain to me how to avoid overfitting ? the hanging is coming randomly with other codes, i suspect overheating of GPU</p>

ptrblck: <p>Your model would overfit to the training dataset if it’s able to learn it properly while the validation dataset shows a gap in its loss or accuracy. This would mean that your model learns individual features of the training dataset samples instead of features which would generalize to the classes.<br>
This is often observed if the dataset is too small compared to the “capacity” of the model (i.e. it’s number of trainable parameters). I’m sure the literature can explain it better and in more detail.<br>
You would usually try to increase the dataset, add a more aggressive data augmentation strategy, or reduce the model capacity either directly by using a smaller model, by adding e.g. dropout during training, or by fine-tuning a frozen and pretrained model.<br>
We have a lot of topics in this discussion board where users are trying to avoid overfitting so reading through some of these might also help and give you some ideas to tackle this issue.</p>

Sylvain_Ard: <p>Ok thank you I have about 1000 images, is a resnet50 would be better ? And how to make dropout please ? Thank you Best regards</p>
<p>Envoyé de mon iPhone</p>

ptrblck: <aside class="quote no-group" data-username="Sylvain_Ard" data-post="7" data-topic="156358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar"> Sylvain_Ard:</div>
<blockquote>
<p>is a resnet50 would be better ?</p>
</blockquote>
</aside>
<p>I don’t know and you would need to run some experiments.</p>
<aside class="quote no-group" data-username="Sylvain_Ard" data-post="7" data-topic="156358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar"> Sylvain_Ard:</div>
<blockquote>
<p>And how to make dropout please ?</p>
</blockquote>
</aside>
<p>You can add it via a module as <code>nn.Dropout</code> to the model directly or via its functional API:</p>
<pre data-code-wrap="python"><code class="lang-python">x = F.dropout(x, training=self.training)
</code></pre>
<p>in the model’s <code>forward</code> method.</p>

Sylvain_Ard: <p>Could you please give me a piece of code including the dropout ?</p>

ptrblck: <p>Sure! Assume you want to add another linear layer as well as a dropout layer to <code>renset50</code>’s classifier (which is a single linear layer assigned to <code>model.fc</code>), you could replace it with a new <code>nn.Sequential</code> container and add the new layers as well as the original linear layer to it as seen here:</p>
<pre><code class="lang-python">model = models.resnet50()

nb_features = model.fc.in_features
model.fc = nn.Sequential(
    nn.Linear(nb_features , nb_features ),
    nn.Dropout(),
    model.fc
)

x = torch.randn(2, 3, 224, 224)
out = model(x)
</code></pre>

Sylvain_Ard: <p>ok thank you and for a ResNet 101 it is the same code ?</p>

Sylvain_Ard: <p>second question : is it will fix the overfitting ?</p>

ptrblck: <aside class="quote no-group" data-username="Sylvain_Ard" data-post="11" data-topic="156358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar"> Sylvain_Ard:</div>
<blockquote>
<p>and for a ResNet 101 it is the same code ?</p>
</blockquote>
</aside>
<p>Yes, it should be the same or really similar and you should be able to check the attributes for any other model using my previous code snippet.</p>
<aside class="quote no-group" data-username="Sylvain_Ard" data-post="12" data-topic="156358">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar"> Sylvain_Ard:</div>
<blockquote>
<p>is it will fix the overfitting ?</p>
</blockquote>
</aside>
<p>Dropout can certainly reduce overfitting, buit as always it depends on your use case and you would have to run the experiments.</p>

Sylvain_Ard: <p>thank you very much ! I will try</p>