Resnet18 optimization

username: Kla
cooked: <p>Hi everyone, I’m trying to use pretrained resnet18 for my project and it fits very good to my train data but not to validation data. I have tried changing batch size, convolutionlayers, lr_scheduler but there’s still no success.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d32ec2f107acfda58150c5a5832a47be257e0578.png" data-download-href="https://discuss.pytorch.org/uploads/default/d32ec2f107acfda58150c5a5832a47be257e0578" title="Screenshot from 2020-10-07 09-47-35"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d32ec2f107acfda58150c5a5832a47be257e0578_2_690x387.png" alt="Screenshot from 2020-10-07 09-47-35" data-base62-sha1="u8cVNKaO7WmS1j7gqqji21PLhS8" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d32ec2f107acfda58150c5a5832a47be257e0578_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d32ec2f107acfda58150c5a5832a47be257e0578_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d32ec2f107acfda58150c5a5832a47be257e0578.png 2x" data-dominant-color="E4DACF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-10-07 09-47-35</span><span class="informations">1366×768 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  This is my first project in DNN and I am a beginner. Can someone please give me a suggestion what can I do? Green is Train Loss and gray Validation Loss.</p>
<p>My Code looks like:</p>
<p>class Resnet_Speaker_Recognition(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Resnet_Speaker_Recognition, self).<strong>init</strong>()</p>
<pre><code>    self.resnet_model = models.resnet18(pretrained=True)
    self.resnet_model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=(7,7), stride=(2,2), bias=True)
    self.resnet_model.conv2 = torch.nn.Conv2d(64, 32, kernel_size=(7,7), stride=(2,2), bias=True)
    self.resnet_model.conv3 = torch.nn.Conv2d(32, 64, kernel_size=(7,7), stride=(2,2), bias=True) 

def forward(self,x):
    x=self.resnet_model(x)
    return x
</code></pre>

username: ptrblck
cooked: <p>How large are your current training, validation, and test datasets?<br>
Generally more samples help in this scenario. Additionally you could increase the regularization (such as adding or increasing dropout or adding weight decay), adding more data augmentation etc.<br>
It can also be useful to check the training and validation datasets manually and make sure the samples come from the same domain and are thus “similar”.</p>

username: Kla
cooked: <p>Hi, thank you for your answer. I am using VoxCeleb 1 as my dataset and it contains 138361 files for training, 8251 for test and 6904 files for validation. There are 1251 speakers to be identified. I was using till now optim.SGD as my optimizer, now I’m trying optim.Adam. It is a bit better but I’m still not having results. Could ypu please give me an simple example how to increase dropout?</p>
<p>I have changed my model again:</p>
<p>class Resnet_Speaker_Recognition(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Resnet_Speaker_Recognition, self).<strong>init</strong>()</p>
<pre><code>    self.resnet_model = models.resnet18(pretrained=True)
    self.resnet_model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=(7,7), stride=(2,2), padding=(3,3), bias=False)
    self.num_ftrs = self.resnet_model.fc.in_features
    self.resnet_model.fc = nn.Linear(self.num_ftrs, 1251)

def forward(self,x):

    x = self.resnet_model(x)
    return x</code></pre>

username: ptrblck
cooked: <p>I think resnets don’t use dropout initially so you could add e.g. an <code>nn.Sequential</code> module as <code>self.resnet_model.fc</code> containing linear and dropout layers and could check, if it can improve the training.</p>

username: Kla
cooked: <p>It’s unfortunately still not working. Train Dataset works perfectly but Validation not. Can you please tell me how to check the Datasets manually?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kla" data-post="5" data-topic="98530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kla/48/23505_2.png" class="avatar"> Kla:</div>
<blockquote>
<p>Can you please tell me how to check the Datasets manually?</p>
</blockquote>
</aside>
<p>Plot random images (or all) and check if the data might come from different domains, might contain different features, has special artifacts etc.</p>