Overfitting in my model when applying on EEG signals for classification

username: Falcon007
cooked: <p>class MSFBCNN(nn.Module):</p>
<pre><code>def __init__(self,input_dim,output_dim,FT=10):

    super(MSFBCNN, self).__init__()

    self.T = input_dim[1]

    self.FT = FT

    self.D = 1

    self.FS = self.FT*self.D

    self.C=input_dim[0]

    self.output_dim = output_dim

            

    # Parallel temporal convolutions

    self.conv1a = nn.Conv2d(1, self.FT, (1, 65), padding = (0,32),bias=False)

    self.conv1b = nn.Conv2d(1, self.FT, (1, 41), padding = (0,20),bias=False)

    self.conv1c = nn.Conv2d(1, self.FT, (1, 27), padding = (0,13),bias=False)

    self.conv1d = nn.Conv2d(1, self.FT, (1, 17), padding = (0,8),bias=False)

    self.batchnorm1 = nn.BatchNorm2d(4*self.FT, False)

    self.pooling1 = nn.AvgPool2d(kernel_size=(1, 16), stride=(1,2), padding=(0,0))

    # Spatial convolution

    self.conv2 = nn.Conv2d(4*self.FT, self.FS, (self.C,1),padding=(0,0),groups=1,bias=False)

    self.batchnorm2 = nn.BatchNorm2d(self.FS, False)

    #Temporal average pooling

    self.pooling2 = nn.AvgPool2d(kernel_size=(1, 32),stride=(1,4),padding=(0,0))

    self.drop=nn.Dropout(0.44)

    

    #LSTM

    self.lstm = nn.LSTM(input_size=131,hidden_size=512,num_layers=2, batch_first=True)

    #Classification       

    self.fc1 = nn.Linear(512, self.output_dim)

def forward(self, x):

    # Layer 1

    x1 = self.conv1a(x);

    x2 = self.conv1b(x);

    x3 = self.conv1c(x);

    x4 = self.conv1d(x);

    x = torch.cat([x1,x2,x3,x4],dim=1)

    x = self.batchnorm1(x)

    x = self.pooling1(x)

    x = self.drop(x)

    

    # Layer 2

    x = torch.pow(self.batchnorm2(self.conv2(x)),2)

    x = self.pooling2(x)

    #x = torch.log(x)

    x = self.drop(x)

    

    #LSTM

    x = x.view(x.size(0),x.size(1),x.size(3))

    #x = x[:, 1, :, :]

    h0 = torch.zeros(2, x.size(0), 512).cuda().requires_grad_()

    c0 = torch.zeros(2, x.size(0), 512).cuda().requires_grad_()

    x, _ = self.lstm(x, (h0, c0))

    # FC Layer

    x = x[:, -1, :]

    x = self.fc1(x)

    return x
</code></pre>
<p>Can anybody tell me how to reduce overfitting? I change every hyperparameter. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: KFrank
cooked: <p>Hi Abdullah!</p>
<aside class="quote no-group" data-username="Falcon007" data-post="1" data-topic="144377" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falcon007/48/49237_2.png" class="avatar"> Falcon007:</div>
<blockquote>
<p>Can anybody tell me how to reduce overfitting?</p>
</blockquote>
</aside>
<p>First a word on “overfitting:”  In my mind, overfitting means that your<br>
out-of-sample prediction performance (e.g., on your validation or test<br>
set) is actually getting <em>worse,</em> even as your training-set performance<br>
is getting better.  It is common for the training-set performance to be<br>
better than that for the validation (or test) set, sometimes significantly.<br>
But I don’t call it overfitting until the out-of-sample performance starts<br>
to degrade with further training.</p>
<p>The best way to address overfitting – if you have the data and resources<br>
to train with a large training set – is to use a larger training set.</p>
<p>Next is data augmentation, where you semi-artificially increase the size<br>
of your training set by generating “new” samples, for example by flipping<br>
images left-to-right or cropping them or adding noise to them.  (But you<br>
can only push augmentation so far – as you start to generate lots of<br>
“augmentation” samples from a single “real” sample, you’ll hit diminishing<br>
returns.)</p>
<p>Adding <a href="https://pytorch.org/docs/1.10.0/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">Dropout</a> layers to your model is an established technique for<br>
reducing overfitting.</p>
<p><em>Weight regularization,</em> such as using the <code>weight_decay</code> parameter in<br>
various pytorch optimizers, is used to reduce overfitting.</p>
<p>There is some lore that <em>batch normalization</em>, for example, pytorch’s<br>
<a href="https://pytorch.org/docs/1.10.0/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc">BatchNorm2d</a>, can reduced overfitting, although I’ve seen this argued<br>
both ways.</p>
<p>Also smaller models (fewer parameters) are in general less prone to<br>
overfitting, but even when well trained, smaller models may not perform<br>
as well on your use case as a better-suited, but larger, model would.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: Falcon007
cooked: <p>Thank you for the reply, <a class="mention" href="/u/kfrank">@KFrank</a>. How to do data augmentation of EEG signals?</p>

username: KFrank
cooked: <p>Hi Abdullah!</p>
<aside class="quote no-group" data-username="Falcon007" data-post="3" data-topic="144377" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falcon007/48/49237_2.png" class="avatar"> Falcon007:</div>
<blockquote>
<p>How to do data augmentation of EEG signals?</p>
</blockquote>
</aside>
<p>I don’t really know anything about EEGs, but I think you could try:</p>
<p>Add some modest noise to the signals.</p>
<p>Rescale the strength of the signals.  I would imagine that you could<br>
use modestly different rescalings on the different channels, and maybe<br>
have the rescalings vary slowly over time.</p>
<p>Shift / crop the signals in a time window.  So let’s say you have 10-minute<br>
EEG recordings.  Maybe you could randomly crop them to 8-minute<br>
windows, and train on the 8-minute samples.</p>
<p>I don’t know how much the exact rhythm matters, but perhaps you could<br>
rescale the time axis a little, perhaps in a time-dependent way.</p>
<p>The idea is that you want to create new “augmentation” samples from<br>
your original samples in which specific irrelevant details are modified,<br>
while maintaining the important characteristics.  So a trained “EEG<br>
reader” looking at a modified sample should come to the same<br>
conclusion as he would looking at the original unmodified sample<br>
(e.g., seizure vs. no seizure).</p>
<p>The idea is that overfitting occurs when the model learns random details<br>
of specific samples, rather than the general properties it’s supposed to<br>
learn.</p>
<p>Let’s say that you are training a cat / dog image classifier, and one of<br>
your dog training images has a big, black dog nose on the left edge<br>
of the image.  A model that is overfitting might learn that a black blob<br>
on the left edge of the image means “dog,” but that’s just an artifact<br>
of a particular training sample.  If you augment by flipping the image<br>
left-to-right, it will still be a dog (and look like one to a person), but the<br>
model can’t use left-edge-blob to mean dog.  Or if you augment by<br>
cropping the image, and snip the dog’s nose off (but the rest of the<br>
dog is still in the image and still looks like a dog), the model can’t use<br>
black blob (absent the rest of the dog) to mean dog.</p>
<p>So, how can you make modified versions of your training-set EEG traces<br>
that preserve all (or most) of the important diagnostic information, but<br>
mess up or hide sample-specific irrelevant details in order to prevent<br>
your model from learning the irrelevant details in lieu of the features that<br>
actually matter?</p>
<p>Best.</p>
<p>K. Frank</p>