Model overfitting - validation not improving

Yana_G: <p>Hi,<br>
I’m facing an overfitting problem, my model get very high accuracy on the training set ~99.8% accuracy<br>
while on the validation set i’m getting worse result ~41% accuracy.<br>
I tried adding more data (30K more images to the training set and validation set) and tried also data augmentation but the model just not improving on the validation set.</p>
<p>Any idea why?</p>
<p>Yana</p>

ptrblck: <p>You could add some regularization like <code>Dropout</code> or weight decay. Did data augmentation help at all?<br>
What kind of images do you have?</p>
<p>Did you make sure to preprocess the training and validation images in the same way (normalizing, etc.)?</p>

Yana_G: <p>Thanks for your reply.<br>
I added Dropout just before the last fc layer which didn’t help much, i’m training resnet50 on my dataset, my dataset contains images 600x600 of jewellery images.<br>
data augmentation didn’t help and plus adding more data didn’t help.<br>
i made sure that i’m using the same preprocessing on my training and my validation images.<br>
not sure what else can i do, and why it’s not converging on my validation set</p>

melgor: <p>Look like sth wrong is in your pipeline. If you have &gt; 50k images, over-fitting with Dropout should not be a big problem.<br>
Try pretrained model and learn just last layer (so dont optimize rest of them, just pass <code>model.fc.parameters()</code> to optimizer)</p>
<p>If this also doesn’t help, maybe make again random split between training and validation images (maybe distribution of images in this two sets is too different)</p>
<p>Last part is training without Data-Augmentation, maybe turn it off and see what will happend.</p>

Yana_G: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/melgor">@melgor</a>,<br>
Here is an example of what is happening to me during training and then evaluating the model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4edc4d23d18a10897071af747742017f81b554f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/4edc4d23d18a10897071af747742017f81b554f6" title="22"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4edc4d23d18a10897071af747742017f81b554f6_2_690x321.png" alt="22" data-base62-sha1="bfDeDClkMyE6Vp0ysKc3yy94lH8" width="690" height="321" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4edc4d23d18a10897071af747742017f81b554f6_2_690x321.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4edc4d23d18a10897071af747742017f81b554f6_2_1035x481.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/4edc4d23d18a10897071af747742017f81b554f6.png 2x" data-dominant-color="1E424B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">22</span><span class="informations">1070×499 34 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I made sure that i set <code>model.eval()</code> when evaluating.</p>
<p>I’m out of ideas</p>

ptrblck: <p>Do you get these bad results for the validation set from the beginning or is it improving a bit?</p>
<p>Could you post your <code>Dataset</code>s with all pre-processing code?<br>
How was the data split into training and validation? Was it a random split or was some logic involved?</p>

Yana_G: <p>i get these bad result after a few epochs when the training accuracy becomes ~78%<br>
the data was randomly split into training and validation using <code>sklearn.model_selection.train_test_split</code> method.</p>
<p>This is my <code>Dataset</code>'s code with all the pre processing:</p>
<pre><code class="lang-auto">class ImageAdapter(metaclass=ABCMeta):
	@abstractmethod
	def get_image(self, index):
		pass

	@abstractmethod
	def get_y(self, index):
		pass

	@abstractmethod
	def __len__(self):
		pass

class FileSystemAdapter(ImageAdapter):
	"""docstring for FileSystemAdapter"""
	def __init__(self, videos_path, dataset_json_filename, width=600, height=600):
		super(FileSystemAdapter, self).__init__()
		self.width = width
		self.height = height
		self.dataset, _ = self.load_dataset(dataset_json_filename)

	def load_dataset(self, dataset_json_filename):
		with open(dataset_json_filename, 'r') as fp:
			data = json.load(fp)
			return data["dataset"], None

	def crop_center(self, img,cropx,cropy):
		y,x,c = img.shape
		startx = x//2 - cropx//2
		starty = y//2 - cropy//2    
		return img[starty:starty+cropy, startx:startx+cropx, :]

	def load_img(self, path):
		img = Image.open(path)
		if img.mode != 'RGB':
			img = img.convert('RGB')
		return img

	def img_to_array(self, img):
		return np.asarray(img, dtype=np.float32)/255.

	def image_loader(self, path):
		img = self.load_img(path)
		img_arr = self.img_to_array(img)
		img_arr = self.crop_center(img_arr, self.width, self.height)
		return img_arr

	def get_y(self, index):
		return self.dataset[index]['target']

	def get_image(self, idx):
		image = self.dataset[idx]['image']
		return self.image_loader(image)

	def __len__(self):
		return len(self.dataset)

class MyDataset(data.Dataset):
    def __init__(self, adapter, spatial_transform=None, normalization=(None, None)):
        self.adapter = adapter
        self.spatial_transform = spatial_transform
        self.mean, self.std = normalization

    def get_y(self, index):
        return self.adapter.get_y(index)

    def __getitem__(self, index):
        image = self.adapter.get_image(index)
        target = self.adapter.get_y(index)

        #normalize image
        if self.mean is not None and self.std is not None:
            image = (image - self.mean)/self.std

        #perform spatial transform
        if self.spatial_transform is not None:
            image = self.spatial_transform(image)

        return image, target

    def __len__(self):
        return len(self.adapter)
</code></pre>

ptrblck: <p>Before the training accuracy reaches ~78% the validation accuracy is rising?<br>
Are the validation predictions biased towards some classes the whole time?</p>
<p>The code looks fine, as long as the normalization is performed in the same way for the training and validation data.<br>
Could you check the data stats (mean, std, min, max) for some samples taken from both <code>DataLoaders</code>?</p>

Yana_G: <p>Yup, the validation accuracy is rising and then it starting to drop giving me huge loss.<br>
at the beginning when the validation accuracy rising the predictions are spread on the whole classes but when<br>
the training accuracy rises the validation predictions are biased towards 1-2 classes (not the same class every epoch - for example in epoch 12 the validation predictions are biased towards classes 1 and 2, and on epoch 13 they are biased towards classes 3 and 7)<br>
I checked my data stats, mean, std, min and max and the training stats are very similar to the validation stats that’s why i’m worried.<br>
I even tried to do inference on my training set and it gave me bad results.<br>
i’m training some of the batchnorm layers in resnet50 but my batch size isn’t small, maybe it has something to do with this?</p>

ptrblck: <p>Based on the screenshot you shared, your training accuracy and confusion matrix looks good.<br>
What do you mean by “tried to do inference on my training set and it gave me bad results”?<br>
Did you switch to <code>eval</code> and the results were also bad on the training set?</p>
<p>If your batch size isn’t small, it’s not problematic to use <code>BatchNorm</code> layers. The opposite could be problematic.</p>

Yana_G: <p>i switched the model to <code>eval()</code> and used my train data loader to test the model on the training set to see if my problem is occurring on the training set also (got bad results on my training set) - which it did so i guess it’s not a normalization problem (which i thought at the beginning)</p>
<p>i’m using <code>batch_size=64</code> for the training and for the eval phase</p>
<p>I even made sure that the layers weights are not updating in the eval mode.</p>

ptrblck: <p>Did you change the <code>BatchNorm</code> layers somehow (the <code>momentum</code> etc.)?<br>
What happens if you freeze the <code>BatchNorm</code> layers?</p>

Yana_G: <p>i did not make any change in the <code>BatchNorm</code> layers.<br>
I tried freezing all the BatchNorm layers and train the model - same results<br>
I even tried to freeze the <code>BatchNorm</code> layers and do a transfer learning - same results<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=5" title=":sob:" class="emoji" alt=":sob:"></p>

Yana_G: <p>I might be shooting in the dark but maybe it has something to do with this: <a href="https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561">https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561</a>?</p>

ptrblck: <p>You could definitely try the suggestions from this thread!<br>
Let me know, if it helps somehow.</p>

NoobCoder: <p>Hello,</p>
<p>I have same problem, and freezing the BN does not change anything.</p>
<p>I am doing a simple classification task in medical imaging area. I have a dataset about 50k images and 4 class. The model overfitting a lot (below are the training for 4 epochs)</p>
<pre><code class="lang-auto">Validation accuracy before training is 0.28779987373737376

Epoch: 0
Training accuracy: 0.7547950186346696, Validation accuracy: 0.4102746212121212
Training loss: 0.5928439186062924, Validation loss: 1.9086839990182356

Epoch: 1
Training accuracy: 0.9022588855558585, Validation accuracy: 0.38691603535353536
Training loss: 0.25842620344117806, Validation loss: 2.5381281658856554

Epoch: 2
Training accuracy: 0.939960003636033, Validation accuracy: 0.36900252525252525
Training loss: 0.16145963321418264, Validation loss: 2.9992803214776393
</code></pre>
<p>so basically after the first iteration, it is overfitting. What should I do? I have tried everything I know from dropout, augmentation, increasing dataset, using WeightedSampler, weight decay.</p>
<p>I really do not understand how after the first epoch it is overfitting heavily. I have tested models from MobileNet to Inception. Nothing works.</p>
<p>There are two main questions for me:</p>
<ol>
<li>Why adding more data is not improving the result?</li>
<li>Why it is overfitting only after first epoch?</li>
</ol>

ptrblck: <ol>
<li>
<p>Using more data should improve the ability to generalize. I would start by checking e.g. the preprocessing pipeline and make sure that e.g. the normalization is applied in the training and validation use case. A quick test might be to use the training dataset in the validation loop and check it’s performance.</p>
</li>
<li>
<p>I’m unsure if the model is only overfitting <em>after</em> the first epoch, since <code>epoch0</code> already shows a worse validation accuracy than the training one.</p>
</li>
</ol>

NoobCoder: <p>Thanks for your reply.</p>
<p>I have tested this one, and accuracy for testing training dataset is near 90%, so I can say everything is fine regarding the preprocessing methods.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="18" data-topic="17778" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<ol>
<li>Using more data should improve the ability to generalize. I would start by checking e.g. the preprocessing pipeline and make sure that e.g. the normalization is applied in the training and validation use case. A quick test might be to use the training dataset in the validation loop and check it’s performance.</li>
</ol>
</blockquote>
</aside>
<p>I really do not get how the model is acting like this. If it is not overfitting, what is happnening here? Any suggestion for improving this case?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="18" data-topic="17778" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<ol start="2">
<li>I’m unsure if the model is only overfitting <em>after</em> the first epoch, since <code>epoch0</code> already shows a worse validation accuracy than the training one.</li>
</ol>
</blockquote>
</aside>
<p>Also when I plot the confusion matrix, I see that most of the data are biased toward first class. What should I do for removing this bias? Add weight balancing to loss or try OnevsRest?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/4283fdd8e653286b93f03b97c269b9e44b73f77c.png" alt="image" data-base62-sha1="9uqiPx5U0KdRpnYKepDEl6Tg6Ru" width="271" height="71"></p>