Model overfits and does not improve on validation accuracy

Hungreeee: <p>Hi there,<br>
I am building a feedforward network for a binary text classification task. Here’s my model:</p>
<pre data-code-wrap="class"><code class="lang-class">  def __init__(self, output_size, embedding_dim, hidden_size, dropout=0.2):
    super(FeedForward, self).__init__()
    self.embedding = nn.Embedding.from_pretrained(embedding_matrix)
    self.linear_relu_stack = nn.Sequential(
      nn.Linear(in_features=embedding_dim, out_features=hidden_size),
      nn.ReLU(),
      nn.Dropout(p=dropout),
      nn.Linear(in_features=hidden_size, out_features=hidden_size//2),
      nn.ReLU(),
      nn.Dropout(p=dropout),
      nn.Linear(in_features=hidden_size//2, out_features=output_size),
      nn.Sigmoid()
    ) 

  def forward(self, input):
    emb = self.embedding(input)
    emb = torch.sum(emb, dim=1)
    out = self.linear_relu_stack(emb)
    return out

model = FeedForward(OUTPUT_SIZE, EMBEDDINGS_DIM, HIDDEN_SIZE, 0.5).to(device)
print(model)
</code></pre>
<p>OUTPUT:</p>
<pre><code class="lang-auto">  (embedding): Embedding(204775, 300)
  (linear_relu_stack): Sequential(
    (0): Linear(in_features=300, out_features=512, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.4, inplace=False)
    (3): Linear(in_features=512, out_features=256, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.4, inplace=False)
    (6): Linear(in_features=256, out_features=1, bias=True)
    (7): Sigmoid()
  )
)
</code></pre>
<p>I am training using 64-batch. I have 1,276,686 rows in train set and 159,586 rows in validation set. I am using FastText word embeddings, and use <code>loss_function = nn.BCELoss()</code> and<br>
<code>optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-7)</code>.<br>
<strong>Now, the problem is that my model tends to overfit quickly after a few epochs.</strong> I have tried training in shuffled mini-batches, decreasing the complexity of the model, add stronger dropout and higher weight decay. But no matter what I do, the model will always overfit once the training accuracy gets to around 81% and the validation loss will go up. The model will not improve and go any further than 81% on validation data. How should I approach this issue?</p>
<p>Here is my training graph of loss: (validation is orange, train is blue)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de91b433abc888c655e433fd537cf8e8b20ff36.png" data-download-href="https://discuss.pytorch.org/uploads/default/4de91b433abc888c655e433fd537cf8e8b20ff36" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4de91b433abc888c655e433fd537cf8e8b20ff36_2_683x500.png" alt="image" data-base62-sha1="b7ec08WfXShPf3d7xX3NqSUPm74" width="683" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4de91b433abc888c655e433fd537cf8e8b20ff36_2_683x500.png, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de91b433abc888c655e433fd537cf8e8b20ff36.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de91b433abc888c655e433fd537cf8e8b20ff36.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">703×514 29.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And accuracy:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/900491e02562be4deadb8b5dc1a9f51e8e8e0faf.png" data-download-href="https://discuss.pytorch.org/uploads/default/900491e02562be4deadb8b5dc1a9f51e8e8e0faf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/900491e02562be4deadb8b5dc1a9f51e8e8e0faf_2_683x500.png" alt="image" data-base62-sha1="ky2xSWNV4TPzk1btjXjWVFipQqr" width="683" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/900491e02562be4deadb8b5dc1a9f51e8e8e0faf_2_683x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/0/900491e02562be4deadb8b5dc1a9f51e8e8e0faf.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/0/900491e02562be4deadb8b5dc1a9f51e8e8e0faf.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">699×511 32.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vdw: <p>As far as I am concerned, your plots like as expected as it gets :). That your test/validation loss will stop improving at some point simply means that your test/validation data is in some sense different from the training data. In other words, your test/validation just “looks” different from your training data, which is in practice normally the case.</p>
<p>In some sense this means that you don’t have enough data so that any split into training/validation/test sets will yield sets the “look” the same. Again, this is more the exception in practice.</p>