Transformer erratic training loss when given more training data

username: nicofish
cooked: <p>Hello, I am have a model for classifies single particle trajectories (according to what algorithm was used to generate the trajectories).</p>
<p>My current model structure has 2 convolutional layers a bidirectional LSTM of length 3 and some Linear layers for output.</p>
<p>Here is structure of my convolutional LSTM</p>
<pre><code class="lang-auto">ConejeroConvNet(
  (ConvBlock): Sequential(
    (0): Conv1d(1, 20, kernel_size=(3,), stride=(1,))
    (1): ReLU()
    (2): Conv1d(20, 64, kernel_size=(3,), stride=(1,))
    (3): ReLU()
    (4): Dropout(p=0.2, inplace=False)
    (5): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (bi_lstm): LSTM(498, 32, num_layers=3, batch_first=True)
  (linearOuts): Sequential(
    (0): Linear(in_features=4096, out_features=1000, bias=True)
    (1): ReLU()
    (2): Linear(in_features=1000, out_features=50, bias=True)
    (3): ReLU()
    (4): Linear(in_features=50, out_features=5, bias=True)
  )
)
</code></pre>
<p>I replaced the LSTM with a classifier transformer adapted from the classifier transformer here: <a href="http://peterbloem.nl/blog/transformers" class="inline-onebox" rel="noopener nofollow ugc">Transformers from scratch | peterbloem.nl</a></p>
<p>Here is the structure of my transformer</p>
<pre><code class="lang-auto">convTransformer(
  (ConvBlock): Sequential(
    (0): Conv1d(1, 20, kernel_size=(3,), stride=(1,))
    (1): ReLU()
    (2): Conv1d(20, 64, kernel_size=(3,), stride=(1,))
    (3): ReLU()
    (4): Dropout(p=0.2, inplace=False)
    (5): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (cTrans): ClassifierTransformer(
    (transBlocks): Sequential(
      (0): TransformerBlock(
        (attention): SelfAttentionNarrow(
          (toKeys): Linear(in_features=8, out_features=8, bias=False)
          (toQueries): Linear(in_features=8, out_features=8, bias=False)
          (toValues): Linear(in_features=8, out_features=8, bias=False)
          (unifyHeads): Linear(in_features=64, out_features=64, bias=True)
        )
        (norm1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (ff): Sequential(
          (0): Linear(in_features=64, out_features=256, bias=True)
          (1): ReLU()
          (2): Linear(in_features=256, out_features=64, bias=True)
        )
        (dropOut): Dropout(p=0.2, inplace=False)
      )
      (1): TransformerBlock(
        (attention): SelfAttentionNarrow(
          (toKeys): Linear(in_features=8, out_features=8, bias=False)
          (toQueries): Linear(in_features=8, out_features=8, bias=False)
          (toValues): Linear(in_features=8, out_features=8, bias=False)
          (unifyHeads): Linear(in_features=64, out_features=64, bias=True)
        )
        (norm1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (ff): Sequential(
          (0): Linear(in_features=64, out_features=256, bias=True)
          (1): ReLU()
          (2): Linear(in_features=256, out_features=64, bias=True)
        )
        (dropOut): Dropout(p=0.2, inplace=False)
      )
      (2): TransformerBlock(
        (attention): SelfAttentionNarrow(
          (toKeys): Linear(in_features=8, out_features=8, bias=False)
          (toQueries): Linear(in_features=8, out_features=8, bias=False)
          (toValues): Linear(in_features=8, out_features=8, bias=False)
          (unifyHeads): Linear(in_features=64, out_features=64, bias=True)
        )
        (norm1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (ff): Sequential(
          (0): Linear(in_features=64, out_features=256, bias=True)
          (1): ReLU()
          (2): Linear(in_features=256, out_features=64, bias=True)
        )
        (dropOut): Dropout(p=0.2, inplace=False)
      )
..............[more transformer blocks].......................
      (9): TransformerBlock(
        (attention): SelfAttentionNarrow(
          (toKeys): Linear(in_features=8, out_features=8, bias=False)
          (toQueries): Linear(in_features=8, out_features=8, bias=False)
          (toValues): Linear(in_features=8, out_features=8, bias=False)
          (unifyHeads): Linear(in_features=64, out_features=64, bias=True)
        )
        (norm1): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
        (ff): Sequential(
          (0): Linear(in_features=64, out_features=256, bias=True)
          (1): ReLU()
          (2): Linear(in_features=256, out_features=64, bias=True)
        )
        (dropOut): Dropout(p=0.2, inplace=False)
      )
    )
    (dropout): Dropout(p=0.2, inplace=False)
    (linOut): Linear(in_features=64, out_features=5, bias=True)
  )
)
</code></pre>
<p>With data set of size 10k (7.5k for training 2.5 for test) the transfomer outperformed my convolutional LSTM by 9%. However, when I tried it with a data set of size 150k (110k train 40k test) the training loss went all over the place and it performed only slightly better than guessing randomly.</p>
<p>This is what my model does when given 110k training data. the losses are jumping around and never go below 3.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/5178b9d536f823877d33cff80960c4ac80da5aac.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5178b9d536f823877d33cff80960c4ac80da5aac" title="WhatsApp Image 2020-12-01 at 3.00.05 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/5178b9d536f823877d33cff80960c4ac80da5aac_2_690x225.jpeg" alt="WhatsApp Image 2020-12-01 at 3.00.05 AM" data-base62-sha1="bCJkplARafNCLQFuGsStX5FVDf6" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/5178b9d536f823877d33cff80960c4ac80da5aac_2_690x225.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/5178b9d536f823877d33cff80960c4ac80da5aac_2_1035x337.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/1/5178b9d536f823877d33cff80960c4ac80da5aac.jpeg 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2020-12-01 at 3.00.05 AM</span><span class="informations">1036×338 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see below. it behaves as expected with 7.5k training data.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95bee8e7f3fdf67d0d948d2792f11bba1c32379b.png" alt="image" data-base62-sha1="lmIafVAufoTIfRrM3KG64zfdJjB" width="516" height="243"><br>
the losses dont jump as much and end around .7.</p>
<p>I kept everything the same between the runs (batch size, optimizer criterion, epochs, patience, etc.). I should add that I am using a modified version of Early stopping available here: <a href="https://github.com/Bjarten/early-stopping-pytorch/blob/master/MNIST_Early_Stopping_example.ipynb" class="inline-onebox" rel="noopener nofollow ugc">early-stopping-pytorch/MNIST_Early_Stopping_example.ipynb at master · Bjarten/early-stopping-pytorch · GitHub</a>.</p>
<p>I can add more detail if necessary. However, I think that my problem is conceptual as I do not understand why <strong>only</strong> my transformer model is acting like this. When I add more data to my convolutional LSTM model it works fine and it improves the models capability of prediction.</p>
<p>Thank you very much for your help. I appreciate any suggestions to help me fix the problem.</p>
<p>edited: formatting and typos</p>

username: nicofish
cooked: <p>Responding to my own question in case anyone ever has this question in the future.</p>
<p>I introduced gradient clipping as shown here:<br>
<a href="https://stackoverflow.com/questions/54716377/how-to-do-gradient-clipping-in-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/54716377/how-to-do-gradient-clipping-in-pytorch</a></p>
<pre><code class="lang-auto">for p in model.parameters():
    p.register_hook(lambda grad: torch.clamp(grad, -clip_value, clip_value))
</code></pre>
<p>This appears to have remediated the problem.</p>