Transformer model loss is increasing

username: szahan
cooked: <p>Hi<br>
I am trying to design a transformer model using the nn.TransformerEncoderLayer() and nn.TransformerEncoder() to train in a self-supervised fashion to learn 3D coordinates of skeleton data. The model is as below</p>
<pre><code class="lang-auto">Model(
  (joint_embedding): embed(
    (cnn): Sequential(
      (0): norm_data(
        (bn): BatchNorm1d(30, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): cnn1x1(
        (cnn): Conv2d(3, 8, kernel_size=(1, 1), stride=(1, 1))
      )
      (2): ReLU()
    )
  )
  (dif_embedding): embed(
    (cnn): Sequential(
      (0): norm_data(
        (bn): BatchNorm1d(30, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): cnn1x1(
        (cnn): Conv2d(3, 8, kernel_size=(1, 1), stride=(1, 1))
      )
      (2): ReLU()
    )
  )
  (attention): Attention_Layer(
    (att): ST_Joint_Att(
      (fcn): Sequential(
        (0): Conv2d(8, 2, kernel_size=(1, 1), stride=(1, 1))
        (1): BatchNorm2d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): Hardswish()
      )
      (conv_t): Conv2d(2, 8, kernel_size=(1, 1), stride=(1, 1))
      (conv_v): Conv2d(2, 8, kernel_size=(1, 1), stride=(1, 1))
    )
    (bn): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act): Swish()
  )
  (pos_encoder): PositionalEncoding(
    (dropout): Dropout(p=0.5, inplace=False)
  )
  (encoder_layer): TransformerEncoderLayer(
    (self_attn): MultiheadAttention(
      (out_proj): NonDynamicallyQuantizableLinear(in_features=80, out_features=80, bias=True)
    )
    (linear1): Linear(in_features=80, out_features=2048, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
    (linear2): Linear(in_features=2048, out_features=80, bias=True)
    (norm1): LayerNorm((80,), eps=1e-05, elementwise_affine=True)
    (norm2): LayerNorm((80,), eps=1e-05, elementwise_affine=True)
    (dropout1): Dropout(p=0.1, inplace=False)
    (dropout2): Dropout(p=0.1, inplace=False)
  )
  (transformer): TransformerEncoder(
    (layers): ModuleList(
      (0): TransformerEncoderLayer(
        (self_attn): MultiheadAttention(
          (out_proj): NonDynamicallyQuantizableLinear(in_features=80, out_features=80, bias=True)
        )
        (linear1): Linear(in_features=80, out_features=2048, bias=True)
        (dropout): Dropout(p=0.1, inplace=False)
        (linear2): Linear(in_features=2048, out_features=80, bias=True)
        (norm1): LayerNorm((80,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((80,), eps=1e-05, elementwise_affine=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (mlp_head): Sequential(
    (0): LayerNorm((24000,), eps=1e-05, elementwise_affine=True)
    (1): Dropout(p=0.5, inplace=False)
    (2): Linear(in_features=24000, out_features=2048, bias=True)
    (3): ReLU(inplace=True)
    (4): Dropout(p=0.5, inplace=False)
    (5): Linear(in_features=2048, out_features=512, bias=True)
  )
)
</code></pre>
<p>I used simclr loss with the augmented sample as positive sample and all other samples in the minibatch as negative samples. Below is the loss curve</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d5e1c82b1f42b675da31eb38c3cc7acf8a9cd42.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d5e1c82b1f42b675da31eb38c3cc7acf8a9cd42" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d5e1c82b1f42b675da31eb38c3cc7acf8a9cd42_2_690x190.png" alt="image" data-base62-sha1="8KSL1cfUlXEJJrAvBg28tNWMPWG" width="690" height="190" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d5e1c82b1f42b675da31eb38c3cc7acf8a9cd42_2_690x190.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d5e1c82b1f42b675da31eb38c3cc7acf8a9cd42_2_1035x285.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d5e1c82b1f42b675da31eb38c3cc7acf8a9cd42_2_1380x380.png 2x" data-dominant-color="343332"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1397×386 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I used learning rate 0.001 with 10% reduction every 10th epoch with a warmup epoch of 100. Below is the learning rates graph</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76786efc2c293a033e89fc5a407979a9cc636999.png" data-download-href="https://discuss.pytorch.org/uploads/default/76786efc2c293a033e89fc5a407979a9cc636999" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76786efc2c293a033e89fc5a407979a9cc636999_2_690x192.png" alt="image" data-base62-sha1="gU2mT68hTpDOhbCHBtzF6boOiA9" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76786efc2c293a033e89fc5a407979a9cc636999_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76786efc2c293a033e89fc5a407979a9cc636999_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76786efc2c293a033e89fc5a407979a9cc636999.png 2x" data-dominant-color="333232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1322×368 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t understand why loss is increasing. Initial input shape: Nx3x300x10 and input to the transformer encoder layer is: Nx300x80<br>
My concerns are<br>
First, my model implementation may be incorrect<br>
Second, maybe the loss function is faulty.<br>
I would really appreciate it if you could suggest what should I appoch.</p>