My multi-dimensional Transformer doesn't seem to learn anything

username: shamoons
cooked: <p>I’m trying to go <code>seq2seq</code> with a Transformer model. My input and output are the same shape (<code>torch.Size([499, 128])</code> where 499 is the sequence length and 128 is the number of features.</p>
<p>My input looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/4528cbcfb63a8e4b9fc8d9203c6a3c2d5bfb99bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/4528cbcfb63a8e4b9fc8d9203c6a3c2d5bfb99bc" title="spec.x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4528cbcfb63a8e4b9fc8d9203c6a3c2d5bfb99bc_2_345x114.png" alt="spec.x" data-base62-sha1="9ROlKpDbMWR7sKcNvbajN97zhPC" width="345" height="114" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4528cbcfb63a8e4b9fc8d9203c6a3c2d5bfb99bc_2_345x114.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4528cbcfb63a8e4b9fc8d9203c6a3c2d5bfb99bc_2_517x171.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4528cbcfb63a8e4b9fc8d9203c6a3c2d5bfb99bc_2_690x228.png 2x" data-dominant-color="98A0DB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">spec.x</span><span class="informations">1200×400 44.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My output looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8ff6d8bd73fc310bb779661e1ae01d8a75f400a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ff6d8bd73fc310bb779661e1ae01d8a75f400a6" title="spec.y"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff6d8bd73fc310bb779661e1ae01d8a75f400a6_2_345x114.png" alt="spec.y" data-base62-sha1="kxz8Yt5PCH4FP29524gQWsi3B8q" width="345" height="114" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff6d8bd73fc310bb779661e1ae01d8a75f400a6_2_345x114.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff6d8bd73fc310bb779661e1ae01d8a75f400a6_2_517x171.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff6d8bd73fc310bb779661e1ae01d8a75f400a6_2_690x228.png 2x" data-dominant-color="99A1DB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">spec.y</span><span class="informations">1200×400 45.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My training loop is:</p>
<pre><code class="lang-auto">    for batch in tqdm(dataset):
        optimizer.zero_grad()
        x, y = batch

        x = x.to(DEVICE)
        y = y.to(DEVICE)

        pred = model(x, torch.zeros(x.size()).to(DEVICE))

        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()
</code></pre>
<p>My model is:</p>
<pre><code class="lang-auto">import math
from typing import final
import torch
import torch.nn as nn

class Reconstructor(nn.Module):
    def __init__(self, input_dim, output_dim, dim_embedding, num_layers=4, nhead=8, dim_feedforward=2048, dropout=0.5):
        super(Reconstructor, self).__init__()

        self.model_type = 'Transformer'
        self.src_mask = None
        self.pos_encoder = PositionalEncoding(d_model=dim_embedding, dropout=dropout)
        self.transformer = nn.Transformer(d_model=dim_embedding, nhead=nhead, dim_feedforward=dim_feedforward, num_encoder_layers=num_layers, num_decoder_layers=num_layers)
        self.decoder = nn.Linear(dim_embedding, output_dim)
        self.decoder_act_fn = nn.PReLU()

        self.init_weights()

    def init_weights(self):
        initrange = 0.1
        nn.init.zeros_(self.decoder.weight)
        nn.init.uniform_(self.decoder.weight, -initrange, initrange)

    def forward(self, src, tgt):

        pe_src = self.pos_encoder(src.permute(1, 0, 2))  # (seq, batch, features)
        transformer_output = self.transformer_encoder(pe_src)
        decoder_output = self.decoder(transformer_output.permute(1, 0, 2)).squeeze(2)
        decoder_output = self.decoder_act_fn(decoder_output)
        return decoder_output
</code></pre>
<p>My output has a shape of <code>torch.Size([32, 499, 128])</code> where <code>32</code> is batch, <code>499</code> is my sequence length and <code>128</code> is the number of features. But the output has the same values:</p>
<pre><code class="lang-auto">tensor([[[0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         ...,
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017]]],
       grad_fn=&lt;PreluBackward&gt;)
</code></pre>
<p>What am I doing wrong? Thank you so much for any help.</p>

username: utkuumetin
cooked: <p>Hi, i’m facing similar issue. Mine gives me equal probability for every output.</p>

username: shamoons
cooked: <p>Can you share your model? Perhaps we can try to solve this issue together?</p>

username: utkuumetin
cooked: <p>Hello again, i add my model to here but when i decrease learning rate its giving right outputs, now new problem arisen, whatever i give input its predict same value; example:</p>
<pre><code class="lang-auto">Input:
[90, 91, 26, 62, 92, 93, 26, 94, 95, 96]
incumbering soil and washed into immediate and glittering popularity possibly
Masked Input:
[90, 91, 26, 62, 92, 93, 26, 1, 95, 96]
incumbering soil and washed into immediate and unnk popularity possibly
Output:
[90, 91, 26, 62, 92, 93, 26, 33, 95, 96]
incumbering soil and washed into immediate and the popularity possibly
</code></pre>
<p>As you can see like this, it always predict “the” token.</p>
<p>Model:</p>
<pre><code class="lang-auto">class Kemal(nn.Module):
    def __init__(self, src_vocab_size, embedding_size, num_heads, dim_forward, num_encoder_layers, max_len, src_pad_idx, dropout, device):
        super(Kemal, self).__init__()
        
        self.src_word_embedding = nn.Embedding(src_vocab_size, embedding_size)
        self.src_position_embedding = nn.Embedding(max_len, embedding_size)
        
        self.device = device
        
        self.encoder_norm = nn.LayerNorm(embedding_size) 
 
        self.encoder_layer = nn.TransformerEncoderLayer(embedding_size, num_heads, dim_feedforward=dim_forward, dropout=dropout, activation='gelu')
        self.encoder = nn.TransformerEncoder(self.encoder_layer, num_encoder_layers, self.encoder_norm)
        
        self.fc = nn.Linear(embedding_size, src_vocab_size)
        
        self.src_pad_idx = src_pad_idx
        
    def make_src_pad_mask(self, src):
        src_mask = src.transpose(0, 1) == self.src_pad_idx
        return src_mask
        # (N, src_len)
        
    def forward(self, src):
        src_seq_lenght, N = src.shape
        
        src_mask = nn.Transformer.generate_square_subsequent_mask(None, src_seq_lenght).to(self.device)
        
        src_positions = (
            torch.arange(0, src_seq_lenght).unsqueeze(1).to(self.device)
        )
        
        embed_src = (self.src_word_embedding(src) + self.src_position_embedding(src_positions))
        src_padding_mask = self.make_src_pad_mask(src)
        out = self.encoder(embed_src, mask=src_mask, src_key_padding_mask=src_padding_mask)
        out = self.fc(out)
         
        return out
</code></pre>
<p>With CrossEntropyLoss</p>

username: shamoons
cooked: <p>What is the purpose of the <code>make_src_pad_mask</code>?</p>

username: utkuumetin
cooked: <p><strong>key_padding_mask</strong> – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored</p>
<p>from <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html</a></p>

username: shamoons
cooked: <p>I understand that - but why are you using it?</p>

username: utkuumetin
cooked: <p>I’m not using now <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">, i new to PyTorch so i follow tutorials and pad_mask remains from the examples, not in the current version.</p>
<p>Do you have any idea about my problem?</p>

username: shamoons
cooked: <p>How about we do a 30 min Zoom to hash it out?</p>

username: utkuumetin
cooked: <p>Sorry for late update, i solve my problem.</p>
<p>I was give to input to model in (N, seq_len) shape but i need to give (seq_len, N). <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
All the time i generate wrong src_mask and positional embeddings.<br>
And problem about predict wrong tokens is becouse of word frequencies</p>