Confusing about the dimension of Seq2Seq model

QUANG_HUY_CHU: <p>I am new to Seq2Seq and hope to find a proper guildances, advices.</p>
<blockquote>
<p>I am doing a Project from an online course so I can not give the material but I got my Project notebook on <a href="https://github.com/CQHofsns/LSTM-based-Seq2Seq-QA-bots-Udacity/blob/main/(Starter%20Code)%20Chatbot%20With%20LSTM%20and%20Pretrained%20Embeddings.ipynb" rel="noopener nofollow ugc">Github</a></p>
</blockquote>
<p>I want to ask about my understanding about the architecture as well as the data dimension after each layer. Suppose I have a Seq2Seq model as below:</p>
<blockquote>
<pre><code> Seq2Seq(
  (encoder): Encoder(
    (embedding): Embedding(5678, 512)
    (lstm): LSTM(512, 512, batch_first=True)
  )
  (decoder): Decoder(
    (embedding): Embedding(4297, 512)
    (lstm): LSTM(512, 512, batch_first=True)
    (fc): Linear(in_features=512, out_features=4297, bias=True)
    (dropout): Dropout(p=0.2, inplace=False)
    (softmax): LogSoftmax(dim=1)
  )
)
</code></pre>
</blockquote>
<p>Where 5678 is source_vocab size, 512 is desired embedding size, 4297 is target_vocab size. You can check my Encoder, Decoder, Seq2Seq class as below:</p>
<pre><code class="lang-auto">device= torch.device("cuda" if torch.cuda.is_available() else "cpu")
#Force cpu
#device= "cpu"
print(device)

class Encoder(nn.Module):
    
    def __init__(self, input_size, hidden_size):
        
        super(Encoder, self).__init__()
        
        self.input_size= input_size
        self.hidden_size= hidden_size
        
        self.embedding= nn.Embedding(self.input_size, self.hidden_size)
        self.lstm= nn.LSTM(self.hidden_size, self.hidden_size, batch_first= True)

    def forward(self, i):
        print(i.size())
        embedded= self.embedding(i)
        print(embedded.size())
        o,(h,c)= self.lstm(embedded)
        
        return h, c
    

class Decoder(nn.Module):
      
    def __init__(self, hidden_size, output_size):
        
        super(Decoder, self).__init__()
        
        self.hidden_size= hidden_size
        self.output_size= output_size

        self.embedding= nn.Embedding(self.output_size, self.hidden_size)
        self.lstm= nn.LSTM(self.hidden_size, self.hidden_size, batch_first= True)
        self.fc = nn.Linear(self.hidden_size, self.output_size)
        self.dropout= nn.Dropout(0.2)
        self.softmax= nn.LogSoftmax(dim= 1)
        
    def forward(self, i, h, c):
        embedded= self.embedding(i)
        o,(h,c)= self.lstm(embedded, (h, c))
        o= self.fc(o[0])
        o= self.dropout(o)
        o= self.softmax(o)
        
        return o, h, c
        
        

class Seq2Seq(nn.Module):
    
    def __init__(self, encoder_input_size, encoder_hidden_size, decoder_hidden_size, decoder_output_size):
        
        super(Seq2Seq, self).__init__()
        
        self.input_size= encoder_input_size
        self.hidden_size= encoder_hidden_size
        self.output_size= decoder_output_size
        
        self.encoder= Encoder(self.input_size, self.hidden_size)
        self.decoder= Decoder(self.hidden_size, self.output_size)
    
    def forward(self, src, trg, teacher_forcing_ratio = 0.5):
        
        output_seq= []
        
        encoder_hidden, encoder_cell= self.encoder(src)
        
        decoder_hidden= encoder_hidden
        decoder_cell= encoder_cell

        decoder_input= torch.Tensor([[target_vocab.token_to_index("&lt;SOS&gt;")]]).long().to(device)
        
        for time_step in range(trg.size(0)):
            output_token, decoder_hidden, decoder_cell= self.decoder(
                decoder_input,
                decoder_hidden,
                decoder_cell
            )
            output_seq.append(output_token)
            
            if self.training:
                if random.random() &lt; teacher_forcing_ratio:
                    decoder_input= trg[time_step]
            else:
                _, top_index= output_token.data.topk(1)
                decoder_input= top_index.squeeze().detach()
        
        return output_seq
</code></pre>
<p>My quesions is that the Input of Encoder is the soure vocabualry size, which mean that <strong>each token in the input sequence should be converted into one-hot vector before parsing to the Encoder before hand</strong> (for example, should a batch has a dimension of (batch_size, seq_len, vocab_size) instead of (batch_size, seq_len)?</p>
<p>I search others notebook and saw that they just parse a batch of (batch_size, seq_len) into the Encoder and I got confused.</p>
<p>Any help is appreciated.</p>
<p>I have tried pass (batch_size, seq_len, vocab_size) and the Embedding layer output dimension is (batch_size, seq_len, vocab_size, embedding_dim), which make me more confused, isn’t it should be (batch_size, seq_len, embedding_dim)</p>

vdw: <p>Yes, the input for the encoder is <code>(batch_size, seq_len)</code>.</p>
<p>Each sequence in a batch is a list/array of integers reflecting the indices of the tokens in the vocabulary. For example, a match might look like this:</p>
<pre><code class="lang-auto">[
    [12, 40, 8, 105, 86, 6],
    [35, 105, 86, 35, 40, 6]
]
</code></pre>
<p>Representing the 2 sentences “i like to watch movies .” and “you watch movies you like .” This means your vocabulary provides a mapping like</p>
<pre><code class="lang-auto">{6: ".", 8: "to", 12: "i", 35: "you", 40: "like", ...}
</code></pre>
<p>There is no need to convert the tokens / token indices into one-hot vector. This is what the <code>nn.Embedding</code> layer is for. To clarify, this layer does not create one-hot vectors either, but accepts individual token indices as input.</p>
<p>You only need to appreciate that, say, token index <code>40</code> and a vocabulary size of <code>5678</code> carries the same information as a one-hot vector of size <code>5678</code> with a <code>1</code> at index 40. You can also check out this <a href="https://discuss.pytorch.org/t/how-does-nn-embedding-work/88518/3">post</a>.</p>

QUANG_HUY_CHU: <p>Thank your for your reply,</p>
<p>The reason make I confuse is that when implement this layer, we do:<br>
<code>nn.Embedding(vocab_size, embedding_dim)</code></p>
<p>But when we parse input to this layer we input (batch_size, seq_len), there is no <strong>vocab_size</strong> here. But after read your answer and the post you provide, I guess that the reason behind giving the Embedding layer vocab_size is to let it know the maximum index range so that it can create embedding vectors that cover all the index?</p>

vdw: <aside class="quote no-group" data-username="QUANG_HUY_CHU" data-post="3" data-topic="185235">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/quang_huy_chu/48/62831_2.png" class="avatar"> QUANG_HUY_CHU:</div>
<blockquote>
<p>But when we parse input to this layer we input (batch_size, seq_len), there is no <strong>vocab_size</strong> here.</p>
</blockquote>
</aside>
<p>It’s kind of there but only implicitly :). As mentioned earlier, each sequence in a batch is a list of token indices. This indices are between <code>0</code> and <code>vocab_size-1</code>. This is why <code>nn.Embedding</code> needs <code>vocab_size</code> as input parameter, to know what values to expect.</p>
<p>Instead of <code>nn.Embedding</code> you could also use <code>nn.Linear</code> like</p>
<pre><code class="lang-auto">self.embedding = nn.Linear(self.input_size, self.hidden_size)
</code></pre>
<p>In this case, yes, this would require an input of <code>(batch_size, seq_len, vocab_size)</code>, with each token represented by a one-hot vector. The result would be exactly the same! <code>nn.Embedding</code> “only” simplifies this, making it much more convenient in practice for many reasons.</p>
<p>I’ve actually prepared a detailed <a href="http://185.170.113.47/data/downloads/nn-embedding-explained.ipynb" rel="noopener nofollow ugc">Jupyter notebook</a> for my students explaining this with an elaborate example. I hope this makes it fully clear. It’s not quite polished and thus not on GitHub yet.</p>

QUANG_HUY_CHU: <p>Thanks for your reply, the turtorial I am currently studying from does not explain this !! Got it now.</p>