How to apply Shared embedding

Aiman_Mutasem-bellh: <p>Dear all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m working on a grammatical error correction (GEC) task based on neural machine translation (NMT). The only difference between GEC and NMT is the shared embedding.</p>
<p>NMT embedding:</p>
<blockquote>
<p>SRC = Field(tokenize= tokenizer, init_token=‘’, eos_token=‘’,  batch_first=True)<br>
TRG = Field(tokenize= tokenizer, init_token=‘’, eos_token=‘’,  batch_first=True)</p>
<p>train_data, valid_data = TabularDataset.splits(path=‘…/data/’,train=‘train.csv’,<br>
validation=‘valid.csv’ , format=‘csv’,<br>
fields=[(‘src’, SRC), (‘trg’, TRG)], skip_header=True)</p>
</blockquote>
<p>My implementation of shared embedding is like this:</p>
<blockquote>
<p>TRG = Field(tokenize= tokenizer, init_token=‘’, eos_token=‘’,  batch_first=True)</p>
<p>train_data, valid_data = TabularDataset.splits(path=‘…/data/’,train=‘train.csv’,<br>
validation=‘valid.csv’ , format=‘csv’,<br>
fields=[(‘src’, TRG), (‘trg’, TRG)], skip_header=True)</p>
</blockquote>
<p>But the results are not good, what is the optimal implementation of shared embedding in Pytorch?</p>
<p>Kind regards,<br>
Aiman Solyman</p>

Aiman_Mutasem-bellh: <p>Hope to get a little support <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Unity05: <p>Hi Aiman,</p>
<p>firstly, could you provide a link to a paper where you’re trying to implement the GEC model from (if that’s the case) as I’m not familiar with GEC.<br>
However, aren’t your two field objects in the NMT code block redundant? Therefore the second implementation would be equal to the first one (assuming that ‘train.csv’ and ‘valid.csv’ have not changed).<br>
As already mentioned, I’m not familiar with GEC models, but why don’t you just use an embedding layer?</p>
<p>Regards, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Unity05</p>

Aiman_Mutasem-bellh: <aside class="quote no-group" data-username="Unity05" data-post="3" data-topic="120756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unity05/48/20944_2.png" class="avatar"> Unity05:</div>
<blockquote>
<p>but why don’t you just use an embedding layer</p>
</blockquote>
</aside>
<p>Thank you sir for your kind feedback.</p>
<p>I’m using this <a href="https://github.com/dksifoua/Neural-Machine-Translation/blob/master/notebooks/5%20-%20SeqToSeq%20Model%20with%20Transformer.ipynb" rel="noopener nofollow ugc">example</a>, and my implementation is like :</p>
<pre><code class="lang-auto">class InputEmbeddingAndPositionalEncodingLayer(nn.Module):

    def __init__(self, vocab_size, max_len, d_model, dropout):
        super(InputEmbeddingAndPositionalEncodingLayer, self).__init__()
        self.vocab_size = vocab_size
        self.max_len = max_len
        self.d_model = d_model
        self.dropout = nn.Dropout(p=dropout)
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_encoding = PositionalEncodingLayer(d_model=d_model, max_len=max_len)

    def forward(self, sequences):
        """
        :param Tensor[batch_size, seq_len] sequences
        :return Tensor[batch_size, seq_len, d_model]
        """
        token_embedded = self.token_embedding(sequences) # [batch_size, seq_len, d_model]
        position_encoded = self.position_encoding(sequences) # [batch_size, seq_len, d_model]
        return self.dropout(token_embedded) + position_encoded # [batch_size, seq_len, d_model]

</code></pre>
<pre><code class="lang-auto">shared_embeddings = InputEmbeddingAndPositionalEncodingLayer(
    vocab_size=len(TRG.vocab),
    max_len=MAX_LEN,
    d_model=D_MODEL,
    dropout=DROPOUT
)
</code></pre>
<pre><code class="lang-auto">transformer = Transformer(
    encoder=EncoderLayer(
        in_emb_pos_enc_layer=shared_embeddings,
        d_model=D_MODEL,
        n_heads=N_HEADS,
        hidden_size=HIDDEN_SIZE,
        dropout=DROPOUT,
        n_layers=N_LAYERS
    ),
    decoder=DecoderLayer(
        in_emb_pos_enc_layer=shared_embeddings,
        vocab_size=len(TRG.vocab),
        d_model=D_MODEL,
        n_heads=N_HEADS,
        hidden_size=HIDDEN_SIZE,
        dropout=DROPOUT,
        n_layers=N_LAYERS
    ),
    src_pad_index=TRG.vocab.stoi[TRG.pad_token],
    dest_pad_index=TRG.vocab.stoi[TRG.pad_token]
).to(DEVICE)
</code></pre>
<p>The result still not good as same as the implementation without shared embedding, could you give me an example?</p>
<p>One more thing, I used shared vocabulary size so no worries about the ‘train.csv’ and ‘valid.csv’ changes.</p>

Unity05: <p>The code you’ve sent looks fine to me.</p>
<blockquote>
<p>The result still not good as same as the implementation without shared embedding, could you give me an example?</p>
</blockquote>
<p>You’ve got better results on your GEC task with the same implementation but seperate embeddings? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

Aiman_Mutasem-bellh: <aside class="quote no-group" data-username="Unity05" data-post="5" data-topic="120756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unity05/48/20944_2.png" class="avatar"> Unity05:</div>
<blockquote>
<p>You’ve got better results on your GEC task with the same implementation but seperate embeddings?</p>
</blockquote>
</aside>
<p>Yes sir, as bellow example:</p>
<pre><code class="lang-auto">SRC = Field(tokenize= tokenizer, init_token=’’, eos_token=’’, batch_first=True)
TRG = Field(tokenize= tokenizer, init_token=’’, eos_token=’’, batch_first=True)
</code></pre>
<p>But all the papers in GEC mentioned that Shared embedding is the only difference between GEC and NMT, which is the source and target are in the same language. So, I thought there is an issue with my implementation.</p>

Unity05: <p>Well, the thing that comes up in my mind first is that your using the same identical layer twice. Therefore there might be problems with the gradients as they are accumulated I assume.</p>

Aiman_Mutasem-bellh: <aside class="quote no-group" data-username="Unity05" data-post="7" data-topic="120756">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unity05/48/20944_2.png" class="avatar"> Unity05:</div>
<blockquote>
<p>ey are accumulated I assum</p>
</blockquote>
</aside>
<p>Correct, could you help ?</p>

Unity05: <p>If this really causes the problem, my first approach would be to try modifying the respective gradients ‘manually’ (i.e. the embedding gradients taking mean of both gradient by hooking the grad and sub the embedding weights ‘manually’).</p>