How to get embeddings from two embedding layers

Sirui_Li: <p>I would like to freeze only one line of the embedding layer so that the weight of this line would not be updated after each epoch. Some people suggested using two separate embedding layers: one for trainable embeddings and another for the freezing embedding. But I am not sure how to get embeddings from two layers and concatenate them in a fast way. Any help would be apprieciated.</p>
<pre><code class="lang-auto">class myModel(nn.Module):
    def __init__(self, config):
        self.config = config
        self.trainable_rel_embeddings = nn.Embedding(self.config.relTotal, self.config.dim) # e.g. 200x100, where 200 is the number of trainable elements, 100 is the dimension
        self.freeze_rel_embeddings = nn.Embedding(1, self.config.dim)
        self.freeze_rel_embeddings.weight.data = self.config.freeze_init_rel_embs
        self.freeze_rel_embeddings.weight.requires_grad = False
        self.trainable_rel_embeddings.weight.data = self.config.init_rel_embs
    def forward(batch):
        '''
        Batch are indexes, e.g. there is one batch with 5 groups:
        tensor([[  3, 200],                                  
        [  3, 200],
        [  3, 200],
        [188, 2],
        [188, 200]])
        '''
       #How could I get embedings whose indexes are less than 200 from self.trainable_rel_embeddings and get embeddings whose index is 200 from self.freeze_rel_embeddings?

</code></pre>

tom: <p>Given that the Embedding’s forward is literally just</p>
<pre><code class="lang-auto">return F.embedding(
            input, self.weight, self.padding_idx, self.max_norm,
            self.norm_type, self.scale_grad_by_freq, self.sparse)
</code></pre>
<p>you could do this in your forward yourself and pass in the <code>torch.cat</code>ed weights.<br>
You might even just store the weight as a parameter (or even a buffer) instead of having its own Embedding layer. (Or subclass Embedding to EmbedddingWithFrozen that does all this under the hood.)</p>
<p>There is a slight drawback that you’ll be copying the embedding quite a bit. With more trickery, you could look into reducing that by doing the embedding in two stages, but it is considerably less straightforward.</p>
<p>Some unsolicited style advice:</p>
<aside class="quote no-group" data-username="Sirui_Li" data-post="1" data-topic="130598">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sirui_li/48/13015_2.png" class="avatar"> Sirui_Li:</div>
<blockquote>
<pre><code class="lang-auto">        self.freeze_rel_embeddings.weight.data = self.config.freeze_init_rel_embs
        self.freeze_rel_embeddings.weight.requires_grad = False
        self.trainable_rel_embeddings.weight.data = self.config.init_rel_embs
</code></pre>
</blockquote>
</aside>
<p>I would probably write this as</p>
<pre><code class="lang-auto">        with torch.no_grad():
            self.freeze_rel_embeddings.weight.copy_(self.config.freeze_init_rel_embs)
        self.freeze_rel_embeddings.weight.requires_grad_(False)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>