Albert Model Display and Copying weights inside Albert

Ritam_9: <p>0</p>
<p>I wanted to try something.</p>
<p>There are 2 experiments, one on Bert an one on Albert. The task is, Train Bert upto Layers 5 , keeping the other 7 fixed, and then during test time, copy the weights of the 5th layer onto layers 6-12. I was successfully able to do this, as Bert has seperate parameters, and I can manipulate them seperately.</p>
<p>Whereas in Albert, the Architecture is such that all the weights are shared from layers 1-12, so, although I can train them till layer 5, they aren’t accessible seperately, so, I am unable to copy over the weights onto further layers.</p>
<p>Any suggestions how I might go about this?</p>
<p>I tried printing the entire model, for both Bert and Albert.</p>
<pre><code class="lang-auto">&gt; (embeddings): BertEmbeddings(
&gt;       (word_embeddings): Embedding(30522, 768, padding_idx=0)
&gt;       (position_embeddings): Embedding(512, 768)
&gt;       (token_type_embeddings): Embedding(2, 768)
&gt;       (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;       (dropout): Dropout(p=0.1, inplace=False)
&gt;     )
&gt; (encoder): BertEncoder(
&gt;       (layer): ModuleList(
&gt;         (0): BertLayer(
&gt;           (attention): BertAttention(
&gt;             (self): BertSelfAttention(
&gt;               (query): Linear(in_features=768, out_features=768, bias=True)
&gt;               (key): Linear(in_features=768, out_features=768, bias=True)
&gt;               (value): Linear(in_features=768, out_features=768, bias=True)
&gt;               (dropout): Dropout(p=0.1, inplace=False)
&gt;             )
&gt;             (output): BertSelfOutput(
&gt;               (dense): Linear(in_features=768, out_features=768, bias=True)
&gt;               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;               (dropout): Dropout(p=0.1, inplace=False)
&gt;             )
&gt;           )
&gt;           (intermediate): BertIntermediate(
&gt;             (dense): Linear(in_features=768, out_features=3072, bias=True)
&gt;           )
&gt;           (output): BertOutput(
&gt;             (dense): Linear(in_features=3072, out_features=768, bias=True)
&gt;             (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;             (dropout): Dropout(p=0.1, inplace=False)
&gt;           )
&gt;         )
&gt; (1): BertLayer(
&gt;           (attention): BertAttention(
&gt;             (self): BertSelfAttention(
&gt;               (query): Linear(in_features=768, out_features=768, bias=True)
&gt;               (key): Linear(in_features=768, out_features=768, bias=True)
&gt;               (value): Linear(in_features=768, out_features=768, bias=True)
&gt;               (dropout): Dropout(p=0.1, inplace=False)
&gt;             )
&gt;             (output): BertSelfOutput(
&gt;               (dense): Linear(in_features=768, out_features=768, bias=True)
&gt;               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;               (dropout): Dropout(p=0.1, inplace=False)
&gt;             )
&gt;           )
&gt;           (intermediate): BertIntermediate(
&gt;             (dense): Linear(in_features=768, out_features=3072, bias=True)
&gt;           )
&gt;           (output): BertOutput(
&gt;             (dense): Linear(in_features=3072, out_features=768, bias=True)
&gt;             (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;             (dropout): Dropout(p=0.1, inplace=False)
&gt;           )
&gt;         )
</code></pre>
<p>And so on, for layers uptil 11. (They are seperately accessible).</p>
<p>For Albert:</p>
<pre><code class="lang-auto">&gt; Model(
&gt;   (albert): AlbertModel(
&gt;     (embeddings): AlbertEmbeddings(
&gt;       (word_embeddings): Embedding(30000, 128, padding_idx=0)
&gt;       (position_embeddings): Embedding(512, 128)
&gt;       (token_type_embeddings): Embedding(2, 128)
&gt;       (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)
&gt;       (dropout): Dropout(p=0, inplace=False)
&gt;     )
&gt; (encoder): AlbertTransformer(
&gt;       (embedding_hidden_mapping_in): Linear(in_features=128, out_features=768, bias=True)
&gt;       (albert_layer_groups): ModuleList(
&gt;         (0): AlbertLayerGroup(
&gt;           (albert_layers): ModuleList(
&gt;             (0): AlbertLayer(
&gt;               (full_layer_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;               (attention): AlbertAttention(
&gt;                 (query): Linear(in_features=768, out_features=768, bias=True)
&gt;                 (key): Linear(in_features=768, out_features=768, bias=True)
&gt;                 (value): Linear(in_features=768, out_features=768, bias=True)
&gt;                 (dropout): Dropout(p=0, inplace=False)
&gt;                 (dense): Linear(in_features=768, out_features=768, bias=True)
&gt;                 (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
&gt;               )
&gt;               (ffn): Linear(in_features=768, out_features=3072, bias=True)
&gt;               (ffn_output): Linear(in_features=3072, out_features=768, bias=True)
&gt;             )
&gt;           )
&gt;         )
&gt;       )
&gt;     )
</code></pre>
<p>The weights can’t be copied directly, as they are the same weights. (I want to have weights trained only till layer 5, and then copy layer 5 weights till layer 12, without training it explicitly. How do I go about this?</p>