Understanding BERT from huggingface

Mohammed_Elhamamsy: <p>Hello all,</p>
<p>I’m new to BERT, and I’m trying to wrap my head around its structure.</p>
<pre><code class="lang-auto">from transformers import BertTokenizer, BertModel 
model = BertModel.from_pretrained('bert-base-uncased')

named_params = list(model.named_parameters())
print(f"{len(named_params)} Paramaters \n\n\n")


for p in named_params:
    print("{:&lt;55} {:&gt;12}".format(p[0],str(tuple(p[1].size()))))
</code></pre>
<p>The output of this is:</p>
<blockquote>
<p>199 Paramaters</p>
<p>embeddings.word_embeddings.weight                       (30522, 768)<br>
embeddings.position_embeddings.weight                     (512, 768)<br>
embeddings.token_type_embeddings.weight                     (2, 768)<br>
embeddings.LayerNorm.weight                                   (768,)<br>
embeddings.LayerNorm.bias                                     (768,)<br>
encoder.layer.0.attention.self.query.weight               (768, 768)<br>
encoder.layer.0.attention.self.query.bias                     (768,)<br>
encoder.layer.0.attention.self.key.weight                 (768, 768)<br>
encoder.layer.0.attention.self.key.bias                       (768,)<br>
encoder.layer.0.attention.self.value.weight               (768, 768)<br>
encoder.layer.0.attention.self.value.bias                     (768,)<br>
encoder.layer.0.attention.output.dense.weight             (768, 768)<br>
encoder.layer.0.attention.output.dense.bias                   (768,)<br>
encoder.layer.0.attention.output.LayerNorm.weight             (768,)<br>
encoder.layer.0.attention.output.LayerNorm.bias               (768,)<br>
encoder.layer.0.intermediate.dense.weight                (3072, 768)<br>
encoder.layer.0.intermediate.dense.bias                      (3072,)<br>
encoder.layer.0.output.dense.weight                      (768, 3072)<br>
encoder.layer.0.output.dense.bias                             (768,)<br>
encoder.layer.0.output.LayerNorm.weight                       (768,)<br>
encoder.layer.0.output.LayerNorm.bias                         (768,)<br>
…<br>
encoder.layer.11.attention.self.query.weight              (768, 768)<br>
encoder.layer.11.attention.self.query.bias                    (768,)<br>
encoder.layer.11.attention.self.key.weight                (768, 768)<br>
encoder.layer.11.attention.self.key.bias                      (768,)<br>
encoder.layer.11.attention.self.value.weight              (768, 768)<br>
encoder.layer.11.attention.self.value.bias                    (768,)<br>
encoder.layer.11.attention.output.dense.weight            (768, 768)<br>
encoder.layer.11.attention.output.dense.bias                  (768,)<br>
encoder.layer.11.attention.output.LayerNorm.weight            (768,)<br>
encoder.layer.11.attention.output.LayerNorm.bias              (768,)<br>
encoder.layer.11.intermediate.dense.weight               (3072, 768)<br>
encoder.layer.11.intermediate.dense.bias                     (3072,)<br>
encoder.layer.11.output.dense.weight                     (768, 3072)<br>
encoder.layer.11.output.dense.bias                            (768,)<br>
encoder.layer.11.output.LayerNorm.weight                      (768,)<br>
encoder.layer.11.output.LayerNorm.bias                        (768,)<br>
pooler.dense.weight                                       (768, 768)<br>
pooler.dense.bias                                             (768,)</p>
</blockquote>
<p>I understand that there are 12 layers (encoder stacks) each containing 12 attention heads, however, I can see that for each layer there’s a single Wq matrix, Wv matrix, and Wk matrix.</p>
<p>Shouldn’t there be 12 of them? What I’m getting wrong here?</p>

J_Johnson: <p>The attention head mechanism does not increase the number of parameters. It just divides up the attention calculations into 12 distinct sections during the attention forward pass.</p>
<p>Here is a code example:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/lucidrains/PaLM-rlhf-pytorch/blob/main/palm_rlhf_pytorch/attention.py">
  <header class="source">

      <a href="https://github.com/lucidrains/PaLM-rlhf-pytorch/blob/main/palm_rlhf_pytorch/attention.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/lucidrains/PaLM-rlhf-pytorch/blob/main/palm_rlhf_pytorch/attention.py" target="_blank" rel="noopener nofollow ugc">lucidrains/PaLM-rlhf-pytorch/blob/main/palm_rlhf_pytorch/attention.py</a></h4>


      <pre><code class="lang-py">import torch
from torch import nn, einsum
import torch.nn.functional as F

from collections import namedtuple
from functools import wraps
from packaging import version

from einops import rearrange

# constants

Config = namedtuple('EfficientAttentionConfig', ['enable_flash', 'enable_math', 'enable_mem_efficient'])

# helpers

def exists(val):
    return val is not None

def once(fn):
</code></pre>



  This file has been truncated. <a href="https://github.com/lucidrains/PaLM-rlhf-pytorch/blob/main/palm_rlhf_pytorch/attention.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Mohammed_Elhamamsy: <p>Thank you, I’ll check it out</p>

vdw: <p>This is directly taken from the original <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noopener nofollow ugc">Transformers paper</a>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/193d3258b2ac68790599c05553f7f075b7f12c7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/193d3258b2ac68790599c05553f7f075b7f12c7f" title="Screenshot from 2023-12-10 16-25-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/193d3258b2ac68790599c05553f7f075b7f12c7f_2_690x290.png" alt="Screenshot from 2023-12-10 16-25-46" data-base62-sha1="3Bh36fz0Tlx7T1jiWRuekXzjiRF" width="690" height="290" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/193d3258b2ac68790599c05553f7f075b7f12c7f_2_690x290.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/193d3258b2ac68790599c05553f7f075b7f12c7f_2_1035x435.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/193d3258b2ac68790599c05553f7f075b7f12c7f_2_1380x580.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-12-10 16-25-46</span><span class="informations">1384×583 75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(see the last paragraph)</p>

Mohammed_Elhamamsy: <p>Thanks for sharing Chris,</p>
<p>Yes, I got that after some time. I was just confused at first when I saw one weight matrix per layer.<br>
However, thinking about that, the idea of multiplying matrix A with 8 Wi matrices then concatenating the results is the same as concatenating the 8 Wi matrices (side by side) then performing the multiplication.</p>
<p>Have a good day!</p>

vdw: <p>Yes, having 1 big matrix “containing” the matrices for each head has performance benefits.</p>