Multiheaded attention - What are the query, key and values?

username: Mole_m7b5
cooked: <p>I’m looking at using attention for the first time, and looking at the documentation for <code>nn.MultiheadAttention()</code> I’m not sure what data I’m meant to be using as input for the Q,K, and V arguments.</p>

username: my3bikaht
cooked: <p>IMO, this question is barely related to pytorch since it’s a transformer architecture and MultiHeadAttention is an inner element for the Encoder and Decoder layers.</p>
<p>Anyway, here’s an explanation from architecture standpoint:</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/211707/sean" target="_blank" rel="noopener nofollow ugc">
    <img alt="Sean" src="https://www.gravatar.com/avatar/380b6f3a74c842769369d618fa27b451?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms" target="_blank" rel="noopener nofollow ugc">What exactly are keys, queries, and values in attention mechanisms?</a>
</h4>

<div class="tags">
  <strong>neural-networks, natural-language, attention, machine-translation</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/211707/sean" target="_blank" rel="noopener nofollow ugc">
    Sean
  </a>
  on <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms" target="_blank" rel="noopener nofollow ugc">09:00AM - 13 Aug 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Mole_m7b5
cooked: <p>Thanks very much for your response and the link.</p>
<p>While it’s helped my conceptual understanding, I’m still not sure exactly what I should be using as inputs for those arguments. Hence why I posted to the PyTorch forum as it’s the framework I’m using.</p>
<p>If the following is true (as per one of the answers in the link):</p>
<pre><code class="lang-auto">Query = I x W(Q)

Key = I x W(K)

Value = I x W(V)
</code></pre>
<blockquote>
<p>where I is the input (encoder) state vector, and W(Q), W(K), and W(V) are the corresponding matrices to transform the I vector into the Query, Key, Value vectors.</p>
</blockquote>
<p>Assuming <code>I</code> is just the input into the attention block , how are W(Q), W(K), and W(V) generated or are these linear layers as the literature states they are “learnable” matrices?</p>

username: Mole_m7b5
cooked: <p>This seems to have answered my question.</p>
<p><a href="https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial6/Transformers_and_MHAttention.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial6/Transformers_and_MHAttention.html</a></p>

username: my3bikaht
cooked: <p>I find the original paper better explaining why:</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1706.03762.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1706.03762.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1706.03762.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1706.03762.pdf" target="_blank" rel="noopener nofollow ugc">1706.03762.pdf</a></h3>

  <p class="filesize">2.10 MB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>