Is it true that Bahdanau's attention mechanism is not Global like Luong's according to Pytorch's tutorial?

pinocchio: <p>I was reading the pytorch tutorial on <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="noopener nofollow ugc">a chatbot task and attention</a> where it said:</p>
<blockquote>
<p>Luong et al. improved upon Bahdanau et al.’s groundwork by creating “Global attention”. The key difference is that with “Global attention”, we consider all of the encoder’s hidden states, as opposed to Bahdanau et al.’s “Local attention”, which only considers the encoder’s hidden state from the current time step.</p>
</blockquote>
<p>I think that description is plain wrong (or at least confusing). As far as I understand attention in general is the idea that we use a Neural network that depends on the source (or endoder state) and the current target (or decoder) to compute a weight to determine the importance of the current encoder/source in determining the traget/decoder output. Then we do a weighted sum over all context vectors to determine the importance $c_t = \sum^{Tx}<em>{s=1} \alpha</em>{s,t} \bar h_s$ where $\alpha_{s,t}$ is the attention from source/encoder $s$ and $\bar h_s$ is the hidden state from the encoder/source at step $s$.</p>
<p>What is confusing is that the Pytorch tutorials claims that Bahdanau’s work is NOT global. I don’t understand why they say that about Bahdanau’s attention mechanism since to me the following is true:</p>
<blockquote>
<p>uses all encoders/source states to compute the context vectors via $c_t = \sum^{Tx}<em>{s=1} \alpha</em>{s,t} \bar h_s$, especially because $\alpha_{s,t}$ is a function of <strong>each</strong> source/encoder states. So of course it uses all encoder/source states.</p>
</blockquote>
<p>Is there something that I am missing? What is the tutorial reffering to?</p>
<hr>
<h2><a name="appendix-1" class="anchor" href="#appendix-1"></a>Appendix</h2>
<p>Perhaps if I go through the equations here carefully I can outline why I think what I do:</p>
<p><em><strong>Luong’s:</strong></em></p>
<p>Attention is computed as follows:</p>
<p>$$ \alpha_(t) = \alpha_{s,t} = align(h_t, \bar h_s) = \frac{exp( score(h_t, \bar h_s) ) }{\sum^{T_x}<em>{s’=1} exp( score(h_t, \bar h</em>{s’}) )}$$</p>
<p>and the context vector must be:</p>
<p>$$ c_t = \sum^{Tx}<em>{s=1} \alpha</em>{s,t} \bar h_s$$</p>
<p>cuz in the paper it says:</p>
<blockquote>
<p>Given the alignment vector as weights, the context<br>
vector $c_t$ is computed as the weighted average over<br>
all the source hidden states.</p>
</blockquote>
<p><em><strong>Bahdanau’s:</strong></em></p>
<p>First I will unify their notation.</p>
<ul>
<li>target/decoder hidden state $h_t = s_t$</li>
<li>encoder/source hidden state $\bar h)s = h_s $</li>
<li>score $score(h_t, \bar h_s ) = e_{t,s}$</li>
<li>aligment $\alpha_{s,t} = \alpha_t(s)$</li>
</ul>
<p>but the key is that they both use the same equation to compute context vectors:</p>
<p>$$ c_t = \sum^{Tx}<em>{s=1} \alpha</em>{s,t} h_s = \sum^{Tx}_{s=1} \alpha_t(s) \bar h_s $$</p>
<p>Of course there are difference in how the compute hidden states and scores but <strong>they BOTH are global attention mechanism</strong>. Or am I missing something?</p>
<hr>
<p>Crossposted:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/unanswered/Is-it-true-that-Bahdanaus-attention-mechanism-is-not-global-like-Luongs-1?ch=10&amp;share=1e3b740c&amp;srid=ovS7">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="" height="">

      <a href="https://www.quora.com/unanswered/Is-it-true-that-Bahdanaus-attention-mechanism-is-not-global-like-Luongs-1?ch=10&amp;share=1e3b740c&amp;srid=ovS7" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://qph.cf2.quoracdn.net/main-custom-t-980-600x315-sqxxospfciejqkjfytssmfbdcvzbbxla.jpeg" class="thumbnail">

<h3><a href="https://www.quora.com/unanswered/Is-it-true-that-Bahdanaus-attention-mechanism-is-not-global-like-Luongs-1?ch=10&amp;share=1e3b740c&amp;srid=ovS7" target="_blank" rel="noopener nofollow ugc">Is it true that Bahdanau's attention mechanism is not global like Luong's?</a></h3>

  <p>1 person wants answers to this question. Be the first to answer.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/413805/is-it-true-that-bahdanaus-attention-mechanism-is-not-global-like-luongs">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/413805/is-it-true-that-bahdanaus-attention-mechanism-is-not-global-like-luongs" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/28986/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/413805/is-it-true-that-bahdanaus-attention-mechanism-is-not-global-like-luongs" target="_blank" rel="noopener nofollow ugc">Is it true that Bahdanau's attention mechanism is not Global like Luong's?</a>
</h4>

<div class="tags">
  <strong>machine-learning, neural-networks, natural-language, recurrent-neural-network</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/28986/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stats.stackexchange.com/questions/413805/is-it-true-that-bahdanaus-attention-mechanism-is-not-global-like-luongs" target="_blank" rel="noopener nofollow ugc">05:12PM - 19 Jun 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>