AttentionDecoderRNN without MAX_LENGTH

alvations: <p>From the PyTorch Seq2Seq tutorial, <a href="http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder" rel="nofollow noopener">http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder</a></p>
<p>We see that the attention mechanism is heavily reliant on the <code>MAX_LENGTH</code> parameter to determine the output dimensions of the <code>attn -&gt; attn_softmax -&gt; attn_weights</code>, i.e.</p>
<pre><code class="lang-auto">class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)
</code></pre>
<p>More specifically</p>
<pre><code class="lang-auto">self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
</code></pre>
<p>I understand that the <code>MAX_LENGTH</code> variable is the mechanism to reduce the no. of parameters that needs to be trained in the <code>AttentionDecoderRNN</code>.</p>
<p>If we don’t have a <code>MAX_LENGTH</code> pre-determined. What values should we initialize the <code>attn</code> layer with?</p>
<p>Would it be the <code>output_size</code>? If so, then that’ll be learning the attention with respect to the full vocabulary in the target language. Isn’t that the real intention of the Bahdanau (2015) attention paper?</p>

Maxisawesome: <p>My understanding is that MAX_LENGTH is used to initialize the size of the attention matrix. You make it as long as the longest sequence you have so that you have room to put any sequence in. So you have a max_len x max_len matrix, and if your input is less than that, it’ll simply have 0’s in the extra space, and it’ll get ignored. You need a hard number because it’s part of the architecture, kinda like how you use hard numbers in CNN’s. Does that help?</p>

pinocchio: <p>The tutorial is ridiculously stupid. Ignore it. Read this:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stats.stackexchange.com/questions/411736/why-do-attention-models-need-to-choose-a-maximum-sentence-length" target="_blank" rel="nofollow noopener">stats.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/37632/pinocchio" target="_blank" rel="nofollow noopener">
    <img alt="Pinocchio" src="https://www.gravatar.com/avatar/ba92064b084638cb903cbd271120be64?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stats.stackexchange.com/questions/411736/why-do-attention-models-need-to-choose-a-maximum-sentence-length" target="_blank" rel="nofollow noopener">Why do attention models need to choose a maximum sentence length?</a>
</h4>

<div class="tags">
  <strong>neural-networks, natural-language, rnn, attention</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/37632/pinocchio" target="_blank" rel="nofollow noopener">
    Pinocchio
  </a>
  on <a href="https://stats.stackexchange.com/questions/411736/why-do-attention-models-need-to-choose-a-maximum-sentence-length" target="_blank" rel="nofollow noopener">01:02AM - 06 Jun 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>but the point is that their attention layer is fixed matrix. From 2h to max_len as you noted. Note that the attention weights are computed as follow:</p>
<pre><code>exp( target_embedding[t] * self.att ) # (1, self.max_length)
</code></pre>
<p>so therefore we don’t have the attention be dependent on the actual representation of the source (since it is NOT using the embeddings of the ecodning source vectors). It’s just an attention based on position, which is stupid and makes no sense. It took no effort to use the encoders as input. Say something like:</p>
<p>exp( target_embeddings[t] * enconder_outputs )</p>
<p>or</p>
<pre><code>attn_weights = F.softmax( embedded[0]*encoder_outputs )
</code></pre>
<p>something like that.</p>

pinocchio: <aside class="quote no-group" data-username="alvations" data-post="1" data-topic="13473">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alvations/48/1960_2.png" class="avatar"> alvations:</div>
<blockquote>
<p>If we don’t have a <code>MAX_LENGTH</code> pre-determined. What values should we initialize the <code>attn</code> layer with?</p>
</blockquote>
</aside>
<p>Do not initialize it to be fixed length. Don’t have a fixed matrix for <code>self.att</code>. In fact, <code>self.att</code> should NOT exist at all really. Remove that line of code and use the encoder source as the input. I’d recommend Luong’s attention so something like:</p>
<pre><code>alpha_{s,t} = align( d_t, e_s ) ~ exp( &lt; d_t, e_s &gt; )
</code></pre>
<p>so have the encoders and decoders determine the weights. The whole point of attention is that the actual semantics of the encoding vector and target vector to determine the output of the RNN. Plus, attention makes things variable length using that. It’s ridiculous how the tutorial destroys all the advantages of RNNs in one go.</p>

pinocchio: <p>For the sake of the “speed” tutorial, one can STILL have the quick speed and still use “max_len” be the maximum sequence length and at the same time be variable length and use the embeddings of the encoder. Using a fixed matrix is just plain lazy and confusing. The max_len isn’t the problem, the fact they used a fixed matrix is.</p>