Seq2seq attention tutorial understanding

username: aeddung
cooked: <p>Hello, I’m studying attention mechanism via “<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">TRANSLATION WITH A SEQUENCE TO SEQUENCE NETWORK AND ATTENTION</a>”.<br>
But I have a problem to understand the below image. [prev_hidden] indicates decoder hidden state and [input] indicates decoder input, how these two tensors make attention weight?<br>
I thought decoder hidden state and all hidden states of encoder make attention weight in Attention mechanism. So how decoder hidden state and decoder input?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e1833e9657b56f6dd2ae8d09cc3361a00582449.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e1833e9657b56f6dd2ae8d09cc3361a00582449" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e1833e9657b56f6dd2ae8d09cc3361a00582449_2_313x500.png" alt="image" data-base62-sha1="8RjsmbFattLNH6RBpFIJOpUmXQl" width="313" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e1833e9657b56f6dd2ae8d09cc3361a00582449_2_313x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e1833e9657b56f6dd2ae8d09cc3361a00582449.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e1833e9657b56f6dd2ae8d09cc3361a00582449.png 2x" data-dominant-color="EDEFE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">395×629 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        attn_weights = F.softmax(
            self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
        attn_applied = torch.bmm(attn_weights.unsqueeze(0),
                                 encoder_outputs.unsqueeze(0))

        output = torch.cat((embedded[0], attn_applied[0]), 1)
        output = self.attn_combine(output).unsqueeze(0)

        output = F.relu(output)
        output, hidden = self.gru(output, hidden)

        output = F.log_softmax(self.out(output[0]), dim=1)
        return output, hidden, attn_weights

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>

username: vdw
cooked: <p>There are different approaches towards attention. If I remember correctly, this tutorial implements the Bahdanau Attention. Very popular is also Luong Attention, which is arguably simply, at least with respect to coding – I don’t make any claims about effectiveness.</p>
<p>This <a href="https://stackoverflow.com/questions/44238154/what-is-the-difference-between-luong-attention-and-bahdanau-attention" rel="noopener nofollow ugc">Stackoverflow post</a> discusses both approaches and contains good illustrations and additional links to other good sources.</p>

username: Yichong_Shen
cooked: <p>No, that is not true. The main difference between Bahdanau Attention and Luong Attention is that Bahdanau attention is additive attention and Luong Attention is dot-product attention. But their inputs are the same: the encoder output and the decoder hidden state.<br>
so the question still remains, why in this tutorial the author use decoder hidden state and decoder input    as the inputs to compute attention weights?</p>

username: dreidizzle
cooked: <p>So let the decoder’s hidden state be kt and the encoder’s be ht and let T be the length of the encoder input. Let yt be the decoder’s input and xt be the encoder’s inputs and zt be the decoder’s targets.</p>
<ul>
<li>
<p>Loung Attention computes attention between k_t and h_1, h_2, …, h_{T} and gives various formulas.</p>
</li>
<li>
<p>Bahdanau gets the attention between k_{t-1} and h_1, …, h_{T} and uses some tanh.</p>
</li>
</ul>
<p>The point is you are time step t and want to predict zt. What do you have access to? You have k_{t-1}, y_{t} and h_1, …, h_{T} and all of the encoder’s inputs (x_1, …, x_T) and you’d like to predict z_{t}.</p>
<p>To do this, you form some vector c_{t} via some sort of weighted average of the vectors h_{s}, the encoder hidden states. For example,</p>
<ul>
<li>
<p>c_t = sum_{s=1}^{T}{h_s * F(k_{t-1}, h_s)}</p>
</li>
<li>
<p>Or, c_t = sum_{s=1}^{Tx}{h_s * F(k_{t-1}, x_t, h_s)}</p>
</li>
<li>
<p>Or, k_t = G(k_{t-1}, y_t) and then c_t = sum_{s=1}^{T}{h_s * F(k_t, h_s)}</p>
</li>
</ul>
<p>G can be a GRU cell above and F is some sort of normalized scoring metric. If you consider F(k_t, h_s) you can compute an inner product dot(k_t, h_s) for each s in {1,…, T} and then normalize by softmax to get probabilities, for example.</p>
<p>The format of F can be anything you like. Also, the ARGUMENTS to F can be anything you like <em><strong>so long as you don’t look ahead</strong></em>: the part associated with the decoder in computing F can be (k_t), or (k_{t-1}, y_t), or (k_{t-1}, k_{t-2}, y_{t-2}), whatever, you know all these at time step t and you are not using forward information. This tutorial uses (k_t, y_t), and that’s fine. I guess you want to keep all the information when you cross h_s with (k_t, y_t). You could have also crossed h_s with k_t since k_t has y_t’s information since k_t = GRUCell(k_{t-1}, y_t), it is a function of k_{t-1} and y_t, but this is up to you. You have options to do as you like but you have the above restriction with not looking ahead …</p>