Bahdanau's Attention

divyesh_rajpura: <p><a href="https://arxiv.org/pdf/1406.1078.pdf" rel="nofollow noopener">Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation</a> proposed modified version of RNN which is used in <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="nofollow noopener">Neural Machine Translation by Jointly Learning to Align and Translate</a>. In modification of RNN they are changing hidden layer computation of RNN. Many people have implemented this in PyTorch but I seen all of them have just used base RNN models in both Encoder and Decoder which actually needs internal modification. So <strong>is there any way we can change hidden layer computation of RNN or LSTM? or If not, how can I build it from scratch in PyTorch?</strong></p>

jerinphilip: <aside class="quote no-group" data-username="divyesh_rajpura" data-post="1" data-topic="48277">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/divyesh_rajpura/48/12485_2.png" class="avatar"> divyesh_rajpura:</div>
<blockquote>
<p>Many people have implemented this in PyTorch but I seen all of them have just used base RNN models in both Encoder and Decoder which actually needs internal modification.</p>
</blockquote>
</aside>
<p>Posting links here would make responding easier.</p>
<p>Bahdanau’s formulation doesn’t seem to require any modifications to the built in RNN.</p>
<p>From the equations in Section A (Appendix) of the first paper, you could possibly write an inefficient(?) implementation using some primitives. Check the following pseudocode:</p>
<pre data-code-wrap="python"><code class="lang-python">class EncoderRNNCell(nn.Module):
    def __init__(self, args):
        def _linear():
            return nn.Linear(args.input_size, args.hidden_size)
        self.We = _linear()
        self.Ue = _linear()

        self.Wz = _linear()
        self.Uz = _linear()

        self.Wr = _linear()
        self.Ur = _linear()

    def forward(self, xs):
        # xs: T x args.input_size
        T, H = x.size()
        # TODO: Init h, r with legit values

        hs = [h]
        for i in range(T):
            x = xs[i, :]
            h_ = F.tanh(self.We(x) + self.Ue(r*h))
            z = F.sigmoid(self.Wz(x) + self.Uz(h))
            r = F.sigmoid(self.Wr(x) + self.Ur(h))
            h = z*h + r*h_
            hs.append(h)

        hiddens = torch.stack(hs, dim=...)
        return hiddens

</code></pre>
<p>You can use <code>hiddens[-1]</code> as  <code>h&lt;T&gt;</code>.</p>

divyesh_rajpura: <p>I just find out that proposed model in 1st paper is nothing but GRU. But problem comes in the decoder part of 2nd paper. They have give additional input to the GRU which is context vector ‘c’.</p>
<p>Thank you for reply <a class="mention" href="/u/jerinphilip">@jerinphilip</a>. RNN or LSTM is combination of Linear layers, is it how it is implemented??</p>

jerinphilip: <p>Under the hood, not exactly. There should be some more optimizations to parallel-process (something like <a href="https://youtu.be/-uyXE7dY5H0?t=927" rel="nofollow noopener">this</a>)  taking advantage of GPUs for class of RNN Models. These are implemented in the CPP backend (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_VF.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/_VF.py</a>) .</p>
<p>But you should be able to approximate the behaviour using composition of nn.Linear layers, I think. GRU and LSTM will serve as a substitute which could be adapted for your use case - but your query seemed like you wanted more flexibility.</p>

divyesh_rajpura: <p><a class="mention" href="/u/jerinphilip">@jerinphilip</a>, I think create RNN/GRU from scratch would require too many other implementation and when it comes to Bidirectional RNN or Stacked RNN it would become more complex to implement from scratch.</p>
<p>I found one implementation of <a href="https://github.com/divyeshrajpura4114/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" rel="nofollow noopener">Bahdanau’s Attention</a> where he just combine this context vector with input embedding vector to overcome 3 input RNN/GRU problem.</p>