Self-Attention (on words) and masking

cbaziotis: <p>I have a simple model for text classification. It has an attention layer after an RNN, which computes a weighted average of the hidden states of the RNN. I sort each batch by length and use <code>pack_padded_sequence</code> in order to avoid computing the masked timesteps. The model works but i want to apply masking on the attention scores/weights.</p>
<p>Here is my Layer:</p>
<pre><code class="lang-python">class SelfAttention(nn.Module):
    def __init__(self, hidden_size, batch_first=False):
        super(SelfAttention, self).__init__()

        self.hidden_size = hidden_size
        self.batch_first = batch_first

        self.att_weights = Parameter(torch.Tensor(1, hidden_size),
                                     requires_grad=True)

        init.xavier_uniform(self.att_weights.data)

    def get_mask(self):
        pass

    def forward(self, inputs):

        if isinstance(inputs, PackedSequence):
            # unpack output
            inputs, lengths = pad_packed_sequence(inputs,
                                                  batch_first=self.batch_first)
        if self.batch_first:
            batch_size, max_len = inputs.size()[:2]
        else:
            max_len, batch_size = inputs.size()[:2]

        # att = torch.mul(inputs, self.att_weights.expand_as(inputs))
        # att = att.sum(-1)
        weights = torch.bmm(inputs,
                            self.att_weights  # (1, hidden_size)
                            .permute(1, 0)  # (hidden_size, 1)
                            .unsqueeze(0)  # (1, hidden_size, 1)
                            .repeat(batch_size, 1, 1) # (batch_size, hidden_size, 1)
                            )

        attentions = F.softmax(F.relu(weights.squeeze()))

        # apply weights
        weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))

        # get the final fixed vector representations of the sentences
        representations = weighted.sum(1).squeeze()

        return representations, attentions

</code></pre>
<p>I tried adding this, but it obviously fails with an error <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</code></p>
<pre><code class="lang-python">        ...
        attentions = F.softmax(F.relu(weights.squeeze()))

        # apply masking based on the sentence lengths
        for i, l in enumerate(lengths[1:], 1):  # skip the first sentence
            attentions[i, l:] = 0

        # apply weights
        weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))
        ...
</code></pre>
<p>How can i mask the attention weights?</p>

ruotianluo: <p>you can instead create a 0,1 mask variable, and</p>
<pre><code>attentions = attentions * mask</code></pre>

cbaziotis: <p><a class="mention" href="/u/ruotianluo">@ruotianluo</a>  Thanks. I will try it and post back. Do i have to <code>detach()</code> the variable or something like that? I am not sure how this operation will affect the flow of the gradients.</p>

cbaziotis: <p>So this is what i came up with:</p>
<pre><code class="lang-python">class SelfAttention(nn.Module):
    def __init__(self, hidden_size, batch_first=False):
        super(SelfAttention, self).__init__()

        self.hidden_size = hidden_size
        self.batch_first = batch_first

        self.att_weights = Parameter(torch.Tensor(1, hidden_size),
                                     requires_grad=True)

        init.xavier_uniform(self.att_weights.data)

    def get_mask(self):
        pass

    def forward(self, inputs):

        if isinstance(inputs, PackedSequence):
            # unpack output
            inputs, lengths = pad_packed_sequence(inputs,
                                                  batch_first=self.batch_first)
        if self.batch_first:
            batch_size, max_len = inputs.size()[:2]
        else:
            max_len, batch_size = inputs.size()[:2]

        # apply attention layer
        weights = torch.bmm(inputs,
                            self.att_weights  # (1, hidden_size)
                            .permute(1, 0)  # (hidden_size, 1)
                            .unsqueeze(0)  # (1, hidden_size, 1)
                            .repeat(batch_size, 1, 1)
                            # (batch_size, hidden_size, 1)
                            )

        attentions = F.softmax(F.relu(weights.squeeze()))

        # create mask based on the sentence lengths
        mask = Variable(torch.ones(attentions.size())).cuda()
        for i, l in enumerate(lengths):  # skip the first sentence
            if l &lt; max_len:
                mask[i, l:] = 0

        # apply mask and renormalize attention scores (weights)
        masked = attentions * mask
        _sums = masked.sum(-1).expand_as(attentions)  # sums per row
        attentions = masked.div(_sums)

        # apply attention weights
        weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))

        # get the final fixed vector representations of the sentences
        representations = weighted.sum(1).squeeze()

        return representations, attentions
</code></pre>
<p>it works and the networks performs better.<br>
Hope it helps someone.</p>
<p>However, I am concerned about the gradients of the attention layer. I mean how does the masking operation affect them? Is this the right way? Or is there a more elegant way of achieving the same thing?</p>

tom: <p>Hello Christos,</p>
<p>With</p>
<pre><code class="lang-auto">attentions = Variable(torch.randn(5,10).cuda())
max_len = attentions.size(1)
lengths = ((torch.arange(0,5)+5).long().cuda())
</code></pre>
<p>as an example, I think you can do this</p>
<aside class="quote no-group" data-username="cbaziotis" data-post="4" data-topic="5671">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbaziotis/48/38067_2.png" class="avatar"> cbaziotis:</div>
<blockquote>
<pre><code class="lang-auto">        # create mask based on the sentence lengths
        mask = Variable(torch.ones(attentions.size())).cuda()
        for i, l in enumerate(lengths):  # skip the first sentence
            if l &lt; max_len:
                mask[i, l:] = 0
</code></pre>
</blockquote>
</aside>
<p>faster if you do</p>
<pre><code class="lang-auto">idxes = torch.arange(0,max_len,out=torch.LongTensor(max_len)).unsqueeze(0).cuda() # some day, you'll be able to directly do this on cuda
mask = Variable((idxes&lt;lengths.unsqueeze(1)).float())
</code></pre>
<p>(works on master / 0.2, you need to expand_as(attention) or so on 0.1.12)</p>
<aside class="quote no-group" data-username="cbaziotis" data-post="4" data-topic="5671">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbaziotis/48/38067_2.png" class="avatar"> cbaziotis:</div>
<blockquote>
<p>However, I am concerned about the gradients of the attention layer. I mean how does the masking operation affect them? Is this the right way? Or is there a more elegant way of achieving the same thing?</p>
</blockquote>
</aside>
<p>If you multiply the output by 0 with the mask, this will propagate a gradient of 0 to <code>attention</code>. I think this is the right way in general, but I don’t have the expertise to say how best  handle the end of the sequence…</p>
<p>Best regards</p>
<p>Thomas</p>

lyan62: <aside class="quote no-group" data-username="cbaziotis" data-post="1" data-topic="5671">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbaziotis/48/38067_2.png" class="avatar"> cbaziotis:</div>
<blockquote>
<p>representations = weighted.sum(1).squeeze()</p>
</blockquote>
</aside>
<p>hi <a class="mention" href="/u/cbaziotis">@cbaziotis</a>, thanks for sharing,<br>
for the final representation, can you point me to the related paper or material? thanks</p>

cbaziotis: <p><a class="mention" href="/u/lyan62">@lyan62</a>  I’ve used this attention mechanism, in the following paper: <a href="https://arxiv.org/abs/1804.06659" rel="nofollow noopener">https://arxiv.org/abs/1804.06659</a>. I will share the source code of our models in this repo <a href="https://github.com/cbaziotis/ntua-slp-semeval2018-task3" rel="nofollow noopener">https://github.com/cbaziotis/ntua-slp-semeval2018-task3</a> in a few days. In the meantime you can find my implementation of the simple self-attention mechanism here: <a href="https://gist.github.com/cbaziotis/94e53bdd6e4852756e0395560ff38aa4" rel="nofollow noopener">https://gist.github.com/cbaziotis/94e53bdd6e4852756e0395560ff38aa4</a></p>

yngtodd: <p><a class="mention" href="/u/cbaziotis">@cbaziotis</a>, Can you explain a bit about the lengths parameter in <code>forward()</code>? I was looking through your gist <a href="https://gist.github.com/cbaziotis/94e53bdd6e4852756e0395560ff38aa4#file-selfattention-py-L31" rel="nofollow noopener">https://gist.github.com/cbaziotis/94e53bdd6e4852756e0395560ff38aa4#file-selfattention-py-L31</a>. It looks like that parameter is a <code>Variable</code>, but I am not sure where it comes from.</p>

cbaziotis: <p><a class="mention" href="/u/yngtodd">@yngtodd</a> <code>lengths</code> is a list of integers, in which each number corresponds to the actual length of each sentence.</p>
<p>It is need in order to take into account the zero-padded timesteps. Only by knowing the length of each sentence we can properly calculate the attention weights. We want to assign weight only on the actual words. Without knowing the length, the zero padded words will be assigned a non-zero weight by the attention mechanism.</p>

Elidor: <aside class="quote no-group" data-username="cbaziotis" data-post="4" data-topic="5671">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbaziotis/48/38067_2.png" class="avatar"> cbaziotis:</div>
<blockquote>
<p><code>return representations, attentions</code></p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/cbaziotis">@cbaziotis</a>, I have a little doubt about how the <code>SelfAttention</code> output should be used …<br>
The <code>representations.shape</code> should be <code>(batch_size, hidden_size)</code> and <code>attentions.shape</code> should be <code>(batch_size, max_len)</code>.<br>
How should they be used inside an lstm? What exactly do these two variables represent to me?</p>