Applying Attention (Single and MultiHead Attention)

shakeel608: <p>Applying Attention from paper</p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://www.danielpovey.com/files/2018_interspeech_xvector_attention.pdf" target="_blank" rel="nofollow noopener">danielpovey.com</a>
  </header>
  <article class="onebox-body">
    <a href="https://www.danielpovey.com/files/2018_interspeech_xvector_attention.pdf" target="_blank" rel="nofollow noopener"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://www.danielpovey.com/files/2018_interspeech_xvector_attention.pdf" target="_blank" rel="nofollow noopener">2018_interspeech_xvector_attention.pdf</a></h3>

<p class="filesize">266.24 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Suppose my Hidden audio representation  shape is  (after few CNN operations/layers)</p>
<pre><code class="lang-auto"> H = torch.Size([128, 32, 64])    [Batch Size X FeatureDim X Length]

</code></pre>
<p>and I want to apply self-attention weights to the audio hidden frames as</p>
<pre><code class="lang-auto">A = softmax(ReLU(AttentionWeight1 * (AttentionWeight2 * H))
</code></pre>
<p>In order to learn these two self attention weight matrices. Do I need to register these two weights as Parameters in the init function like below</p>
<pre><code class="lang-auto">
class Model(nn.Module):
        def __init__(self, batch_size):
           super(Model, self).__init__()
           self.attention1   = nn.Parameter(torch.Tensor(self.batch_size,16, 32))
           self.attention2   = nn.Parameter(torch.Tensor(self.batch_size,1, 16))  
</code></pre>
<p>and in the forward do I need to do like this</p>
<pre><code class="lang-auto">def forward(self, input):
    ....
   H = CNN(input)     #[B X Features X length]
   attention = nn.Softmax(nn.ReLU(torch.mm(self.attention2, torch.mm(self.attention1, H))
   H = H*attention
   return H


</code></pre>
<p>Please help. How can we apply attention here. A the above code is throwing error</p>
<pre><code class="lang-auto">RuntimeError: matrices expected, got 3D, 3D tensors at /opt/conda/conda-bld/pytorch_1591914985702/work/aten/src/TH/generic/THTensorMath.cpp:36

</code></pre>

ptrblck: <p><code>torch.mm</code> expects two matrices (2D tensors), while you seem to use two 3D tensors.<br>
You could use <code>torch.bmm</code> or <code>torch.matmul</code> instead, which would work for these tensors.</p>
<p>However, usually the parameters are not depending on the batch size.<br>
Are you sure you want to initialize them with the batch size in <code>dim0</code>?</p>

shakeel608: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
How to make these weights learnable.<br>
Am I doing it right here</p>
<pre><code class="lang-auto">class Model(nn.Module):
        def __init__(self, batch_size):
           super(Model, self).__init__()
           self.attention1   = nn.Parameter(torch.Tensor(self.batch_size,16, 32))
           self.attention2   = nn.Parameter(torch.Tensor(self.batch_size,1, 16))  
</code></pre>
<p>Attention Mechanism in Forward. The <strong>input</strong> here is the output after few CNN operations with Shape</p>
<pre><code class="lang-auto">[BatchSize X DimFeature X Length]   = [128 X 32 X 64]
</code></pre>
<pre><code class="lang-auto">       """ Get Attention Weights """
        attn = input    
        attention = attn.permute(0, 2, 1).matmul(self.attention1)
        attention = attention.matmul(self.attention2)
        attention = self.relu(attention)      
        attention = attention.view(attention.size(0), -1)
        attention = F.softmax(attention, 1)  
        """ Multiply Attention Weights with Audio Frames"""
        input = input * attention.unsqueeze(1)          #To Make it comptable with BatchSize X FeatureDim X FixedLength

</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is this okay now?<br>
I have also another question when I use<code> nn.Softmax</code> in place of <code>F.softmax(attention, 1)</code>, why doesn’t it work ?</p>

ptrblck: <p>The code looks alright code-wise and you should be able to see valid gradients in <code>model.attention1.grad</code> and <code>model.attention2.grad</code> after a <code>backward()</code> call.</p>
<p><code>nn.Softmax</code> should work like <code>F.softmax</code>, but you might have forgotten to create the module before calling it via:</p>
<pre><code class="lang-python">nn.Softmax(dim=1)(input)
</code></pre>
<p>What kind of error are you seeing with <code>nn.Softmax</code>?</p>

shakeel608: <p>Thank you for your feedback.</p>
<p>I think I was using the syntax wrongly  as below</p>
<p><code>nn.Softmax(input, 1)</code></p>
<p>but it is actually like this</p>
<pre><code class="lang-auto">nn.Softmax(dim=1)(input)
</code></pre>

krishna511: <p><a class="mention" href="/u/shakeel608">@shakeel608</a> Have you done your task ?<br>
I am using a transformer network for my audio, ofcourse the encoder part only for multihead attention using key quey and value matrices.<br>
Could you plz explain what is the purpose of this H at last? Is this only for rewighted H, for better classification.</p>
<p>Regards</p>