Attention in ConvGRU

username: zimmer550
cooked: <p>Hello, the size of the output from my decoder GRU is (4, 1, 256, 4, 4) and the size of the hidden layer from my encoder is (4, 5, 256, 4, 4). Where the order of the dimensions is: (batch, time_step, image_depth, height, width). I use the following piece of code to compute the attention:</p>
<pre><code class="lang-auto">attn_energies = torch.sum(decoder_output * encoder_output, dim=2)
attn_energies = attn_energies.permute(0, 3, 2, 1)
final_output = F.softmax(attn_energies, dim=1).unsqueeze(1)
</code></pre>
<p>The result of the above is a new tensor of dimensions: (4, 5, 4, 4). Is this correct? I am following the official PyTorch ChatBot tutorial: <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html#decoder" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html#decoder</a></p>
<p>How do I apply the next step, namely batch multiplication? It’s a bit confusing because here there are images and not word embeddings</p>
<p>Edit: This is the method I use to calculate attention. Please tell me whether this is right:</p>
<pre><code class="lang-auto">            #hidden[-1] has size (4, 1, 256, 4, 4), all_hidden has size (4, 5, 256, 4, 4)
            attn_weights = self.attn(hidden[-1], all_hidden)
            #attn_weights has size (4, 1, 256, 4, 4)
            #self.relu(p_tmp) is the output from all the GRU cells in encoder stage, size (4, 256, 4, 4)
            #self.relu(p_tmp).unsqueeze(1) has size (4, 1, 256, 4, 4)
            context = attn_weights.matmul(self.relu(p_tmp).unsqueeze(1))
            #context has size (4, 1, 256, 4, 4)
            concat_input = torch.cat((hidden[-1], context), 1)
            #concat_input has size (4, 2, 256, 4, 4)
            (B, N, C, H, W) = concat_input.shape
            concat_input = concat_input.view(B * N, C, H, W)
            #concat_input now has size (8, 256, 4, 4)
            concat_output = torch.tanh(self.concat(concat_input))
            #concat_output has size (8, 256, 4, 4)
            hidden = self.out(concat_output)
            #size of hidden is (8, 256, 4, 4)
            hidden = hidden.view(B, N, C, H, W)
            #size of hidden is (4, 2, 256, 4, 4)
            hidden = hidden.permute(0, 2, 1, 3, 4)
            #size of hidden is (4, 256, 2, 4, 4)
            hidden = F.avg_pool3d(hidden, (self.last_duration, 1, 1), stride=(1, 1, 1))
            hidden = hidden.permute(0, 2, 1, 3, 4)
            #size of hidden is (4, 1, 256, 4, 4)
            hidden = hidden[:,-1,:]
            #and final size of hidden is (4, 256, 4, 4)
</code></pre>
<p>where:</p>
<pre><code class="lang-auto">def attn(hidden, encoder_outputs):
    attn_energies = torch.sum(hidden * encoder_output, dim=1)
    first = F.softmax(attn_energies, dim=2)
    second = F.softmax(first, dim=3)
    return second.unsqueeze(1)
self.concat = nn.Sequential(
            nn.Conv2d(self.param['feature_size'], self.param['feature_size'], kernel_size=1, padding=0),
            nn.ReLU(inplace=True)
        )
self.out = nn.Sequential(
            nn.Conv2d(self.param['feature_size'], self.param['feature_size'], kernel_size=1, padding=0),
            nn.ReLU(inplace=True)
        )
#self.param['feature_size'] is 256
</code></pre>