How to use attention and encoding layers in pytorch

username: Neskelogth
cooked: <p>I have a working neural network with two data streams as a backbone network. Now I’m working on the classification head and would like to sum the features extracted from the two streams in the backbone using an attention mechanism, but unfortunately I didn’t understand how I could do it. Using pytorch, I have this code:</p>
<pre><code class="lang-auto">def forward(self, x):

    x1, x2 = self.pooling_layer(x[0]), self.pooling_layer(x[1])

    assert x1.shape == x2.shape, 'The dimension of the rgb features and the pose features should be the same'

    x1 = x1.view(x1.size(0), -1)
    x2 = x2.view(x2.size(0), -1)

    x = # Perform sum with attention

    assert x.shape[1] == self.in_channels

    if self.dropout_layer is not None:
        x = self.dropout_layer(x)

    scores = self.fc_layer(x)
    return scores
</code></pre>
<p>How can I implement this?</p>

username: HELLORPG
cooked: <p>If you have x1 in (N, C) and x2 in (N, C), then you can not directly merge these two into (N, C) via attention in my understanding, because the shape of the query is the same as the shape of the outputs.</p>
<p><code>torch.cat(x1, x2)</code> has the shape of (2N, C). Thus if you only use an attention layer to produce the outputs, the output should also be (2N, C). Maybe you should provide more information.</p>

username: Neskelogth
cooked: <p>What I would like to implement is not actually a concatenation, so I wouldn’t use <code>torch.cat(x1, x2)</code>, but a weighted sum. I would like to implement an attention mechanism to weight the sum, so in my idea it would look more like <code>torch.add(att_weight_1 * x1, att_weight_2 * x2)</code>. Unfortunately. this is my first time working with attention, so I may have misunderstood the way this layer works, in this case I would be grateful if you could explain to me what I got wrong.</p>

username: HELLORPG
cooked: <p>So, could you please tell me the shape that you have mentioned? I think x1 and x2 are both (N, C). Thus, the attn_weight_1 is (N, N), right?</p>

username: HELLORPG
cooked: <p>For <code>attn_weight_1</code>, what are your query, key, and value? For example, for me, maybe the x1 is the query, and x2 is the key and value. Then, this attention can model the relationship between x1 and x2. But I’m not sure whether this is what you want.</p>

username: Neskelogth
cooked: <p>To give further information, I have two 3D CNN branches that extract features from RGB images in one branch and heatmaps in the other. As of now, the branches produce features with different shapes (I modified the branches), which are (1, 17, 1, 1, 1) for the heatmaps and (1, 3, 1, 1, 1) for the RGB after the pooling operation. After the reshaping operation, I would like to sum these two output feature tensors based on an attention-weighted sum, in order to make sure the network learns which branch carries the most information. Is it possible to do so? Or is it possible to obtain this result with and encoder layer otherwise? I hope that my problem is clear enough now</p>

username: HELLORPG
cooked: <p>In (1, 17, 1, 1, 1) for example, <code>17</code> is the num of channels right? or some thing else?</p>

username: HELLORPG
cooked: <p>If I do not misunderstand, <code>17</code> is the number of channels, then the attention layer will not make sense.</p>
<p>Because the attention layer is aimed to get the scores between different tokens, but your outputs of these two branches do not have multi-tokens. And I don’t think it is reasonable to make an attention layer in the channel-wise data in your situation.</p>

username: Neskelogth
cooked: <p>Yes, 17 is the number of channels in this case. If the attention layer is not a suitable option, is there any way I can make the network learn which branch of the neural network carries the most information and perform a weighted sum based on that?</p>

username: HELLORPG
cooked: <p>I believe there must be various designs to achieve your goal.</p>
<p>For me, I may consider to use <code>adaptive weight</code> as I do in <a href="https://arxiv.org/pdf/2307.15700.pdf" rel="noopener nofollow ugc">this paper</a>, shown in Fig 2 (orange block).</p>
<p>It seems like, given an input X, whose shape is (N, C), I will use an MLP module to output a W in shape (N, C). In W, w_ij means the weight of the j th channel for the i th input instance. Then, use X’ = X dot W to get the weighted X, thus deciding which channel has more information.</p>
<p>As for your situation, you can change both x1 and x2 into the same number of channels independently. Then use the above <code>adaptive weight</code> to decide the weight of each branch’s information (each branch needs a <code>adaptive weight</code> module). Then you can merge these two branches by <code>add</code>.</p>
<p>The formulation is:<br>
x1’ = MLP_1(x1)<br>
x2’ = MLP_2(x2)<br>
W_1 = MLP_w1(x1’)<br>
W_2 = MLP_w2(x2’)<br>
final_x = x1’ dot W_1 + x2’ dot W_2</p>

username: HELLORPG
cooked: <p>And there are also some keywords that may be useful when you search on the webs:</p>
<ul>
<li>adaptive weight</li>
<li>channel-wise convolution</li>
<li>channel-wise attention</li>
<li>channel-wise</li>
<li>channel-wise weight</li>
</ul>
<p>In my memory, there should be a lot of this kind of work, but I can’t remember it immediately.</p>

username: J_Johnson
cooked: <p>Sounds like <code>torch.nn.parameter.Parameter</code> is all you need.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html</a></p>
<p>In your init, you can define:</p>
<pre><code class="lang-auto">self.att_weight_1 = Parameter(torch.randn(1))
self.att_weight_2 = Parameter(torch.randn(1))
</code></pre>
<p>The above would be just one learnable value per weight. You could also make it elementwise by passing in the appropriate sized tensor that matches the size of your x1/x2 tensors.</p>
<p>Then you can just call in your forward pass:</p>
<pre><code class="lang-auto">x = self.att_weight_1 * x1 + self.att_weight_2 * x2
</code></pre>
<p>Alternatively, if you want them to always add to one, you can just use one weight as follows:</p>
<pre><code class="lang-auto">x = self.att_weight_1 * x1 + (1 - self.att_weight_1) * x2
</code></pre>