Attention for image classification

username: vainaijr
cooked: <p>for an input image of size, 3x28x28</p>
<pre><code class="lang-auto">inp = torch.randn(1, 3, 28, 28)
x = nn.MultiheadAttention(28, 2)
x(inp[0], torch.randn(28, 28), torch.randn(28, 28))[0].shape
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">torch.Size([3, 28, 28])
</code></pre>
<p>while</p>
<pre><code class="lang-auto">x(inp[0], torch.randn(28, 28), torch.randn(28, 28))[1].shape
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">torch.Size([28, 3, 1])
</code></pre>
<p>what is the correct way of using MultiHeadAttention for images?</p>
<p>does this multiheadattention work something liike paying attention only on a part of image?</p>

username: dejanbatanjac
cooked: <p>I think <a href="https://pytorch.org/docs/master/nn.html#multiheadattention" rel="nofollow noopener">this function</a> is for the sequence models, and not for image classification. Based on the paper <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noopener">Attention is all you need</a>.</p>

username: vainaijr
cooked: <p>but attention is applicable to images also, like here,<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/tengshaofeng/ResidualAttentionNetwork-pytorch/blob/master/Residual-Attention-Network/model/attention_module.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/tengshaofeng/ResidualAttentionNetwork-pytorch/blob/master/Residual-Attention-Network/model/attention_module.py" target="_blank" rel="nofollow noopener">tengshaofeng/ResidualAttentionNetwork-pytorch/blob/master/Residual-Attention-Network/model/attention_module.py</a></h4>
<pre><code class="lang-py">import torch
import torch.nn as nn
from torch.nn import init
import functools
from torch.autograd import Variable
import numpy as np

from .basic_layers import ResidualBlock


class AttentionModule_pre(nn.Module):

    def __init__(self, in_channels, out_channels, size1, size2, size3):
        super(AttentionModule_pre, self).__init__()
        self.first_residual_blocks = ResidualBlock(in_channels, out_channels)

        self.trunk_branches = nn.Sequential(
            ResidualBlock(in_channels, out_channels),
            ResidualBlock(in_channels, out_channels)
         )
</code></pre>

  This file has been truncated. <a href="https://github.com/tengshaofeng/ResidualAttentionNetwork-pytorch/blob/master/Residual-Attention-Network/model/attention_module.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>is nn.MultiheadAttention applicable for images?</p>
<p>I want neural network to focus on part of image, I think attention is doing something similar with text, focusing on some part of text.</p>

username: dejanbatanjac
cooked: <p>But <code>nn.MultiheadAttention</code> the is just for sequence models.</p>
<p>I guess you meant some techniques to apply attention to convolution networks.</p>
<p>Attention is like a new wave for convnets.<br>
You can do it either by changing the architecture or changing the loss function or both.</p>
<p>The problem with convolution is that it has local receptive field.<br>
Opposite to that <strong>fc</strong> layers have the global receptive field. So the idea to combine that using SE blocks is <a href="https://arxiv.org/pdf/1709.01507.pdf" rel="nofollow noopener">here</a>.</p>
<p>Also, there is an idea to concat convolution and attentional featuremaps <a href="https://arxiv.org/abs/1904.09925" rel="nofollow noopener">here</a>.</p>
<p>Same ideas but for GAN, you can find in <a href="https://arxiv.org/abs/1805.08318" rel="nofollow noopener">SAGAN</a>.</p>
<p>But these papers <em>I think</em> haven’t been implemented in PyTorch yet. It may take about few months for the good paper to be inside the PyTorch.</p>

username: phan_phan
cooked: <p><a href="https://arxiv.org/pdf/1804.02391.pdf" rel="nofollow noopener">This paper</a> can also be useful, depending on what you are trying to do.</p>

username: vainaijr
cooked: <p>What is dv, dk here?</p>
<pre><code class="lang-auto">We now formally describe our proposed Attention Augmentation method.
We use the following naming conventions: H, W and Fin refer to the height, 
width and number
of input filters of an activation map. Nh, dv and dk respectively refer the 
number of heads, the depth of values and the
depth of queries and keys in multihead-attention (MHA).
We further assume that Nh divides dv and dk evenly and
denote dhv and dhk the depth of values and queries/keys per
attention head.
</code></pre>
<p>if I do nn.MultiheadAttention(28, 2), then Nh = 2, but, dv, dk, dhv, dhk = ???</p>
<p>If I want to transform an image to another image, then</p>
<pre><code class="lang-auto">transformer_model = nn.Transformer(img_size, n_heads)
transformer_model(source_image, target_image)
</code></pre>
<p>is this the correct way to use nn.Transformer for images?</p>

username: rudra_saha
cooked: <p>Were you able to figure out how to use MHA for images? Would appreciate pointers if you have!</p>