Attention in image classification

username: theairbend3r
cooked: <p>Hi all,</p>
<p>I recently started reading up on attention in the context of computer vision. In my research, I found a number of ways attention is applied for various CV tasks. However, it is still unclear to me as to what’s really happening.</p>
<p>When I say attention, I mean a mechanism that will focus on the important features of an image, similar to how it’s done in NLP (machine translation).</p>
<p>I’m looking for resources (blogs/gifs/videos) with PyTorch code that explains how to implement attention for, let’s say, a simple image classification task.</p>
<p>Alternatively, It would be great if you write a small implementation of only the attention mechanism in the following way -</p>
<ol>
<li>Assume a tensor of size (h,w,c)</li>
<li>input tensor =&gt; attention =&gt; output</li>
</ol>
<p>One example is the self-attention from the <a href="https://arxiv.org/pdf/1805.08318.pdf" rel="noopener nofollow ugc">SAGAN paper</a> -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b68fb7910237976cfa90dff3bd72dcc9bbe80ec4.png" data-download-href="https://discuss.pytorch.org/uploads/default/b68fb7910237976cfa90dff3bd72dcc9bbe80ec4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b68fb7910237976cfa90dff3bd72dcc9bbe80ec4_2_690x286.png" alt="image" data-base62-sha1="q30Olj8YOhed6WVfFLXqjSauSy0" width="690" height="286" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b68fb7910237976cfa90dff3bd72dcc9bbe80ec4_2_690x286.png, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b68fb7910237976cfa90dff3bd72dcc9bbe80ec4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b68fb7910237976cfa90dff3bd72dcc9bbe80ec4.png 2x" data-dominant-color="F6F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">829×344 54.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How would I go about implementing the self-attention shown above and eventually use it for image classification?</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: cskarthik7
cooked: <p>Attentions are used to focus on specific features rather than giving importance to all the features. The convolutional network gives out large number of features and some of those features are not so important information but they also consume computation time when passed forward. Here in this case we use attention networks to assign more importance to useful features in which we are interested.</p>
<p>Here are some blogs which might help you!<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://miro.medium.com/fit/c/256/256/1*ChFMdf--f5jbm-AYv6VdYA@2x.png" class="site-icon" width="16" height="16">
      <a href="https://towardsdatascience.com/self-attention-in-computer-vision-2782727021f6" target="_blank" title="12:37PM - 25 September 2019" rel="nofollow noopener">Medium – 25 Sep 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:110/57;"><img src="https://miro.medium.com/max/1200/1*SiwBcud7HdRfb8JnksbHyQ.jpeg" class="thumbnail" width="110" height="57"></div>

<h3><a href="https://towardsdatascience.com/self-attention-in-computer-vision-2782727021f6" target="_blank" rel="nofollow noopener">Self-Attention In Computer Vision</a></h3>

<p>Using the attention mechanism as the building block in computer vision models</p>

  <p><span class="label1">Reading time: 10 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://miro.medium.com/fit/c/256/256/1*ChFMdf--f5jbm-AYv6VdYA@2x.png" class="site-icon" width="16" height="16">
      <a href="https://towardsdatascience.com/learn-to-pay-attention-trainable-visual-attention-in-cnns-87e2869f89f1" target="_blank" title="09:58PM - 10 August 2019" rel="nofollow noopener">Medium – 10 Aug 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:110/146;"><img src="https://miro.medium.com/max/800/1*8syelw0dvzvVTZr6Hn1jXg.jpeg" class="thumbnail" width="110" height="146"></div>

<h3><a href="https://towardsdatascience.com/learn-to-pay-attention-trainable-visual-attention-in-cnns-87e2869f89f1" target="_blank" rel="nofollow noopener">Learn to Pay Attention! Trainable Visual Attention in CNNs</a></h3>

<p>When training an image model, we want the model to be able to focus on important parts of the image. One way of accomplishing this is…</p>

  <p><span class="label1">Reading time: 11 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: AdilZouitine
cooked: <p>The code for the self-attention layer :</p>
<pre><code class="lang-python">import torch.nn as nn

class SelfAttention(nn.Module):
    """ Self attention Layer"""
    def __init__(self,in_dim,activation):
        super(SelfAttention,self).__init__()
        self.chanel_in = in_dim
        self.activation = activation
        
        self.query_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1)
        self.key_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1)
        self.value_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim , kernel_size= 1)
        self.gamma = nn.Parameter(torch.zeros(1))

        self.softmax  = nn.Softmax(dim=-1) #
    def forward(self,x):
        """
            inputs :
                x : input feature maps( B X C X W X H)
            returns :
                out : self attention value + input feature 
                attention: B X N X N (N is Width*Height)
        """
        m_batchsize,C,width ,height = x.size()
        proj_query  = self.query_conv(x).view(m_batchsize,-1,width*height).permute(0,2,1) # B X CX(N)
        proj_key =  self.key_conv(x).view(m_batchsize,-1,width*height) # B X C x (*W*H)
        energy =  torch.bmm(proj_query,proj_key) # transpose check
        attention = self.softmax(energy) # BX (N) X (N) 
        proj_value = self.value_conv(x).view(m_batchsize,-1,width*height) # B X C X N

        out = torch.bmm(proj_value,attention.permute(0,2,1) )
        out = out.view(m_batchsize,C,width,height)
        
        out = self.gamma*out + x
        return out,attention
</code></pre>
<p>Source : <a href="https://github.com/heykeetae/Self-Attention-GAN/blob/master/sagan_models.py#L8" rel="nofollow noopener">here</a></p>
<p>Article :  <a href="https://arxiv.org/pdf/1805.08318.pdf" rel="nofollow noopener">here</a></p>
<p>I hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: sharad
cooked: <p>Hi, <a class="mention" href="/u/adilzouitine">@AdilZouitine</a> I am using self-attention in a personal project, which has the exact same implementation mentioned by you. However, if feature maps sizes get larger even up to (B,128,256,256), the program runs out of memory on my GPU of 12 GB memory. Is there any workaround this apart from going for a bigger GPU? The network is not even able to run one forward pass.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: AdilZouitine
cooked: <p>Hi <a class="mention" href="/u/sharad">@sharad</a> , In the paper I posted, the authors use self-attention after processing the image with a succession of convolution layers.<br>
This means that at the input of the self-attention layer the size of the features map is already reduced and therefore takes less RAM memory.</p>

username: sharad
cooked: <p>Yeah <a class="mention" href="/u/adilzouitine">@AdilZouitine</a> that seems to be the case in DANet paper as well, using self-attention on the last feature maps.  Thanks!</p>

username: Federico_Vasile
cooked: <p>Why is <code>attention</code> permuted in <code>out = torch.bmm(proj_value,attention.permute(0,2,1) )</code>  ?<br>
Couldn’t it just stay the way it is(i.e. without permuting)?</p>

username: Saifeddine_Barkia
cooked: <p>Hello sir ,<br>
Here is a good paper for the attention model with its code impelemented.<br>
I hope it will help you.<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/wvinzh/WS_DAN_PyTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/17807170?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/wvinzh/WS_DAN_PyTorch" target="_blank" rel="nofollow noopener">wvinzh/WS_DAN_PyTorch</a></h3>

<p>PyTorch Implementation Of WS-DAN(See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification) - wvinzh/WS_DAN_PyTorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: batu_tw
cooked: <p><a href="https://arxiv.org/pdf/1805.08318.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1805.08318.pdf</a> author realized the equation (2) in such way if converted to matrix-matrix multiplication</p>

username: AdilZouitine
cooked: <p>Hi it’s a mistake on my part, you can delete this is useless</p>

username: aka4rKO
cooked: <p>Hi,</p>
<p>I’m using a resnet to extract features and the last layer gives me an output of (batchsize x 2048 x 1 x 1).</p>
<p>When i passed this onto this code SelfAttention i get a the same dimensions out. Is this the correct behaviour?</p>
<p>Can i use this vector to flatten and pass it to fully connected layer in order to classify my own classes?</p>
<p>Thank you.</p>

username: krishna511
cooked: <p><a class="mention" href="/u/aka4rko">@aka4rKO</a> Yes I think thats the purpose of your doing it right?<br>
Attention coeff made your input lil more contextual for better classification.</p>

username: AFAF_KELAI
cooked: <p>am tring to use the same self-attention implementation on colab but it runs out of memory , how can i modify it so it can run correctly on colab CUDA GPU ?</p>