Stacking a couple of resnet blocks each with a self-attention module

Mona_Jalal: <p>Imagine I want to feed an image to a couple of stacked resnet blocks. How should I stack the resnet blocks in code and how can I attach self-attention modules to them? Can you please suggest to me some related code that might look like this architecture?</p>
<p>These are natural images.<br>
I am open to using either resnet18 or resnet50.</p>
<p>Also at the very end, how does the final embedding combine all of these?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9b48d3ca7477f2cc833ef83281f669117bd03f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9b48d3ca7477f2cc833ef83281f669117bd03f7" title="Screen Shot 2021-08-23 at 11.04.14 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b48d3ca7477f2cc833ef83281f669117bd03f7_2_690x367.png" alt="Screen Shot 2021-08-23 at 11.04.14 PM" data-base62-sha1="odhwpsObes5sgBDJh39fTtY93bF" width="690" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b48d3ca7477f2cc833ef83281f669117bd03f7_2_690x367.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b48d3ca7477f2cc833ef83281f669117bd03f7_2_1035x550.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b48d3ca7477f2cc833ef83281f669117bd03f7_2_1380x734.png 2x" data-dominant-color="DDDCE3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-23 at 11.04.14 PM</span><span class="informations">2278×1214 232 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mona_Jalal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I think i want to do something like this except I am unsure how to add self-attention to intermediate resnet blocks</p><aside class="quote" data-post="12" data-topic="80147">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aka4rko/48/37713_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/attention-in-image-classification/80147/12">Attention in image classification</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hi, 
I’m using a resnet to extract features and the last layer gives me an output of (batchsize x 2048 x 1 x 1). 
When i passed this onto this code SelfAttention i get a the same dimensions out. Is this the correct behaviour? 
Can i use this vector to flatten and pass it to fully connected layer in order to classify my own classes? 
Thank you.
  </blockquote>
</aside>

<p>also, for self-attention, there are so many options. Is there any self-built module within pytorch that you would suggest?<br>
I found these alternatives, what is your take?</p>
<ol>
<li><a href="https://discuss.pytorch.org/t/attention-in-image-classification/80147/3" class="inline-onebox">Attention in image classification - #3 by AdilZouitine</a></li>
<li><a href="https://github.com/Chenglin-Yang/LESA" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Chenglin-Yang/LESA: Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms</a></li>
<li><a href="https://github.com/leaderj1001/Stand-Alone-Self-Attention" class="inline-onebox" rel="noopener nofollow ugc">GitHub - leaderj1001/Stand-Alone-Self-Attention: Implementing Stand-Alone Self-Attention in Vision Models using Pytorch</a></li>
</ol>