Strategies for Reshaping Tensors to Meet Specific Requirements: Handling Spatial Continuity in Swin-Transformer Architecture

Iyad_alshami: <p>I am trying to implement the <strong>mask_windows</strong> in the <strong>swin-transformer</strong> architecture.</p>
<p>I have a mask tensor that looks like this:</p>
<pre><code class="lang-python">tensor([[0., 0., 0., 0., 1., 1., 2., 2.],
        [0., 0., 0., 0., 1., 1., 2., 2.],
        [0., 0., 0., 0., 1., 1., 2., 2.],
        [0., 0., 0., 0., 1., 1., 2., 2.],
        [3., 3., 3., 3., 4., 4., 5., 5.],
        [3., 3., 3., 3., 4., 4., 5., 5.],
        [6., 6., 6., 6., 7., 7., 8., 8.],
        [6., 6., 6., 6., 7., 7., 8., 8.]])

torch.Size([1, 8, 8, 1])
</code></pre>
<p>I want to convert it to have the shape :</p>
<pre><code class="lang-python">torch.Size([4, 4, 4, 1])
</code></pre>
<p>which must result from partitioning it into quarters, each of size 4 by 4. This explains the presence of the number 4 as the first index in the desired shape.</p>
<p>My initial attempt was:</p>
<pre><code class="lang-python">windows = x.view(-1, window_size, window_size, C)
</code></pre>
<p>However, this approach disrupts the spatial continuity of the quarters.</p>
<p>And the correct way to do this is:</p>
<pre><code class="lang-python">x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)
</code></pre>
<p>My question revolves around the right mindset and step-by-step approach for tackling such problems. What is the proper mental framework or checklist to ensure that the output adheres to the specified requirements, aside from trial and error with small-sized tensors?<br>
I understand that this relates to how the view and permute operations work, but I’m struggling to simplify my understanding.</p>
<p>Could you guide me on how to think about such transformation?</p>

pytorchtester0: <p>To achieve the desired transformation, you should think about it in terms of reshaping and reorganizing the original tensor. Here’s a step-by-step approach:</p>
<ol>
<li><strong>Understand the Desired Shape</strong>: First, understand the desired shape of the output tensor. In this case, it’s <code>[4, 4, 4, 1]</code>, which indicates that you want to partition the original tensor into quarters.</li>
<li><strong>Identify the Dimensions to Divide</strong>: Determine which dimensions of the original tensor need to be divided to achieve the desired shape. In this case, you want to divide the height and width dimensions (2nd and 3rd dimensions), as indicated by <code>H // window_size</code> and <code>W // window_size</code>.</li>
<li><strong>Use View to Divide Dimensions</strong>: Use the <code>view</code> operation to divide the identified dimensions. In this case, you can use <code>view(B, H // window_size, window_size, W // window_size, window_size, C)</code> to create quarters along the height and width dimensions.</li>
<li><strong>Reorganize Dimensions</strong>: To ensure spatial continuity within each quarter, you may need to reorganize the dimensions. In your example, you used <code>permute(0, 1, 3, 2, 4, 5)</code> to rearrange dimensions.</li>
<li><strong>Flatten for Final Shape</strong>: Finally, use <code>contiguous().view</code> to flatten the tensor into the desired shape <code>-1, window_size, window_size, C</code>.</li>
</ol>
<p>So, the key mental framework is to:</p>
<ul>
<li>Understand the desired output shape.</li>
<li>Identify which dimensions of the original tensor need to be divided.</li>
<li>Use <code>view</code> to partition those dimensions.</li>
<li>Reorganize dimensions if necessary.</li>
<li>Flatten the tensor to achieve the desired final shape.</li>
</ul>
<p>Some readings about</p>
<ol>
<li><a href="https://pytorch.org/tutorials/" rel="noopener nofollow ugc">PyTorch Tutorials</a></li>
<li><a href="https://pytorch.org/docs/stable/index.html" rel="noopener nofollow ugc">Official PyTorch Documentation</a></li>
<li><a href="https://deeplizard.com/learn/video/UWlFM0R_x6I" rel="noopener nofollow ugc">PyTorch Fundamentals by Deeplizard</a></li>
<li><a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">PyTorch Reshaping and Squeezing Tensors</a></li>
<li><a href="https://128mots.com/2023/09/03/understanding-pytorch-transformations-a-guide-to-reshaping-tensors/" rel="noopener nofollow ugc">Understanding PyTorch Transformations: A Guide to Reshaping Tensors</a></li>
</ol>