Rearrange 1d tensor with overlapping

seyeeet: <p>I read the unfold documentation but I fail to understand how to use it for my following case.<br>
I have the following case where<br>
<code>tmp</code> is a <code>1D</code> tensor and <code>step</code> is the overlapping parameter.</p>
<p>I have the following loop that concat the <code>tmp</code> based on the <code>step (ovedrlap)</code> as follow</p>
<pre><code class="lang-auto">N = 10
tmp = torch.arange(0,16*N).reshape(1, 16,N ) 
step=2 # this can change to 1,3,.ect
for i in range(N):
     print(i*step,(i+1)*step+(step//2)*2)
tmp = torch.cat([tmp[:,:, i*step:(i+1)*step+(step//2)*2] for i in range(N) if (i+1)*step+(step//2)*2 &lt; tmp.shape[2]], dim=0) 
</code></pre>
<p>is there a way to do such a task via unfolding? or any other way?</p>
<p>the original <code>tmp</code> looks like this:</p>
<pre><code class="lang-auto">tensor([[[  0,   1,   2,   3,   4,   5,   6,   7,   8,   9],
         [ 10,  11,  12,  13,  14,  15,  16,  17,  18,  19],
         [ 20,  21,  22,  23,  24,  25,  26,  27,  28,  29],
         [ 30,  31,  32,  33,  34,  35,  36,  37,  38,  39],
         [ 40,  41,  42,  43,  44,  45,  46,  47,  48,  49],
         [ 50,  51,  52,  53,  54,  55,  56,  57,  58,  59],
         [ 60,  61,  62,  63,  64,  65,  66,  67,  68,  69],
         [ 70,  71,  72,  73,  74,  75,  76,  77,  78,  79],
         [ 80,  81,  82,  83,  84,  85,  86,  87,  88,  89],
         [ 90,  91,  92,  93,  94,  95,  96,  97,  98,  99],
         [100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
         [110, 111, 112, 113, 114, 115, 116, 117, 118, 119],
         [120, 121, 122, 123, 124, 125, 126, 127, 128, 129],
         [130, 131, 132, 133, 134, 135, 136, 137, 138, 139],
         [140, 141, 142, 143, 144, 145, 146, 147, 148, 149],
         [150, 151, 152, 153, 154, 155, 156, 157, 158, 159]]])
</code></pre>
<p>and the final version looks like this (shape=<code>torch.Size([3, 16, 4])</code>):</p>
<pre><code class="lang-auto">tensor([[[  0,   1,   2,   3],
         [ 10,  11,  12,  13],
         [ 20,  21,  22,  23],
         [ 30,  31,  32,  33],
         [ 40,  41,  42,  43],
         [ 50,  51,  52,  53],
         [ 60,  61,  62,  63],
         [ 70,  71,  72,  73],
         [ 80,  81,  82,  83],
         [ 90,  91,  92,  93],
         [100, 101, 102, 103],
         [110, 111, 112, 113],
         [120, 121, 122, 123],
         [130, 131, 132, 133],
         [140, 141, 142, 143],
         [150, 151, 152, 153]],

        [[  2,   3,   4,   5],
         [ 12,  13,  14,  15],
         [ 22,  23,  24,  25],
         [ 32,  33,  34,  35],
         [ 42,  43,  44,  45],
         [ 52,  53,  54,  55],
         [ 62,  63,  64,  65],
         [ 72,  73,  74,  75],
         [ 82,  83,  84,  85],
         [ 92,  93,  94,  95],
         [102, 103, 104, 105],
         [112, 113, 114, 115],
         [122, 123, 124, 125],
         [132, 133, 134, 135],
         [142, 143, 144, 145],
         [152, 153, 154, 155]],

        [[  4,   5,   6,   7],
         [ 14,  15,  16,  17],
         [ 24,  25,  26,  27],
         [ 34,  35,  36,  37],
         [ 44,  45,  46,  47],
         [ 54,  55,  56,  57],
         [ 64,  65,  66,  67],
         [ 74,  75,  76,  77],
         [ 84,  85,  86,  87],
         [ 94,  95,  96,  97],
         [104, 105, 106, 107],
         [114, 115, 116, 117],
         [124, 125, 126, 127],
         [134, 135, 136, 137],
         [144, 145, 146, 147],
         [154, 155, 156, 157]]])
</code></pre>

tom: <pre><code class="lang-python">a = torch.arange(160).view(1, 16, 10)
tmp = a[0].unfold(1, 6, 1)[:,:4,::2].permute(2, 0, 1)
print(tmp)
</code></pre>
<p>Not that I’d know why. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best regards</p>
<p>Thomas</p>

seyeeet: <p>Hi Thomas</p>
<p>Thanks a lot for your answer! It is actually doing it correctly and much faster. I dont know how did you figure it out, but I really appreciate it anyways.<br>
can I ask what is the strategy behind choosing <code>6</code> and <code>:4</code> and <code>::2</code> in <code>.unfold(1, 6, 1)[:,:4,::2]</code> like how should they change for another step (e.g <code>8</code>).</p>
<p>Just since you asked why I want to do it, I am trying to develop something similar to slide window for attention. but I am not sure if it gonna work yet <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>

tom: <p>So I knew <code>Tensor.unfold(dim, size, step)</code> (which is different from nn.functional.unfold) can produce the type of overlapping patches you want. Initially, I thought it would be step 2 but this will make the 0 2 4 6, so we need step 1 and then skip every other slice with the <code>::2</code>. The size = 6 is because you asked for 3 outputs but as we are skipping every other we need 6 (probably 5 would do in hindsight). Then the <code>:4</code> is to limit each row to 4 elements.</p>
<p>There also is the - perhaps more straightforward alternative - to use <code>a.as_strided((3, 16, 4), (2, 10, 1))</code> to specify the layout in memory directly but I am never sure how much to trust <code>as_strided</code> with autograd. Quite likely, everything just works, but at some point in the distant past, it did something that I found strange. In that sense Tensor.unfold seemed more natural even if it looks like black magic with all these numbers derived through iterative fiddeling.</p>
<p>Best regards</p>
<p>Thomas</p>

seyeeet: <p>Thanks a lot for the clarification! I think I understand it now much better <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I was wondering if you any suggestion for this case if now I want to use <code>fold</code> to make it back.<br>
Hmmm I think we also dont have <code>x.fold</code> in pytorch?</p>