How to concatenate feature maps of different batch sizes?

messi: <p>Suppose I have two feature maps <code>F1</code> of shape <code>[K, C, H, W]</code> and <code>F2</code> of shape <code>[M, C, H, W]</code>.<br>
Furthermore, let’s suppose <code>F1</code> is a collection of <code>K</code> feature maps, each of dimension <code>[C, H, W]</code> and <code>F2</code> is a collection of <code>M</code> feature maps, each of dimension <code>[C, H, W]</code>.</p>
<p>Here is the task: For each feature map <code>1 &lt;= i &lt;= K</code> of <code>F1</code>, I want to concatenate <code>i</code> with every feature map <code>1 &lt;= j &lt;= M</code> of <code>F2</code> to obtain <code>MK</code> feature maps of dimensions <code>[2C, H, W]</code>.<br>
In other words, I need to obtain <code>[M, K, 2C, H, W]</code>.</p>
<p>What are some efficient ways to do this?</p>
<p>One way I can think of is the following:</p>
<p><code>F1 = F1.unsquueze(0).repeat(M, 1, 1, 1, 1)      ([M, K, C, H, W])</code><br>
<code>F2 = F2.unsqueeze(1).repeat(1, K, 1, 1, 1)      (M, K ,C, H, W])</code><br>
<code>Fcat = torch.cat([F1, F2], dim=2)  ([M, K, 2C, H, W])</code></p>
<p>Is this a correct  and an efficient way to do this?</p>

vainaijr: <p>maybe,</p>
<pre><code class="lang-auto">k, m, c, h, w = 2, 3, 4, 5, 6 
x = torch.randn(k, c, h, w)
y = torch.randn(m, c, h, w)
a = torch.repeat_interleave(x, m).view(k, m, c, h, w)
b = torch.repeat_interleave(y, k).view(k, m, c, h, w)
ab = torch.cat((a, b), dim=2)
</code></pre>

messi: <p>Is this similar to the following?</p>
<p><code>F1 = F1.unsquueze(0).repeat(M, 1, 1, 1, 1)      ([M, K, C, H, W])</code><br>
<code>F2 = F2.unsqueeze(1).repeat(1, K, 1, 1, 1)      (M, K ,C, H, W])</code><br>
<code>Fcat = torch.cat([F1, F2], dim=2)  ([M, K, 2C, H, W])</code></p>

vainaijr: <p>I think the difference is between repeat_interleave and repeat, when we use,</p>
<pre><code class="lang-auto">a = torch.randn(3, 3)
torch.repeat_interleave(a, 2)
</code></pre>
<p>then it will flatten our ‘a’ matrix first, and then repeat each element two times.<br>
so if ‘a’ is</p>
<pre><code class="lang-auto">tensor([[ 1.0735,  0.9241,  0.5039],
        [ 0.3616, -0.4308,  0.5240],
        [ 1.3007,  0.7306,  0.4381]])
</code></pre>
<p>then</p>
<pre><code class="lang-auto">torch.repeat_interleave(a, 2)
</code></pre>
<p>would give</p>
<pre><code class="lang-auto">tensor([ 0.7962,  0.7962,  0.6094,  0.6094, -0.5283, -0.5283, -0.8183, -0.8183,
        -1.5764, -1.5764, -0.5292, -0.5292,  1.1216,  1.1216, -3.6188, -3.6188,
        -1.1325, -1.1325])
</code></pre>
<p>after reshaping this, we would see matrix like,</p>
<pre><code class="lang-auto">torch.repeat_interleave(a, 2).reshape(2, 3, 3)
</code></pre>
<pre><code class="lang-auto">tensor([[[ 1.0735,  1.0735,  0.9241],
         [ 0.9241,  0.5039,  0.5039],
         [ 0.3616,  0.3616, -0.4308]],

        [[-0.4308,  0.5240,  0.5240],
         [ 1.3007,  1.3007,  0.7306],
         [ 0.7306,  0.4381,  0.4381]]])
</code></pre>
<p>while if we use</p>
<pre><code class="lang-auto">a.repeat(2, 2)
</code></pre>
<p>then, we get,</p>
<pre><code class="lang-auto">tensor([[ 1.0735,  0.9241,  0.5039,  1.0735,  0.9241,  0.5039],
        [ 0.3616, -0.4308,  0.5240,  0.3616, -0.4308,  0.5240],
        [ 1.3007,  0.7306,  0.4381,  1.3007,  0.7306,  0.4381],
        [ 1.0735,  0.9241,  0.5039,  1.0735,  0.9241,  0.5039],
        [ 0.3616, -0.4308,  0.5240,  0.3616, -0.4308,  0.5240],
        [ 1.3007,  0.7306,  0.4381,  1.3007,  0.7306,  0.4381]])
</code></pre>
<p>if we want same element to appear in nearby index, then we use repeat_interleave, otherwise we use repeat.</p>

messi: <p>So among the above two methods, which one do you think is a correct way to approach the problem?</p>

vainaijr: <p>maybe repeat appears better, I cannot give a strong argument to support one against another, we can take an example to see which one might be better, suppose we have,</p>
<pre><code class="lang-auto">a = torch.randn(3, 3); a
</code></pre>
<pre><code class="lang-auto">tensor([[-1.6339,  0.5842,  0.3604],
        [ 0.2360,  0.5282,  1.8202],
        [-0.2033, -0.4426, -0.0329]])
</code></pre>
<p>suppose we wanted a repeated tensor of shape (2, 3, 3)<br>
so one way would be, we do,</p>
<pre><code class="lang-auto">torch.repeat_interleave(a, 2)
</code></pre>
<p>and get,</p>
<pre><code class="lang-auto">tensor([[[-1.6339, -1.6339,  0.5842],
         [ 0.5842,  0.3604,  0.3604],
         [ 0.2360,  0.2360,  0.5282]],

        [[ 0.5282,  1.8202,  1.8202],
         [-0.2033, -0.2033, -0.4426],
         [-0.4426, -0.0329, -0.0329]]])
</code></pre>
<p>another way would be,</p>
<pre><code class="lang-auto">torch.repeat_interleave(a, torch.tensor([2]), dim=0).reshape(2, 3, 3)
</code></pre>
<p>we get,</p>
<pre><code class="lang-auto">tensor([[[-1.6339,  0.5842,  0.3604],
         [-1.6339,  0.5842,  0.3604],
         [ 0.2360,  0.5282,  1.8202]],

        [[ 0.2360,  0.5282,  1.8202],
         [-0.2033, -0.4426, -0.0329],
         [-0.2033, -0.4426, -0.0329]]])
</code></pre>
<p>or we use repeat,</p>
<pre><code class="lang-auto">a.repeat(2, 1).reshape(2, 3, 3)
</code></pre>
<pre><code class="lang-auto">tensor([[[-1.6339,  0.5842,  0.3604],
         [ 0.2360,  0.5282,  1.8202],
         [-0.2033, -0.4426, -0.0329]],

        [[-1.6339,  0.5842,  0.3604],
         [ 0.2360,  0.5282,  1.8202],
         [-0.2033, -0.4426, -0.0329]]])
</code></pre>
<p>if we were applying something like, MaxPool after this, then in the first two cases, it would consider something like this, (if we had 2x2 window)</p>
<pre><code class="lang-auto">[-1.6339,  0.5842]
[-1.6339,  0.5842]
</code></pre>
<p>which appears a bit weird comparing same terms.</p>
<p>or if we were applying Dropout, maybe dropping 2x2 blocks at random, then,<br>
entire block of,</p>
<pre><code class="lang-auto">[-1.6339,  0.5842]
[-1.6339,  0.5842]
</code></pre>
<p>might become zero, and we would not have those values anywhere else, this again appears a bit weird</p>
<p>or if we were applying something like nn.LayerNorm after this,<br>
so, if we have a block which has a lot of same values, like,</p>
<pre><code class="lang-auto">a = torch.tensor([[1., 1., 1.], [2., 2., 2.]])
b = nn.LayerNorm(3)
</code></pre>
<p>then, b(a) would give</p>
<pre><code class="lang-auto">tensor([[0., 0., 0.],
        [0., 0., 0.]], grad_fn=&lt;NativeLayerNormBackward&gt;)
</code></pre>
<p>maybe we do not want to zero out most of the blocks.</p>
<p>it depends on what we apply after this repeat.</p>