Concat two tensors with different dimensions

Lame: <p>Hi all,</p>
<p>Is it possible to concat two tensors have different dimensions?</p>
<p>for example: if <code>A of shape = [16, 512]</code> and <code>B of shape = [16, 32, 2048]</code></p>
<p>How they could combined to be of shape <code>[16, 544, 2048]</code> ?</p>
<p>Any help/suggestion, please?</p>

ptrblck: <p>I’m not sure, how you would like to fill dim2 in your first tensor, but if you just want to repeat the values, this code would work:</p>
<pre><code class="lang-python">a = torch.randn(16, 512)
b = torch.randn(16, 32, 2048)
a = a.unsqueeze(2).expand(-1, -1, 2048)
c = torch.cat((a, b), dim=1)
print(c.shape)
&gt; torch.Size([16, 544, 2048])
</code></pre>

saba: <p>Hi Ptrblck,<br>
I hope you are well. Sorry I need to concatenate two tensors x and y with the size of 64x100x9x9. 64 is batch size, 100 is the number of channel and 9x9x is the width and height.<br>
I want to have concatenated result in 64x100x18x18 form<br>
I used torch.cat[x,y],dim??)</p>

ptrblck: <p>You won’t be able to create the output as <code>[64, 100, 18, 18]</code>, since this would double the number of all available elements.<br>
However, you could create a tensor in the shape <code>[64, 100, 18, 9]</code> or <code>[64, 100, 9, 18]</code> via:</p>
<pre><code class="lang-python">x = torch.randn(64, 100, 9, 9)
y = torch.randn(64, 100, 9, 9)

print(torch.cat((x, y), dim=2).shape)
print(torch.cat((x, y), dim=3).shape)
</code></pre>

AM.MO: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="61977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>, dim=1</code></p>
</blockquote>
</aside>
<p>I have these shapes and I want to use torch.cat<br>
torch.Size([64, 27, 27]) torch.Size([3, 224, 224]) torch.Size([192, 13, 13])<br>
but i got this error<br>
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 64 but got size 3 for tensor number 1 in the list.<br>
I need to visualize them that’s why I need to reshape first and last tensor to the middle one ([3, 224, 224])</p>

ptrblck: <p>You won’t be able to concatenate these tensors, as all dimensions differ in their size (as also given in the error message).<br>
To concatenate tensors all dimensions besides that one used for concatanation must be equal:</p>
<pre><code class="lang-python">a = torch.randn(2, 224, 224)
b = torch.randn(5, 224, 224)
c = torch.cat((a, b), dim=0) # works since only dim0 differs in size
</code></pre>

AM.MO: <p>Actually, I’m trying to visualize the feature of an image using the pre_trained alexnet, and UndoableConvLayer class should transpose the convolution, I think It should return the tensor as same as image shape but I don’t know how can I do that</p>
<pre><code class="lang-auto">net = models.alexnet(pretrained=True)
layer1 = UndoableConvLayer(net.features[0], net.features[2])
layer2 = UndoableConvLayer(net.features[3], net.features[5])

print(x_im[0].shape)
print(layer1(x_im)[0][0].shape)
torch.Size([3, 224, 224])
torch.Size([64, 27, 27])
</code></pre>

ptrblck: <p>I don’t know what <code>UndoableConvLayer</code> is doing internally, but it seems it returns a 5-dimensional tensor in a shape which is neither the input shape nor compatible to an image format so you might want to revisit this layer’s implementation.</p>

AM.MO: <pre><code class="lang-auto">class UndoableConvLayer(nn.Module):
    def __init__(self, conv: nn.Conv2d, pool: nn.MaxPool2d):

        super().__init__()

        self._cache = None

        self.conv = conv

        self.pool = pool

        self.pool.return_indices = True

    @torch.no_grad()

    def undo(self, a: torch.Tensor, idx: torch.Tensor, out_pad: int = 0) -&gt; torch.Tensor:

        from torch.nn.functional import max_unpool2d, conv_transpose2d

        # self.pool.return_indices = True

        x = max_unpool2d(a,indices=idx)

        x = conv_transpose2d(x,output_padding=out_pad)

        return x

    @torch.no_grad()

    def forward(self, x):

        s = self.conv(x)

        out, idx = self.pool(s)

        a = torch.relu(out)

        return a, idx
</code></pre>
<p>Now I understand where is the problem, but I didn’t solve it yet, when I pass the image through ```<br>
layer1</p>
<pre data-code-wrap="it"><code class="lang-nohighlight">It's first time I face this problem and I tried really long time but I couldn't solve it</code></pre>