RuntimeError: Sizes of tensors must match in dimenension 1

Phanikumar2all: <p>RuntimeError: Sizes of tensors must match in dimension 1. Expected size 3 but got 16 for tensor number 1 in the list. This error is occurring at line y = torch.cat([x,h],1). Please help to resolve this.</p>
<p>def forward(self,x,h,c):<br>
x = self.W(x)<br>
y = torch.cat([x,h],1)<br>
i = self.Wy(y)<br>
b = self.Wi(i)<br>
ci = torch.sigmoid(self.Wbi(b)+c<em>self.Wci(c))<br>
cf = torch.sigmoid(self.Wbf(b)+c</em>self.Wcf(c))<br>
cc = cf<em>c+ci</em>self.relu(self.Wbc(b))<br>
c0 = torch.sigmoid(self.Wb0(b)+cc*self.Wc0)<br>
ch = c0+self.relu(cc)<br>
return cc, ch</p>

ptrblck: <p><code>torch.cat</code> will try to concatenate multiple tensors in the specified dimension <code>dim</code>, but expects the tensors to have the same shape in all other dimensions.<br>
Here is a small example:</p>
<pre><code class="lang-python"># works, since dim0 has the same shape
x = torch.randn(3, 2)
h = torch.randn(3, 5)
y = torch.cat([x, h], 1) 

# fails, since both dims have a different shape
x = torch.randn(3, 2)
h = torch.randn(16, 5)
y = torch.cat([x, h], 1) 
# RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 3 but got size 16 for tensor number 1 in the list.
</code></pre>

Phanikumar2all: <p>Shape of x is [3,1024,10,10] and the shape of h is [16,256,10,10].<br>
I tried to reshape ‘x’ to [16,1024,10,10] by using<br>
x1 = x.repeat(5,1,1,1)<br>
x = torch.cat([x1,torch.zeros(1,1024,10,10)],0).<br>
Now, this give shape of x as [16,1024,10,10]. And, now, this gives out other error as “Runtimeerror: Sizes of tensors must match except in dimension 0. Expected size 256 but got size 1024 for tensor number 1 in the list”.</p>
<p>One more thing is that even if tried to reshape ‘h’ and keeping ‘x’ intact. The above error comes out in other way as “Runtimeerror: Sizes of tensors must match except in dimension 0. Expected size 1024 but got size 256 for tensor number 1 in the list”</p>

Phanikumar2all: <p>Could you please help?</p>

ptrblck: <aside class="quote no-group" data-username="Phanikumar2all" data-post="3" data-topic="159637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phanikumar2all/48/22977_2.png" class="avatar"> Phanikumar2all:</div>
<blockquote>
<p>Now, this give shape of x as [16,1024,10,10]. And, now, this gives out other error as “Runtimeerror: Sizes of tensors must match except in dimension 0. Expected size 256 but got size 1024 for tensor number 1 in the list”.</p>
</blockquote>
</aside>
<p>You are most likely trying to concatenate the tensors in <code>dim0</code>, while the original use case used <code>dim1</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn([3,1024,10,10])
h = torch.randn([16,256,10,10])

x1 = x.repeat(5,1,1,1)
x = torch.cat([x1,torch.zeros(1,1024,10,10)],0)

# your error
out = torch.cat((x, h), dim=0) 
# RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 1024 but got size 256 for tensor number 1 in the list.

# works
out = torch.cat((x, h), dim=1)
print(out.shape)
# torch.Size([16, 1280, 10, 10])
</code></pre>

Phanikumar2all: <p>Thanks for the help. But, it gives out other error as “RuntimeError: Expected all tensors to be on the same device, but found atleast two devices, cuda:2 and cuda:0! (when checking argument for argument tensors in method wrapper_cat)”.</p>
<p>I have used only GPU 2. But, still this comes out as an error.</p>

ptrblck: <p>Check the <code>.device</code> of both tensors and make sure they are on the same GPU.</p>

Phanikumar2all: <p>I have set the device now. But, I think the issues are coming up since the data loading isn’t happening properly. I need to load a batch size of 16 images (with 3 channels) as the input. But, the shape of x in the above code says only 3 images taken as its size is [3,1024,10,10]. How can I solve this issue?. Please help.</p>

ptrblck: <p>The data loading shouldn’t change the device at all, so I would recommend to fix the issues one-by-one as it seems you are now mixing up different errors.</p>
<p>Check the <code>.device</code> attribute of all involved tensors and in case they are using an unexpected GPU id, narrow down where this tensor is coming from, which part of the code set its device, and fix it.</p>
<p>Afterwards, check the <code>DataLoader</code> and make sure each batch has the expected shape. Once this is done, check my code to see how tensors with a different shape in one dimension can be concatenated to a single tensor.</p>

Phanikumar2all: <p>Hi. Now, the device attribute is set and working fine. I did the following modifications to the code:<br>
x1 = x.repeat(5,1,1,1)<br>
zero_ten = torch.zeros(1,1024,10,10).to(“cuda:0”)<br>
x = torch.cat([x1,zero_ten],0)<br>
print(x.shape)<br>
x = self.W(x)<br>
print(h.shape)<br>
y = torch.cat([x, h],1) <span class="hashtag">#concatenate</span> input and hidden layers<br>
print(y.shape)<br>
i = self.Wy(y) <span class="hashtag">#reduce</span> to hidden layer size<br>
b = self.Wi(i)	<span class="hashtag">#depth</span> wise 3*3<br>
ci = torch.sigmoid(self.Wbi(b) + c * self.Wci)<br>
cf = torch.sigmoid(self.Wbf(b) + c * self.Wcf)<br>
cc = cf * c + ci * self.relu(self.Wbc(b))<br>
co = torch.sigmoid(self.Wbo(b) + cc * self.Wco)<br>
ch = co * self.relu(cc)</p>
<p>Now, I can see an error: Sizes of tensors must match in dimension 1. Expected size 81 but got size 16 for tensor number 1 in the list.</p>
<p>The sizes of x and h are [81,256,5,5] and [16,64,5,5], respectively.</p>

ptrblck: <aside class="quote no-group" data-username="Phanikumar2all" data-post="10" data-topic="159637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phanikumar2all/48/22977_2.png" class="avatar"> Phanikumar2all:</div>
<blockquote>
<p>The sizes of x and h are [81,256,5,5] and [16,64,5,5], respectively.</p>
</blockquote>
</aside>
<p>Unfortunately, this won’t work since you are again trying to concatenate two tensors which have different sizes in more than a single dimension.<br>
In particular, since you want to concatenate <code>x</code> and <code>h</code> in <code>dim1</code> make sure that all other dimensions have exactly the same size as it would fail otherwise. Based on your code I would guess the <code>repeat</code> approach might not be the right solution and you would need to check why the tensors have differ in multiple dimensions and what the concatenation should actually achieve using these two tensors.</p>

Phanikumar2all: <p>Thank you for the answer</p>