Pytorch tensor dimension wise concatenation

Nafees: <p>I need to achieve my targets by developing computationally efficient code. Here is the scenario:</p>
<pre><code>x     # torch.Size([2, 5, 256]) (Batch, input_1st_dim, input_2nd_dim)
</code></pre>
<p>Now I want to concatenate the <code>input_1st_dim</code> index-wise, like <em>1st</em> with all the following <em>four</em>, then <code>2nd</code> with the next <code>three</code>, then <code>3rd</code> with the next <code>two</code>, then <code>4th</code> with the last <code>one</code>.<br>
Finally, I want to get <code>x</code> as <code>[2, 15, not sure about this dimension]</code></p>
<p>I can do it with nested loops and some extra lines of code. Here is my code:</p>
<pre><code class="lang-auto">   def my_fun(self, x):
        iter = x.shape[0]
        counter = 0
        new_x = torch.zeros((10, x.shape[1]), dtype=torch.float32, device=torch.device('cuda'))
        for i in range(0, x.shape[0] - 1):
            iter -= 1
            for j in range(0, iter):
                mean = (x[i, :] + x[i+j, :])/2
                new_x[counter, :] = torch.unsqueeze(mean, 0)
                counter += 1
        final_T = torch.cat((x, new_x), dim=0)
        return final_T
</code></pre>
<pre><code class="lang-auto">ref = torch.zeros((x.shape[0], 15, x.shape[2]), dtype=torch.float32, device=torch.device('cuda'))
for i in range (x.shape[0]):
    ref[i, :, :] = self.my_fun(x[i, :, :])
</code></pre>
<p>But, is there any computationally efficient way to code it?</p>

ptrblck: <aside class="quote no-group" data-username="Nafees" data-post="1" data-topic="171712">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f07891/48.png" class="avatar"> Nafees:</div>
<blockquote>
<p>I can do it with nested loops and some extra lines of code.</p>
</blockquote>
</aside>
<p>Could you post this slow approach so that we could use it as a reference to compare potentially faster approaches, please?</p>

Nafees: <p>I stuck with one error:<br>
<code>x.shape  #  torch.Size([5, 256])</code><br>
<code>new_x.shape # torch.Size([10, 256])</code><br>
Now I want concatenate these two tensors row wise means in result want to get (15, 256) and for that I am using <code>final_T = torch.cat((x, new_x), -1,  device='cuda')</code>, but getting error:</p>
<p>TypeError: cat() received an invalid combination of arguments - got (tuple, int, device=str), but expected one of:</p>
<ul>
<li>(tuple of Tensors tensors, int dim, *, Tensor out)</li>
<li>(tuple of Tensors tensors, name dim, *, Tensor out)`</li>
</ul>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x = torch.randn(5, 256)
new_x = torch.randn(10, 256)
out = torch.cat((x, new_x), dim=0)
print(out.shape)
# torch.Size([15, 256])
</code></pre>

Nafees: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="171712">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>out = torch.cat((x, new_x), dim=0)</code></p>
</blockquote>
</aside>
<p>your examples work, but in my example case its does not work. I tried <code>final_T = torch.cat((x, new_x), dim=0, device='cuda')</code></p>
<p><strong>Detailed Code is:</strong><br>
iter = x.shape[0]<br>
counter = 0<br>
new_x = torch.zeros((10, x.shape[1]), dtype=torch.float32)<br>
for i in range(0, x.shape[0] - 1):<br>
iter -= 1<br>
for j in range(0, iter):<br>
mean = (x[i, :] + x[i+j, :])/2<br>
new_x[counter, :] = torch.unsqueeze(mean, 0)<br>
counter += 1<br>
final_T = torch.cat((x, new_x), dim=0)</p>

ptrblck: <p>Remove the <code>device</code> in <code>torch.cat</code> argument as it’s unexpected.</p>

Nafees: <p>Thanks for efficiently responding. But after removing <code>device</code>, I got:<br>
<code>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument tensors in method wrapper___cat)</code></p>

ptrblck: <p>In that case move the <code>CPUTensor</code> to the <code>device</code> via <code>tensor.to(device)</code>, but don’t pass it as an argument to <code>torch.cat</code>.</p>

Nafees: <p>Thank you so much. Now its work.</p>
<p>I finished my version and updated the post with my code. Kindly check whether there is a more efficient way to accomplish this. Furthermore, please double-check that my logic matches the problem stated in the post. Thank you very much.</p>

Nafees: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Waiting for your response. Thanks</p>