How to concatenate 2D tensor with 3D tensor without changing the content

username: Ammar_ul_hassan
cooked: <p>I have two tensors A and B of the following sizes</p>
<pre><code>tensor_A = [16, 100]
tensor_B = [16, 3, 64, 64]
</code></pre>
<p>I want to concatenate them on the tensor_B’s channel axis (i.e. 3). First I reshaped tensor_A to have 4 dimensions and then multiplied it with torch.ones() tensor with the desired shape of tensor_B with the following code.</p>
<pre><code>tensor_A  = tensor_A .view(16, 100, 1, 1)
tensor_A  = tensor_A  * torch.ones(16, 100, 64, 64, device='cuda')
concat = torch.cat((tensor_A  , tensor_B ), dim=1)
</code></pre>
<p>I am able to concatenate the two tensors but I am not sure whether</p>
<ol>
<li>It is the right way to do or does there exist a better solution?</li>
<li>By doing so does the meaning of the 2D tensor_A change?</li>
</ol>

username: ZimoNitrome
cooked: <p>What is the meaning of tensor_A?</p>
<p>I can assume that tensor_B is a batch of 64x64 RGB images. Your operation creates a batch of 100 channel 64x64 images (where these 64x64 pixels all have the same value) and adds those channels to each of the tensor_B images.</p>
<p>A more efficient operation would be to change:</p>
<pre><code class="lang-python">tensor_A = tensor_A.view(16, 100, 1, 1)
tensor_A = tensor_A * torch.ones(16, 100, 64, 64, device=‘cuda’)
</code></pre>
<p>to:</p>
<pre><code class="lang-python">tensor_A = tensor_A.view(16, 100, 1, 1).expand(16, 100, 64, 64)
</code></pre>

username: Ammar_ul_hassan
cooked: <p>tensor_A is a batch of 100-dimensional class features extracted from an encoder. I want to concatenate the batch of features (tensor_A) and the batch of images (tensor_B).</p>

username: ZimoNitrome
cooked: <p>Then this method might work alright.</p>
<p>A problem in the original solution could have been that the output tensor called <code>concat</code> would very large since <code>tensor_A</code> would be 64x64=4096 times larger than what it needed to be. Using only <code>.view()</code> and <code>.expand()</code> will probably fix this problem as the 4096 repeated values are not physical but are just references to the same memory addresses.</p>
<p>Another idea could be to concatenate these inputs later in a neural net. I.e. if you are using a convolutional neural network for your images you can have an encoder part that extract information to a linear layer, at which point you add in these 1 dimensional features from tensor_B before subsequent linear layers.</p>

username: Ammar_ul_hassan
cooked: <p>How about if I do the following operation to replicate spatially and concatenate class information.</p>
<pre><code>tensor_A = tensor_A.view(tensor_A .size(0), tensor_A.size(1), 1, 1)
tensor_A = tensor_A.repeat(1, 1, tensor_B.size(2), tensor_B.size(3))
concat = torch.cat([tensor_A , tensor_B], dim=1)
</code></pre>
<p>Which one of the two solutions (the one you proposed and this) seems more logical considering that I want to be computationally efficient and I want to concatenate the extracted class information (tensor_A) into tensor_B.</p>

username: ZimoNitrome
cooked: <p><code>torch.expand()</code> is more efficient than <code>torch.repeat()</code> memory wise</p>
<aside class="quote" data-post="2" data-topic="27969">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-repeat-and-torch-expand-which-to-use/27969/2">Torch.repeat and torch.expand which to use?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi, 
Backprop-wise, they will give the exact same result. 
The difference is that if the original dimension you want to expand is of size 1, you can use torch.expand() to do it without using extra memory. 
If the dimension you want to expand is of size more than 1, then you actually want to repeat what is at that dimension and you should use torch.repeat(). It will use extra memory (there is no way around that).
  </blockquote>
</aside>