Best way to concatenate these tensor dimensions

fermoren: <p>Hello, after reading this post (<a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524" class="inline-onebox">For beginners: Do not use view() or reshape() to swap dimensions of tensors!</a>) regarding the usage of .view() and .reshape(), a doubt has come to my mind.</p>
<p>If I had a tensor ‘A’ of dimensions A=(N, H), being ‘N’ the batch dimensions and ‘H’ the dimensions of a LSTM, and a second tensor ‘B’, of dimensions B=(T) that represents a vector of time event of duration ‘T’, what would be the best way to concatenate both of them to obtain a vector ‘C’ of dimensions C=(N,H,T)?</p>
<p>Thanks!</p>

ksz: <p>You can’t concatenate them in that way.<br>
See tensor ‘A’ contains H numbers N times, tensor ‘B’ only T numbers. How do you want to concatenate first numbers with others?</p>
<p>The only way to concatenate them is to replace all ‘H’ numbers with ‘T’ numbers but ‘H’ times. However in most cases it is a bad idea. You can do it by:</p>
<pre><code class="lang-auto">import torch

A = torch.rand(3, 2)
B = torch.rand(5)
C = B.repeat(A.shape[0], A.shape[1], 1)
</code></pre>
<p>the output will be:</p>
<pre><code class="lang-auto">A torch.Size([3, 2])
tensor([[0.1305, 0.8649],
        [0.2945, 0.3027],
        [0.6107, 0.6925]])
---
B torch.Size([5])
tensor([0.3653, 0.6501, 0.8864, 0.0936, 0.6012])
---
C torch.Size([3, 2, 5])
tensor([[[0.3653, 0.6501, 0.8864, 0.0936, 0.6012],
         [0.3653, 0.6501, 0.8864, 0.0936, 0.6012]],

        [[0.3653, 0.6501, 0.8864, 0.0936, 0.6012],
         [0.3653, 0.6501, 0.8864, 0.0936, 0.6012]],

        [[0.3653, 0.6501, 0.8864, 0.0936, 0.6012],
         [0.3653, 0.6501, 0.8864, 0.0936, 0.6012]]])
</code></pre>

fermoren: <p>Thanks for your answer!</p>
<p>Could it be done more elegantly through a NN? I mean, having two inputs A=(N, H) and B=(T), and desiring to have a third tensor C=(N,H,T) as an output, which could be interpreted as a different vector of dimensions ‘T’ for each of the elements of ‘A’, could I concatenate ‘A’ and ‘B’ though a nn.Linear?</p>
<p>Thanks again!</p>

ksz: <p>Yes, it could be done. However to train NN, you need examples of correct outputs . Sometimes some additional hidden layers are required, not only the single dense layer.</p>
<p>Maybe a better idea is describing the main problem as a whole. Maybe your approach is not well fitted to your original problem. It is possible that we will find a better solution.</p>

fermoren: <p>Thanks again, I will try to briefly summarise the problem and why I need to do this: I’m implementing a small module which is intended to operate as an attention network inside a bigger model.</p>
<p>This attention module has two different inputs: a vector that would be ‘B’ of dimension (F) and a matrix ‘A’, with dimensions (B, E). Combining both inputs, I want to obtain a third tensor ‘C’ of dimensions (B,E,F), which would be the ‘alpha’ of the attention module. That means that this variable ‘C’ multiply other variable ‘D’  of dimensions D=(B,E,F), obtaining the output [also of dimensions (B,E,F)].</p>
<p>Summarised, I need:</p>
<p>C = f(A,B).<br>
C = softmax( C ), to make each element between (0,1).<br>
output = C x D, where ‘x’ would be a dot product attention.</p>
<p>In all this scheme, I’m having troubles finding the optimal way to combine A=(B,E) and B=(F) into a 3D tensor of dimensions (B,E,X) where X could be change into the desired dimensions F with a nn.Linear.</p>
<p>Thanks again!</p>

ksz: <p>You can use this:</p>
<pre><code class="lang-auto">
A = torch.rand(30, 20)
B = torch.rand(10)

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        
        self.linear = torch.nn.Linear(11, 10) #11 -&gt; 1 from x + 10 from y
        self.softmax = torch.nn.Softmax(dim=1)
        
    def forward(self, x, y):
        x = x.unsqueeze(2)
        y = y.repeat(x.shape[0], x.shape[1], 1)
       
        z = torch.cat((x, y), dim = 2)
        z = self.linear(z)
        z = self.softmax(z)
        
        return z
    
net = Network()
C = net(A, B)
print(C.shape)
</code></pre>
<p>Of course you can design the model with additional layers.</p>