Merge two 2D tensors, into a 3D tensor

username: Elidor
cooked: <p>Hi everyone,</p>
<p>I have two 2D tensors of shape:</p>
<pre><code class="lang-auto">y = torch.Size([50, 61])   # (batch_size, max_len)
x = torch.Size([50, 800])  # (batch_size, n_lstm_units)
</code></pre>
<p>what I would like to get is a 3D tensor made like this:</p>
<pre><code class="lang-auto">z = (50, 61, 800)  # (batch_size, max_len, n_lstm_units)
</code></pre>
<p>how can I do?</p>
<p>What I tried to do was to increase the size of <code>x</code> and <code>y</code> via <code>unsqueeze()</code>, resulting in <code>(50, 61, 1)</code> and <code>(50, 1, 800)</code>. The problem is that afterwards I don’t know how to join the tensors without concatenating the values but simply “transferring” a dimension to obtain <code>(50, 61, 800)</code> (the idea is not to concatenate the values, but only to “move” a dimension from one tensor to another … I don’t know if I get the idea).</p>

username: KFrank
cooked: <p>Hello Elidor!</p>
<aside class="quote no-group" data-username="Elidor" data-post="1" data-topic="104743" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/bc8723/48.png" class="avatar"> Elidor:</div>
<blockquote>
<p>I have two 2D tensors of shape:</p>
<pre><code class="lang-auto">y = torch.Size([50, 61])   # (batch_size, max_len)
x = torch.Size([50, 800])  # (batch_size, n_lstm_units)
</code></pre>
<p>what I would like to get is a 3D tensor made like this:</p>
<pre><code class="lang-auto">z = (50, 61, 800)  # (batch_size, max_len, n_lstm_units)
</code></pre>
<p>how can I do?</p>
</blockquote>
</aside>
<p>I’m not sure what you’re asking here.</p>
<p>Please note that your <code>x</code> and <code>y</code> together have <code>50 * (61 + 800)</code><br>
elements, while your desired <code>z</code> will have <code>50 * 61 * 800</code> elements,<br>
a much larger number.  So you can’t simply populate <code>z</code> with elements<br>
of <code>x</code> and <code>y</code>.</p>
<p>I have no idea if this would be what you want, but you can make a<br>
tensor of the desired shape by taking the <em>batch outer product</em>:</p>
<pre data-code-wrap="python"><code class="lang-python">z = torch.bmm (x.unsqueeze (2), y.unsqueeze (1))
</code></pre>
<p>(Pytorch does have an outer-product function, <a href="https://pytorch.org/docs/1.7.1/generated/torch.ger.html" rel="noopener nofollow ugc">torch.ger, torch.outer</a>,<br>
but it only works on vectors, hence the <code>unsqueeze()</code>, above.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Elidor
cooked: <p>Hello <a class="mention" href="/u/kfrank">@KFrank</a> !<br>
Thank you for your answer. I try to explain a little better what my problem is because maybe I haven’t been very detailed.</p>
<p>At the beginning of my code, I have a tensor with shape <code>x = torch.Size([50, 61, 140])</code> <code>(batch_size, seq_len, embedding_dim)</code> and a ndarray <code>x_len = (50,)</code>. These two vectors I give them as input to a bilstm whose code is the following (I’ll save you all the forward function, I don’t think it’s important):</p>
<pre><code class="lang-auto">x = nn.utils.rnn.pack_padded_sequence(x, x_len, batch_first=True)
out1 = self.lstm1(x)
x, lengths = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
</code></pre>
<p>After this code I have two tensors with shape <code>x = ([50, 61, 800])</code> and <code>x_len = ([50])</code>.<br>
I give these two vectors as input to a module to calculate the attention in the following way (<code>x</code> is the parameter corresponding to <code>inputs</code> and <code>x_len</code> is the one corresponding to <code>lengths</code>):</p>
<pre><code class="lang-auto">def forward(self, inputs, lengths):
    batch_size, max_len = inputs.size()[:2]  # batch_size = 50, max_len = 61

    # matrix mult
    # apply attention layer
    weights = torch.bmm(inputs,
                        self.att_weights  # (1, hidden_size)
                        .permute(1, 0)  # (hidden_size, 1)
                        .unsqueeze(0)  # (1, hidden_size, 1)
                        .repeat(batch_size, 1, 1)  # (batch_size, hidden_size, 1)
                        )

    # weights.shape = torch.Size([50, 61, 1])
    attentions = torch.softmax(F.relu(weights.squeeze()), dim=-1)   # torch.Size([50, 61])

    # create mask based on the sentence lengths
    mask = torch.ones(attentions.size(), requires_grad=True).cuda()  # torch.Size([50, 61])
    for i, l in enumerate(lengths):  # skip the first sentence
        if l &lt; max_len:
            mask[i, l:] = 0

    # apply mask and renormalize attention scores (weights)
    masked = attentions * mask
    _sums = masked.sum(-1).unsqueeze(-1)  # sums per row    # torch.Size([50, 1])

    attentions = masked.div(_sums)   # torch.Size([50, 61])

    # apply attention weights
    weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))   # torch.Size([50, 61, 800])

    # get the final fixed vector representations of the sentences
    representations = weighted.sum(1).squeeze()   # ([50, 800])

    return representations, attentions   # representations =  ([50, 800]) , attentions = ([50, 61])
</code></pre>
<p>What I would like to obtain as the output of this forward function is a tensor with shape <code>([50, 61, 800])</code> instead of two tensors made in that way.</p>
<p>Do you think there is a way to do it without “affecting” the attention calculation?</p>
<p>Thank you so much,</p>
<p>best regards.</p>

username: KFrank
cooked: <p>Hello Elidor!</p>
<aside class="quote no-group" data-username="Elidor" data-post="3" data-topic="104743" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/bc8723/48.png" class="avatar"> Elidor:</div>
<blockquote>
<p>What I would like to obtain as the output of this forward function is a tensor with shape <code>([50, 61, 800])</code> instead of two tensors made in that way.</p>
<p>Do you think there is a way to do it without “affecting” the attention calculation?</p>
</blockquote>
</aside>
<p>I don’t have any advice for you, as I don’t have any intuition about what<br>
these values might mean.</p>
<p>Perhaps if you could give a very high level description of your use case,<br>
some expert in that area might have suggestions.  Are you working on<br>
some sort of (semi-) standard model?  Is this a research direction that<br>
others might have experience with?</p>
<p>Let me also reiterate that, at a lower level, a core issue is the amount<br>
of “information” you have.  If you want to turn <code>[50, 61]</code> and <code>[50, 800]</code><br>
into <code>[50, 61, 800]</code>, you either have to get the additional information<br>
from somewhere else, or your <code>[50, 61, 800]</code> has to encode the limited<br>
information you have in some redundant way (such as using  the outer<br>
product I mentioned earlier).</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Elidor
cooked: <blockquote>
<p>Perhaps if you could give a very high level description of your use case,<br>
some expert in that area might have suggestions. Are you working on<br>
some sort of (semi-) standard model? Is this a research direction that<br>
others might have experience with?</p>
</blockquote>
<p>Ok, it seems like a good idea, so i try.<br>
I started studying this model for dependency parsing. Its results are good:</p>
<pre><code class="lang-auto">Model(
  (dropout): Dropout(p=0.6, inplace=False)
  (word_embedding): Embedding(25413, 100, padding_idx=0)
  (tag_embedding): Embedding(20, 40, padding_idx=0)
  (bilstm): LSTM(908, 600, num_layers=3, batch_first=True, dropout=0.3, bidirectional=True)
  (bilstm_to_hidden1): Linear(in_features=1200, out_features=500, bias=True)
  (hidden1_to_hidden2): Linear(in_features=500, out_features=150, bias=True)
  (hidden2_to_pos): Linear(in_features=150, out_features=101, bias=True)
  (hidden2_to_dep): Linear(in_features=300, out_features=47, bias=True)
)
</code></pre>
<p>The bilstm input is formed by:</p>
<pre><code class="lang-auto"># INPUT: x = torch.Size([50, 61, 140]) and x_lengths = (50,)
x = torch.nn.utils.rnn.pack_padded_sequence(x, x_lengths, batch_first=True)
x, _ = self.bilstm(x)
x, _ = torch.nn.utils.rnn.pad_packed_sequence(x, batch_first=True)
# OUTPUT: x = torch.Size([50, 61, 800]) --&gt; (batch_size, seq_len, n_lstm_units)
x = x.contiguous()
x = x.view(-1, x.shape[2])
# OUTPUT: x = torch.Size([3050, 800]) --&gt; (batch_size * seq_len, n_lstm_units)  
etc.
</code></pre>
<p>What i would like to do is add attention to the lstm (bilstm), so I defined a new model.</p>
<pre><code class="lang-auto">NewModel(
   THE PREVIOUS PART IS THE SAME AS THE ORIGINAL MODEL
  (bilstm): MyLSTM(
    (dropout): Dropout(p=0.3, inplace=False)
    (lstm1): LSTM(908, 600, num_layers=3, batch_first=True, dropout=0.3, bidirectional=True)
    (atten1): Attention()
    (lstm2): LSTM(1200, 600, num_layers=3, batch_first=True, dropout=0.3, bidirectional=True)
    (atten2): Attention()
  )
  THE NEXT PART IS THE SAME AS THE ORIGINAL MODEL
</code></pre>
<p>at this point my problem is to be able to report the output dimensions of the bilstm, after applying the attention, as those of the original model, so that the remaining part of the model can work.<br>
The bilstm input is formed by:</p>
<pre><code class="lang-auto"># INPUT: x = torch.Size([50, 61, 140]) and x_len = (50,)
x = nn.utils.rnn.pack_padded_sequence(x, x_len, batch_first=True)
out1, (h_n, c_n) = self.lstm1(x)
x, lengths = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
# OUTPUT: x = torch.Size([50, 61, 800]) --&gt; (batch_size, seq_len, n_lstm_units)
# OUTPUT: lenghts: x = torch.Size([61])
x, att1 = self.atten1(x, lengths)  # skip connect
# OUTPUT: x = Torch.Size(50, 800) and att1 = Torch.Size(50, 61)

out2, (h_n, c_n) = self.lstm2(out1)
y, lengths = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)
y, att2 = self.atten2(y, lengths)
# OUTPUT: y = Torch.Size(50, 800) and att2 = Torch.Size(50, 61)

z = torch.cat([x, y], dim=1)
return z  # torch.Size([64, 1600])
</code></pre>
<p>So at the same point, after the <code>forward()</code> of the bilstm of the original model and after the <code>forward()</code> of the bilstm with attention I get two different results, respectively:  <code>torch.Size([3050, 800])</code> and <code>torch.Size([64, 1600])</code>. The difference comes from the fact that in the first model it is done <code>(batch_size * seq_len, n_lstm_units) = (50 * 61, 800)</code> while in the second model the results of <code>x, att1</code> are the same dimension of <code>y, att2</code> that is (50, 800) and (50, 61), respectively.</p>
<p>Anyone have any idea what i might try to do?<br>
Is there a “standard” way to implement a self-attention module for a bilstm? I’ll put below the code I used if maybe it can be useful to better understand the problem …</p>
<pre><code class="lang-auto">class Attention(nn.Module):    

    def __init__(self, hidden_size, batch_first=False):
        super(Attention, self).__init__()

        self.hidden_size = hidden_size
        self.batch_first = batch_first

        self.att_weights = nn.Parameter(torch.Tensor(1, hidden_size), requires_grad=True)

        stdv = 1.0 / np.sqrt(self.hidden_size)

        for weight in self.att_weights:
            nn.init.uniform_(weight, -stdv, stdv)

    def forward(self, inputs, lengths):
        if self.batch_first: 
            batch_size, max_len = inputs.size()[:2]
        else:
            max_len, batch_size = inputs.size()[:2]

        # matrix mult
        # apply attention layer
        weights = torch.bmm(inputs,
                            self.att_weights  # (1, hidden_size)
                            .permute(1, 0)  # (hidden_size, 1)
                            .unsqueeze(0)  # (1, hidden_size, 1)
                            .repeat(batch_size, 1, 1)  # (batch_size, hidden_size, 1)
                            )

        attentions = torch.softmax(F.relu(weights.squeeze()), dim=-1)

        # create mask based on the sentence lengths
        mask = torch.ones(attentions.size(), requires_grad=True).cuda()
        for i, l in enumerate(lengths):  # skip the first sentence
            if l &lt; max_len:
                mask[i, l:] = 0

        # apply mask and renormalize attention scores (weights)
        masked = attentions * mask
        _sums = masked.sum(-1).unsqueeze(-1)  # sums per row

        attentions = masked.div(_sums)

        # apply attention weights
        weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))

        # get the final fixed vector representations of the sentences
        representations = weighted.sum(1).squeeze()

        return representations, attentions

class MyLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, batch_first, bidirectional, dropout):
        super(MyLSTM, self).__init__()
        self.dropout = nn.Dropout(p=dropout)
        self.lstm1 = nn.LSTM(input_size=input_size,
                             hidden_size=hidden_size,
                             num_layers=num_layers,
                             batch_first=batch_first,
                             bidirectional=bidirectional,
                             dropout=dropout)
        self.atten1 = Attention(hidden_size * 2, batch_first=batch_first)  # 2 is bidrectional
        self.lstm2 = nn.LSTM(input_size=hidden_size * 2,
                             hidden_size=hidden_size,
                             num_layers=num_layers,
                             batch_first=batch_first,
                             bidirectional=bidirectional,
                             dropout=dropout)
        self.atten2 = Attention(hidden_size * 2, batch_first=batch_first)

    def forward(self, x, x_len):
        x = nn.utils.rnn.pack_padded_sequence(x, x_len, batch_first=True)
        out1, (h_n, c_n) = self.lstm1(x)
        x, lengths = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
        x, att1 = self.atten1(x, lengths)  # skip connect

        out2, (h_n, c_n) = self.lstm2(out1)
        y, lengths = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)
        y, att2 = self.atten2(y, lengths)

        z = torch.cat([x, y], dim=1)
        return z
</code></pre>