Merge two variable length sequence together?

username: Yufeng_Ma
cooked: <p>Hi, all</p>
<p>How can I merge two variable sequences together? Like the example below, with word and image token sequence (batch_first=False) and their length</p>
<p>w_input = Variable( torch.randn( 20, 4, 50 ) )<br>
w_len = Variable( torch.LongTensor( [ 20, 15, 10, 5 ] ) )<br>
i_input = Variable( torch.randn( 3, 4, 50 ) )<br>
i_len = Variable( torch.LongTensor( [ 3, 2, 1, 1 ] ) )</p>
<p>What I want is a new variable:<br>
token_input = Variable( torch.FloatTensor( 23, 4, 50 ).fill_(0) )<br>
whose data come from the above two sequences, with their lengths<br>
token_len = w_len + i_len</p>
<p>Any hints here?</p>
<p>Thanks so much for your help.</p>

username: apaszke
cooked: <p>You’re probably looking for <code>torch.cat</code></p>

username: Yufeng_Ma
cooked: <p>Thanks for the note.</p>
<p>Yeah. torch.cat can help. But what I further need to do is to copy the last few tokens’ embedding to previous slots if w_len is less than the max_len (20 here)  in first token sequence. That may needs a for loop. So do you know any other efficient ways of dong this. Thanks for your precious time.</p>

username: apaszke
cooked: <p>um no, there’s no automatic way of padding the tensors. I’d just fill them with 0s and copy only the relevant parts in a for loop <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

username: Yufeng_Ma
cooked: <p><a class="mention" href="/u/apaszke">@apaszke</a> Thanks.</p>
<p>BTW, do we need to fill them with 0’s after ‘torch.cat’ if we’ve provided the length for each sequence? I don’t think RNN will compute sequence tokens after the timestep of sequence length , right?</p>

username: apaszke
cooked: <p>Oh yeah actually if you use <code>torch.nn.utils.rnn.pack_padded_sequence</code> then you can just put any random values in the padding. They will be ignored anyway</p>

username: leoribeiro
cooked: <p>Hi <a class="mention" href="/u/yufeng_ma">@Yufeng_Ma</a>,</p>
<p>did you solve your problem? I have the same problem. I have two tensors with variable sequences and I want to concat them, but I want to put the zeros in the end.<br>
For example, I have a sequence 5 6 7 8 0 0 and another sequence 6 9 1 2 3 4 0 0. I need a merged sequence as follows: 5 6 7 8 6 9 1 2 3 4 0 0 0 0. I don’t want just to concat, because if I concat I would have 5 6 7 8 0 0 6 9 1 2 3 4 0 0.</p>
<p>Does your problem is similar to mine?</p>

username: MariosOreo
cooked: <p>Hi,</p>
<p>Does this fit your requirement? This may be not the efficient way, but it works.</p>
<pre><code class="lang-auto">seq_a = torch.tensor([5,6,7,8,0,0])
seq_b = torch.tensor([6,9,1,2,3,4,0,0])
result = torch.cat([seq_a[seq_a.nonzero()], seq_b[seq_b.nonzero()]])
zeros = len(seq_a) - seq_a.nonzero().numel() + len(seq_b) - seq_b.nonzero().numel()
result = torch.cat([result, torch.zeros(zeros).unsqueeze(-1).long()]).squeeze(-1)
</code></pre>

username: Allan_Jie
cooked: <p>I think this works, but it seems only works for one-dimension tensor</p>

username: Allan_Jie
cooked: <p>Hi <a class="mention" href="/u/leoribeiro">@leoribeiro</a> I think this general solution helps your case.</p>
<pre><code class="lang-python">x = torch.tensor(
       [[  -2.0813,   -1.3431, 0],
        [   1.7704, 0, 0],
        [   0.1229,    1.2149,    0.5455]])
y= torch.tensor(
        [[   1.6999,   -0.7491,   -1.9227, 0],
        [   0.4784,   -0.2545, 0, 0],
        [  -1.5127,   -0.1439,   -0.1829,   -0.3703]])

## first we concat the tensor
concated_tensor = torch.cat([x,y], dim=-1)
## The key is to create this index:
index = torch.tensor([[2,3,4,5], [1,2,3,4], [3,4,5,6]])
## Finally we call
concated_tensor.scatter_(1, index, y)
</code></pre>
<p>We finally obtain something like this:</p>
<pre><code class="lang-python">tensor([[  -2.0813,   -1.3431,    1.6999,   -0.7491,   -1.9227, 0, 0],
        [   1.7704,    0.4784,   -0.2545, 0,0,0,0],
        [   0.1229,    1.2149,    0.5455,   -1.5127,   -0.1439,   -0.1829, -0.3703]])
</code></pre>
<p><strong>How to create that index tensor</strong>?</p>
<pre><code class="lang-python">tmp_idx = torch.bincount((x!=0).nonzero()[:,0], minlength=3) ##3 is batch size
## tmp_idx = tensor([2, 1, 3])
## Create a ones tensor
pad_ones = torch.ones(y.size(0), y.size(1)-1)
##&gt;&gt;&gt; pad_ones
##tensor([[1., 1., 1.],
  ##      [1., 1., 1.],
  ##      [1., 1., 1.]])
update_index = torch.cat([num.unsqueeze(-1), pad_ones.long()], dim=-1)
index = torch.cumsum(update_index,dim=1)
</code></pre>
<p>You will end up with this index:</p>
<pre><code class="lang-python">tensor([[2, 3, 4, 5],
        [1, 2, 3, 4],
        [3, 4, 5, 6]])
</code></pre>

username: Felix_Kreuk
cooked: <p>Hi <a class="mention" href="/u/allan_jie">@Allan_Jie</a> thanks for this solution, it’s very nice!<br>
However, there is a small bug, you can try it for the input:</p>
<pre><code class="lang-auto">x = torch.tensor(
       [[  -2.0813,   -1.3431, 0],
        [   1.7704, 0, 0],
        [   0.1229,    1.2149,   **0**]])
y= torch.tensor(
        [[   1.6999,   -0.7491,   -1.9227, 0],
        [   0.4784,   -0.2545, 0, 0],
        [  -1.5127,   -0.1439,   -0.1829,   -0.3703]])
</code></pre>
<p>this output will be:</p>
<pre><code class="lang-auto">tensor([[-2.0813, -1.3431,  1.6999, -0.7491, -1.9227,  0.0000,  0.0000],
        [ 1.7704,  0.4784, -0.2545,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.1229,  1.2149, -1.5127, -0.1439, -0.1829, -0.3703, -0.3703]])
</code></pre>
<p>while the correct output should be:</p>
<pre><code class="lang-auto">tensor([[-2.0813, -1.3431,  1.6999, -0.7491, -1.9227,  0.0000,  0.0000],
        [ 1.7704,  0.4784, -0.2545,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.1229,  1.2149, -1.5127, -0.1439, -0.1829, -0.3703, 0]])
</code></pre>
<p>(see bottom right).</p>
<p>The solution is to zero-out the remainder of the sequence after the scatter operation:</p>
<pre><code class="lang-auto">def length_to_mask(length, max_len=None):
        assert len(length.shape) == 1, 'Length shape should be 1 dimensional.'
        max_len = length.max() if not max_len else max_len
        return torch.arange(max_len)[None, :].to(length.device) &lt; length[:, None]

...
merged_len = length_to_mask(len_x + lens_y, max_len=concated_tensor.shape[1])
concated_tensor.scatter_(1, index, y)
concated_tensor[~merged_len] = 0
</code></pre>
<p>(assuming you know <code>lens_x</code> and <code>lens_y</code>)</p>
<p>Hope that helps!</p>

username: Allan_Jie
cooked: <p>Yes, you are right. Didn’t consider that edge case <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>