Merge two variable length sequence together?

Yufeng_Ma: Hi, all How can I merge two variable sequences together? Like the example below, with word and image token sequence (batch_first=False) and their length Any hints here? Thanks so much for your help.

apaszke: 

Yufeng_Ma: Thanks for the note. Yeah. torch.cat can help. But what I further need to do is to copy the last few tokens’ embedding to previous slots if w_len is less than the max_len (20 here)  in first token sequence. That may needs a for loop. So do you know any other efficient ways of dong this. Thanks for your precious time.

apaszke: 

Yufeng_Ma: BTW, do we need to fill them with 0’s after ‘torch.cat’ if we’ve provided the length for each sequence? I don’t think RNN will compute sequence tokens after the timestep of sequence length , right?

apaszke: 

leoribeiro: Does your problem is similar to mine?

MariosOreo: Hi, Does this fit your requirement? This may be not the efficient way, but it works. <SCODE>seq_a = torch.tensor([5,6,7,8,0,0])
seq_b = torch.tensor([6,9,1,2,3,4,0,0])
result = torch.cat([seq_a[seq_a.nonzero()], seq_b[seq_b.nonzero()]])
zeros = len(seq_a) - seq_a.nonzero().numel() + len(seq_b) - seq_b.nonzero().numel()
result = torch.cat([result, torch.zeros(zeros).unsqueeze(-1).long()]).squeeze(-1)
<ECODE>

Allan_Jie: I think this works, but it seems only works for one-dimension tensor

Allan_Jie: <SCODE>x = torch.tensor(
       [[  -2.0813,   -1.3431, 0],
        [   1.7704, 0, 0],
        [   0.1229,    1.2149,    0.5455]])
y= torch.tensor(
        [[   1.6999,   -0.7491,   -1.9227, 0],
        [   0.4784,   -0.2545, 0, 0],
        [  -1.5127,   -0.1439,   -0.1829,   -0.3703]])

## first we concat the tensor
concated_tensor = torch.cat([x,y], dim=-1)
## The key is to create this index:
index = torch.tensor([[2,3,4,5], [1,2,3,4], [3,4,5,6]])
## Finally we call
concated_tensor.scatter_(1, index, y)
<ECODE> We finally obtain something like this: <SCODE>tensor([[  -2.0813,   -1.3431,    1.6999,   -0.7491,   -1.9227, 0, 0],
        [   1.7704,    0.4784,   -0.2545, 0,0,0,0],
        [   0.1229,    1.2149,    0.5455,   -1.5127,   -0.1439,   -0.1829, -0.3703]])
<ECODE> <SCODE>tmp_idx = torch.bincount((x!=0).nonzero()[:,0], minlength=3) ##3 is batch size
## tmp_idx = tensor([2, 1, 3])
## Create a ones tensor
pad_ones = torch.ones(y.size(0), y.size(1)-1)
##>>> pad_ones
##tensor([[1., 1., 1.],
  ##      [1., 1., 1.],
  ##      [1., 1., 1.]])
update_index = torch.cat([num.unsqueeze(-1), pad_ones.long()], dim=-1)
index = torch.cumsum(update_index,dim=1)
<ECODE> You will end up with this index: <SCODE>tensor([[2, 3, 4, 5],
        [1, 2, 3, 4],
        [3, 4, 5, 6]])
<ECODE>

Felix_Kreuk: <SCODE>x = torch.tensor(
       [[  -2.0813,   -1.3431, 0],
        [   1.7704, 0, 0],
        [   0.1229,    1.2149,   **0**]])
y= torch.tensor(
        [[   1.6999,   -0.7491,   -1.9227, 0],
        [   0.4784,   -0.2545, 0, 0],
        [  -1.5127,   -0.1439,   -0.1829,   -0.3703]])
<ECODE> this output will be: <SCODE>tensor([[-2.0813, -1.3431,  1.6999, -0.7491, -1.9227,  0.0000,  0.0000],
        [ 1.7704,  0.4784, -0.2545,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.1229,  1.2149, -1.5127, -0.1439, -0.1829, -0.3703, -0.3703]])
<ECODE> while the correct output should be: <SCODE>tensor([[-2.0813, -1.3431,  1.6999, -0.7491, -1.9227,  0.0000,  0.0000],
        [ 1.7704,  0.4784, -0.2545,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.1229,  1.2149, -1.5127, -0.1439, -0.1829, -0.3703, 0]])
<ECODE> (see bottom right). The solution is to zero-out the remainder of the sequence after the scatter operation: <SCODE>def length_to_mask(length, max_len=None):
        assert len(length.shape) == 1, 'Length shape should be 1 dimensional.'
        max_len = length.max() if not max_len else max_len
        return torch.arange(max_len)[None, :].to(length.device) < length[:, None]

...
merged_len = length_to_mask(len_x + lens_y, max_len=concated_tensor.shape[1])
concated_tensor.scatter_(1, index, y)
concated_tensor[~merged_len] = 0
<ECODE> Hope that helps!

Allan_Jie: