How to perform repeat padding for variable length data?

username: hadaev8
cooked: <p>I have variable length data and want to pack it to batches with size of max sample len in batch repeating shorter samples.<br>
For example like this<br>
[[0, 1, 2, 3, 4], [0, 1, 2]] =&gt; [[0, 1, 2, 3, 4], [0, 1, 2, 0, 1]]</p>

username: ptrblck
cooked: <p>You could use some rnn util functions:</p>
<pre><code class="lang-python">x = [torch.tensor([0, 1, 2, 3, 4]), torch.tensor([0, 1, 2])]
x = torch.nn.utils.rnn.pack_sequence(x)
out = torch.nn.utils.rnn.pad_packed_sequence(x, batch_first=True)
print(out)
&gt; (tensor([[0, 1, 2, 3, 4],
        [0, 1, 2, 0, 0]]), tensor([5, 3]))
</code></pre>
<p>Where the first return value will be the padded tensor, while the second will give you the lengths before padding.</p>

username: hadaev8
cooked: <p>But this is zero padding, not repeat padding.</p>

username: ptrblck
cooked: <p>Oh sorry, I apparently missed the most important part of the question.<br>
I’m not sure, if there is a function for this, but this code snippet should work:</p>
<pre><code class="lang-python">x = [torch.tensor([0, 1, 2, 3, 4]), torch.tensor([0, 1, 2])]
max_len = max([t.size(0) for t in x])
res = [torch.cat((t, t[:max_len-t.size(0)])) for t in x]
</code></pre>

username: hadaev8
cooked: <p>With repeat padding my attention even worse since where is now non zero data in padding.<br>
                    <a href="https://i.imgur.com/gZVF0K7.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/gZVF0K7.png" width="659" height="500">
          </a>

</p>
<p>Should you advise how to perform padding in query in multi head attention?<br>
I tried to make mask, but got nan in softmax.</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit more regarding the NaN output in softmax?</p>

username: hadaev8
cooked: <p>Well, i wanted to mask attentions also by query axis.<br>
But with default attn_mask setup it case nans.<br>
Google say, its because only -infs in axis.</p>
<p>Now i edited source code of multi head attention forward like this:</p>
<pre><code>if attn_mask is not None:
    attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)
    attn_output_weights = attn_output_weights.masked_fill(attn_mask, 1e-9)
    attn_output_weights = attn_output_weights.view(bsz * num_heads, tgt_len, src_len)

attn_output_weights = softmax(
    attn_output_weights, dim=-1)
</code></pre>
<p>And made masks like this</p>
<pre><code>def get_mask_from_lengths_3d(batch_size, lengths_query, lengths_key, nheads):
    mask = torch.zeros(batch_size, lengths_key.max(),
                    lengths_query.max()).cuda()

    max_len = torch.max(lengths_key).item()
    ids = torch.arange(0, max_len, out=torch.cuda.LongTensor(max_len))
    mask[ids &gt; lengths_key.unsqueeze(1) - 1] = 1

    mask = mask.transpose(1, 2)

    max_len = torch.max(lengths_query).item()
    ids = torch.arange(0, max_len, out=torch.cuda.LongTensor(max_len))
    mask[ids &gt; lengths_query.unsqueeze(1) - 1] = 1

    return mask.unsqueeze(1).repeat(1, nheads, 1, 1).bool()


def generate_square_subsequent_mask_3d(batch_size, lengths_query, nheads):
    sz = lengths_query.max().item()
    mask = torch.triu(torch.ones(sz, sz), 1).cuda(
    ).unsqueeze(0).repeat(batch_size, 1, 1)

    ids = torch.arange(0, sz, out=torch.cuda.LongTensor(sz))
    mask[ids &gt; lengths_query.unsqueeze(1) - 1] = 1

    return mask.unsqueeze(1).repeat(1, nheads, 1, 1).bool()
</code></pre>
<p>Alignment for one layer seems to be right</p>
<p>                    <a href="https://i.imgur.com/rAKLJAa.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/rAKLJAa.png" width="428" height="378">
          </a>

</p>
<p>With mask value float(-inf) it became nan immediately.</p>

username: hadaev8
cooked: <p>How should i zero pad 2d data?</p>
<pre><code>out = {}
x = [torch.randn(10, 10), torch.randn(5, 5)]
x = torch.nn.utils.rnn.pack_sequence(x, enforce_sorted=False)
out = torch.nn.utils.rnn.pad_packed_sequence(x, batch_first=True)
</code></pre>
<blockquote>
<p>RuntimeError: The expanded size of the tensor (10) must match the existing size (5) at non-singleton dimension 1.  Target sizes: [5, 10].  Tensor sizes: [5, 5]</p>
</blockquote>

username: ptrblck
cooked: <p>In your example <code>dim1</code> should be equal, so you could pad the second tensor with <code>F.pad</code>:</p>
<pre><code class="lang-python">import torch.nn.functional as F
F.pad(torch.randn(5, 5), (2, 3, 0, 0))
</code></pre>
<p>Note that I’ve used a padding of 2 and 3 for the “left” and “right” side of <code>dim1</code>, but you could of course also only pad on one side with 5 values or chose any other valid configuration.</p>

username: hadaev8
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="75006" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Oh sorry, I apparently missed the most important part of the question.<br>
I’m not sure, if there is a function for this, but this code snippet should work:</p>
<pre><code class="lang-auto">x = [torch.tensor([0, 1, 2, 3, 4]), torch.tensor([0, 1, 2])]
max_len = max([t.size(0) for t in x])
res = [torch.cat((t, t[:max_len-t.size(0)])) for t in x]
</code></pre>
</blockquote>
</aside>
<p>I finally tried your snippet, but it do not work if one sample x2 longer then another</p>