PackedSequence with DataLoader

username: miguelvr
cooked: <p>Hey,</p>
<p>I’m trying to reproduce some previous work I did with Theano in PyTorch, with RNNs.</p>
<p>I want to be able to mask the sequences I pass as input to an RNN based model. This should be easy enough…</p>
<p>However, there’s a couple of annoying issues that are bugging me:</p>
<ol>
<li>
<p>PackedSequences inputs are only supported by RNNs, which means that I have to constantly use pack_padded_sequences and pad_packed_sequences constantly, back and forth, in order to have a model with RNN layers that interact with other types of layers.</p>
</li>
<li>
<p>The requirement of pack_padded_sequences of having a sorted sequence length list. This is pretty much the same as bucketing and it is conditioning training by not allowing random sapling. By having this requirement, how can I combine a DataLoader with PackedSequence without having to fully sort my dataset by length?</p>
</li>
</ol>
<p>And finally, one last question:<br>
Is it possible to mask the loss function natively in PyTorch?</p>
<p>Cheers!</p>

username: jekbradbury
cooked: <ol>
<li>Yes, for now you have to constantly pack and unpack if you’re mixing RNNs and conv layers. If you’re mixing RNNs and fully connected layers, you don’t have to unpack at all – you can actually call the linear layer directly on the packed sequence.</li>
<li>The list just has to be sorted within an individual batch, so you can still shuffle your dataset and randomly sample a batch at a time, then sort each batch to send it to pack_padded_sequence.</li>
</ol>

username: miguelvr
cooked: <p>What’s the best way to sort / rearrange tensors in PyTorch?</p>

username: ShigekiKarita
cooked: <aside class="quote no-group" data-username="jekbradbury" data-post="2" data-topic="1495">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jekbradbury/48/14_2.png" class="avatar"> jekbradbury:</div>
<blockquote>
<p>you can actually call the linear layer directly on the packed sequence.</p>
</blockquote>
</aside>
<p>is this correct usage?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
x = torch.autograd.Variable(torch.randn(5, 3, 2))
l = torch.nn.Linear(2, 2)
r = torch.nn.RNN(2, 2)

px = torch.nn.utils.rnn.pack_padded_sequence(x, [5, 3, 1])
ph = torch.nn.utils.rnn.PackedSequence(l(px.data), px.batch_sizes)
py, h = r(ph)
</code></pre>

username: jekbradbury
cooked: <p>Yes, that looks right to me.</p>

username: stefanonardo
cooked: <blockquote>
<p>If you’re mixing RNNs and fully connected layers, you don’t have to unpack at all – you can actually call the linear layer directly on the packed sequence.</p>
</blockquote>
<p>Is this true? It is not working for me since nn.Linear requires attribute ‘dim’.</p>

username: eplu
cooked: <aside class="quote no-group quote-modified" data-username="stefanonardo" data-post="7" data-topic="1495">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stefanonardo/48/2322_2.png" class="avatar"> stefanonardo:</div>
<blockquote>
<blockquote>
<p>If you’re mixing RNNs and fully connected layers, you don’t have to unpack at all – you can actually call the linear layer directly on the packed sequence.</p>
</blockquote>
</blockquote>
</aside>
<p>I’m observing the same behavior. I don’t think linear layers can take packed sequences as inputs.</p>

username: nlgranger
cooked: <p>I suppose you can do it by passing a custom <code>collate_fn</code> to the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">DataLoader</a>.</p>

username: Moore
cooked: <p>It seems that the trick is to use <code>.data</code> when invoke the <code>nn.linear()</code> as shown in <a class="mention" href="/u/shigekikarita">@ShigekiKarita</a>’s code.</p>

username: Moore
cooked: <p>From the documentation of <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.PackedSequence.html" rel="noopener nofollow ugc"><code>torch.nn.utils.rnn.PackedSequence</code></a>:</p>
<blockquote>
<p>Instances of this class should never be created manually. They are meant to be instantiated by functions like <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_padded_sequence.html#torch.nn.utils.rnn.pack_padded_sequence" rel="noopener nofollow ugc"><code>pack_padded_sequence()</code></a>.</p>
</blockquote>
<p>Is there possible any better way now to apply the packed sequence with the linear layers?</p>