How to use pack_sequence if we are going to use word embedding and BiLSTM

coyote: <p>Hi everyone,</p>
<p>I see that there is a <code>pack_sequence</code> utility function used with Recurrent neural nets. There is a simple <a href="https://discuss.pytorch.org/t/simple-working-example-how-to-use-packing-for-variable-length-sequence-inputs-for-rnn/2120/33">example</a> to demonstrate usage of it. However, it does not include the word embedding usage. When I try to do the following I got an error:</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.rnn as rnn_utils
import torch.nn as nn


embeddings = nn.Embedding(6,10)
lstm = nn.LSTM(10, 200,num_layers=1, bidirectional=True)


a = torch.Tensor([[1], [2], [3]])
b = torch.Tensor([[4], [5]])
c = torch.Tensor([[6]])
packed = rnn_utils.pack_sequence([a, b, c])
emb  = embeddings(packed) # This line raises an error
</code></pre>
<blockquote>
<p>Error message: TypeError: embedding(): argument ‘indices’ (position 2) must be Tensor, not PackedSequence</p>
</blockquote>
<blockquote>
<p>If I was succesful on the above line, I was going to give it as an input to lstm as below:</p>
</blockquote>
<p><code>packed_output, (h,c) = lstm(emb)</code></p>

tom: <p>A couple of months ago,  following (not the cleanest, but even works for multiple args) worked:</p>
<pre><code class="lang-python">def elementwise_apply(fn, *args):
    return torch.nn.utils.rnn.PackedSequence(fn(*[(arg.data if type(arg)==torch.nn.utils.rnn.PackedSequence else arg) for arg in args]), args[0].batch_sizes)
</code></pre>
<p>and then<br>
<code>emb = elementwise_apply(embeds, packed)</code></p>
<p>Best regards</p>
<p>Thomas</p>

coyote: <p>Thank you so much for your response Thomas. If you don’t mind could you explain your function a little bit ? For example, could you tell me what is fn, what should be passed into to args etc.  if they are well-known things in pytorch I am sorry for my request but  I am also pretty new in pytorch. If possible a simple example demonstrating usage of this function would be awesome.</p>
<p>I tried following but I got an error:</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.rnn as rnn_utils
import torch.nn as nn
embeddings = nn.Embedding(6,10)
lstm = nn.LSTM(10, 200,num_layers=1, bidirectional=True)

a = torch.Tensor([[1], [2], [3]])
b = torch.Tensor([[4], [5]])
c = torch.Tensor([[6]])
packed = rnn_utils.pack_sequence([a, b, c])
emb = elementwise_apply(embeddings, packed)
</code></pre>
<blockquote>
<p>return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘indices’ to have scalar type Long; but got CPUFloatTensor instead (while checking arguments for embedding)</p>
</blockquote>

tom: <p>Fair enough!</p>
<p>Let’s make a simpler version:</p>
<pre><code class="lang-auto">def simple_elementwise_apply(fn, packed_sequence):
    """applies a pointwise function fn to each element in packed_sequence"""
    return torch.nn.utils.rnn.PackedSequence(fn(packed_sequence.data), packed_sequence.batch_sizes)
</code></pre>
<p>What this does is a) apply <code>fn</code> to the <code>.data</code> (which is where the flattened sequence elements live) and b) return a packed sequence with the result and the “bookkeeping” of <code>.batch_sizes</code>.<br>
The more elaborate version above does the same, but a) takes multiple arguments b) when the arguments are packed sequences it passes the <code>.data</code> to fn and otherwise the full argument.</p>
<p>Best regards</p>
<p>Thomas</p>

RylanSchaeffer: <p>Is this still the standard, 3 years later? Why is working with sequences so hard?</p>

tom: <p>Hi Rylan!</p>
<p>I am afraid it is.<br>
As much as I sympathize with a utility function “apply” would not hurt:</p>
<ul>
<li>It seems that the use of packed sequences is relatively narrow and in particular it is probably not terribly common to need to intermix the operations on packed sequences a lot with other things. For example, I don’t think HuggingFace’s transformers library - quite popular for processing sequences - is using PackedSequence at all.</li>
<li>Quite likely, the “figuring out what is the right thing to do” is still much more difficult than “implementing it”.</li>
<li>If you want, I can offer two issues with discussion, some as recent as 2020: <a href="https://github.com/pytorch/pytorch/issues/1927" class="inline-onebox" rel="noopener nofollow ugc">[feature request] time-distributed layers for application of normal layers to sequence data · Issue #1927 · pytorch/pytorch · GitHub</a> <a href="https://github.com/pytorch/pytorch/issues/8921" class="inline-onebox" rel="noopener nofollow ugc">[feature request] More methods for PackedSequence · Issue #8921 · pytorch/pytorch · GitHub</a> on the subject, some with more recent posts than on the forums.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

RylanSchaeffer: <p>Is there a clear tutorial about how to properly call the sequence of <code>torch.nn.utils.rnn.pad_sequence</code>, <code>pad_packed_sequence</code> and <code>pack_padded_sequence</code>? I can’t find one in the documentation.</p>

tom: <p>There might not be…<br>
So there are three equivalent representations here with different types:</p>
<ol>
<li>
<code>List[Tensor]</code> a list of sequences.</li>
<li>
<code>Tensor</code> a single tensor of sequences with padding. Crucial extra information: pad value, whether it is seq first or batch first.</li>
<li>
<code>PackedSequence</code> an object containing packed sequences. Contains the extra informaiton: batch sizes, indices from reordering.</li>
</ol>
<p>So then the conversion functions all go between them, and you can just go by the type signatures to see which is appropriate: <code>pad_sequence</code>: 1 → 2  <code>pad_packed_sequence</code> 3 → 2 , <code>pack_padded_sequence</code> 2 → 3, <code>pack_sequence</code> 1 → 3.</p>

sanket_gandhi: <p>can we go from 3-&gt;1?</p>