[Solved] Multiple PackedSequence input ordering

aron-bordin: <p>Hi !</p>
<p>I’m new on pytorch (moving from torch), and I’m having some problems to implement a model …</p>
<p>I’ve two variable length time-serie sequences that will be forwarded in the same network, and its output will be compared using the cosine distance.</p>
<p>The problem here is that PackedSequence expects the sequences to be ordered by seq length. However, my model expects two sequences as input, so if I order them, then first seq of the first input may not be related with the first seq of the second input.</p>
<p>What can I use to be able to compare the correct pair of sequence ?</p>

jekbradbury: <p>This comes up fairly frequently (e.g. machine translation) and the current solution is to use <code>torch.sort</code> to order examples by length on each side and store the indices in order to reverse that order later. But it’s easy to get this wrong, and it should probably be added as a convenience function to the rnn utils. I may do that if I have time, or if anyone else wants to make a PR they should go ahead.</p>

aron-bordin: <p>My forward function looks like:</p>
<pre><code>   def forward(self, dict_index, features, prev_hidden, seq_sizes, original_index):
        i2e = self.embedding(dict_index)
        data = torch.cat((i2e, features), 2)

        packed = pack_padded_sequence(data, list(seq_sizes.data), batch_first=True)
        output, _ = self.rnn(packed, prev_hidden)

        output, _ = pad_packed_sequence(output, batch_first=True)
        # get the last time step for each sequence
        idx = (seq_sizes - 1).view(-1, 1).expand(output.size(0), output.size(2)).unsqueeze(1)
        decoded = output.gather(1, idx).squeeze()

        # restore the sorting
        decoded[original_index] = decoded

        return decoded
</code></pre>
<p>and then I compare the decoding using the cosine embedding. Will pytorch use the correct gradients on backward ? Do I need to modify something ? Because I’m changing the order of the data in the last step of the forward, so do I need to “reorder” the loss values ?</p>

AjayTalati: <p>Hi,</p>
<p>I did this a few days ago,</p>
<pre><code class="lang-auto"># Helper functions

def argsort(seq):
    # http://stackoverflow.com/questions/3071415/efficient-method-to-calculate-the-rank-vector-of-a-list-in-python
    return sorted(range(len(seq)), key=seq.__getitem__)

# apply padding
def pad(tensor, length):
    return torch.cat([tensor, tensor.new(length - tensor.size(0), *tensor.size()[1:]).zero_()])

lengths=[]
for i in range(batch_size):
    length_i = input_variables[i].size()[0]
    lengths.append( length_i )

# sort in numpy - TODO do this in pure python or PyTorch
#lengths_np = np.array((lengths))
#indx = np.argsort(lengths_np, axis=0)[::-1]
#inverse_indx = np.argsort(indx, axis=0)

#sort - #see http://stackoverflow.com/questions/2483696/undo-or-reverse-argsort-python
indx = argsort( lengths ) # ascending
indx = indx[::-1]         # descending
inverse_indx = argsort(indx)

# check
#lengths_dec = [lengths[i] for i in indx] # sorted lengths
#remap_lengths = [lengths_dec[i] for i in inverse_indx] # map 
#lengths == remap_lengths

padded_vec_lst = []
for i in range(batch_size):
    padded_vec = pad(vec_lst[i], max_length)
    padded_vec_lst.append( padded_vec )

</code></pre>
<p>It works <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>HTHs</p>

aron-bordin: <p>Hi ! Thx for your code <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"><br>
Currently I’m ordering it using pytorch’s torch.sort function.</p>
<p>But I’m not sure if the gradients are correct, because my grad_output contains the original data ordering (I need to maintain the original order to compare the distance of two sequences), but each model receives sequences in a different ordering.</p>
<p>So I assume that I’ll need to reorder the grad_output (loss derivatives) to match the forward order.</p>
<p>But I’m not sure if I really need to do it, or if pytorch does it internally. And if necessary, I don’t know how to reorder the grad_output <img src="//discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"></p>

aron-bordin: <p>To update, I solved this problem.<br>
I checked and yes, pytorch backward does the gradient reordering. I just updated the</p>
<pre><code>decoded[original_index] = decoded
</code></pre>
<p>with gather.</p>
<p>Working now, thx everyone <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

nitish: <p><a class="mention" href="/u/aron-bordin">@aron-bordin</a> could you put a snippet of the final version? I am a bit confused.</p>

aron-bordin: <p><a class="mention" href="/u/nitish">@nitish</a> I just updated my forward funcion above. The final forward is:</p>
<pre><code>def forward(self, dict_index, features, prev_hidden, seq_sizes, original_index):
    i2e = self.embedding(dict_index)
    data = torch.cat((i2e, features), 2)

    packed = pack_padded_sequence(data, list(seq_sizes.data), batch_first=True)
    output, _ = self.rnn(packed, prev_hidden)

    output, _ = pad_packed_sequence(output, batch_first=True)
    # get the last time step for each sequence
    idx = (seq_sizes - 1).view(-1, 1).expand(output.size(0), output.size(2)).unsqueeze(1)
    decoded = output.gather(1, idx).squeeze()


    # restore the sorting
    odx = original_index.view(-1, 1).expand(args.batch_size, output.size(-1))
    decoded = decoded.gather(0, Variable(odx))
    return decoded

    return decoded</code></pre>

nitish: <p><a class="mention" href="/u/aron-bordin">@aron-bordin</a> So using this, if you have multiple mini-batches (with different sorting permutations) going in different RNNs,  this forward pass returns them in the original order and then they are directly comparable. Pytorch takes care of this permutation while backpropagating. I hope I understand this correctly.<br>
One question, how did you check that the grads are permuted correctly?</p>
<p>Thanks!</p>

aron-bordin: <p>Yes, its correct.</p>
<p>I don’t have any sample here, but I tested implementing a simple equation, then I passed multiple values to this equation, change the order with gather, applied another function and the backpropagated. Then I calculated the derivatives manually and compared with the pytorch output, so I could verify pytorch was considering the ordering while backpropagating.</p>

chihyaoma: <p><a class="mention" href="/u/aron-bordin">@aron-bordin</a> Hey, wouldn’t it be easier if you change this from:</p>
<aside class="quote no-group quote-modified" data-username="aron-bordin" data-post="8" data-topic="2106">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aron-bordin/48/505_2.png" class="avatar"> aron-bordin:</div>
<blockquote>
<p>output, _ = self.rnn(packed, prev_hidden)</p>
<p>output, _ = pad_packed_sequence(output, batch_first=True)</p>
<h1>get the last time step for each sequence</h1>
<p>idx = (seq_sizes - 1).view(-1, 1).expand(output.size(0), output.size(2)).unsqueeze(1)<br>
decoded = output.gather(1, idx).squeeze()</p>
</blockquote>
</aside>
<p>to</p>
<pre data-code-wrap="python"><code class="lang-python">output, (ht, ct) = self.rnn(packed, prev_hidden)
decoded = ht[-1]
</code></pre>
<p>Both the <em>decoded</em> above should be the same.</p>
<p>Isn’t the output of <em>self.rnn</em> already gave you the output for t=seq_len, even though we have variable-length input?</p>
<p>Also, may I ask what’s your <em>prev_hidden</em>? I thought it should be the initial state for hidden and cell.</p>

tushar: <blockquote>
<p>Both the decoded above should be the same.</p>
</blockquote>
<p>It does look like the two are equivalent. Could anyone please confirm if using <code>ht[-1]</code> is appropriate?</p>

aron-bordin: <p>No, they are not the same.<br>
The problem here is that I’m passing a batch of variable sequence lengths. So. some of the sequences will be padded with 0’s vectors, leading to an unwanted output in the rnn.<br>
So, the rnn will produce one output per timestep, and as long as some of the timesteps are zero (due to the padding), it’s necessary to get the proper output.</p>

chihyaoma: <p>My input is also a batch of variable-length input. I also padded them and pack them using <code>pack_padded_sequence</code>.</p>
<p>using <code>ht[-1]</code> return the same result as your solution.</p>
<p>I think you are expecting <code>ht[-1]</code> will return zeros for short inputs that were padded, right? But, that’s not the case when I tested it. Would you mind to double-check that? Or am I missing something here?</p>

aron-bordin: <p>I’ll check this on Monday and I post here. But some points to consider is<br>
that I’m using both LSTM and GRU rnn, and with the bidirectional<br>
architecture. And I remember that I first tested with -1, but this was not<br>
working in my case, and the I used the solution above. I’ll confirm it in<br>
the beginning of the week <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>
<p>My prev_hidden is usually zero while training, I just have the parameter to<br>
evaluate the model in some specific starting points.</p>

danish: <p>Does anyone have sample code for batched multiple packed sequences with Attention?</p>
<p>Here’s where I am confused:</p>
<p><strong>Background</strong> I have a data set with a series of sentence tuples (s1, s2). I have a Bidi lstm for each of them and I would like to attend on the two sequences s1, and s2 before I send them off to a linear layer  – I would also like to do this in a batch setting though the pseudo code below is written in a per instance (i.e. per s1,s2 tuple) forward pass.</p>
<p>Something like this (not working code – pseudo code)</p>
<pre><code>    combined_input_2_attention=torch.cat((s1_lstm_out, s2_lstm_out), 0)
	attention_alphas=self.softmax(self.attention_layer(combined_input_2_attention))
	attn_applied = torch.bmm(attention_alphas.unsqueeze(0),
                             combined_input_2_attention)
	
	output_embedding=self.hidden_layer(attn_applied)
	output=self.softmax(output_embedding)
</code></pre>
<p>where s1_lstm_out and s2_lstm_out are the outputs of sending one s1, s2 tuple in the forward pass.</p>
<p><strong>Q1</strong>.  If this was batched – Do the attention weights (alphas) need to be in dimensions of length of the max sequence length (s1+s2 since I am concatenating) per batch(?) or globally?.  I doubt this is something can be per batch – because how would i initialize the dimensions of the linear layer that does the attention?</p>
<p><strong>Q2</strong> Either way I need to have a packed sequence for the two LSTMs before I attend over them or concatenate them – but the problem is padding requires them to be sorted individually.<br>
I read the answer of using sort but I could not follow it completely.</p>
<p>Here’s what I understood from the post above:</p>
<ol>
<li>I take a set of tuples (s1, s2). Sort them individually using tensor.sort and keep track of their original indices.</li>
<li>Individually pack s1 and s2 into padded sequences based on their individual max lengths per batch and send it to my forward pass</li>
<li>At the end of it – I return my outputs by reverse mapping them based on the sorted order generated in (1)? If yes, wouldn’t the network have seen an instance of s1 and s2 in different orders from my original – why is this correct/why does this work? What am I missing?</li>
</ol>
<p>Any leads /clarification would be extremely helpful! Thanks!</p>

danish: <p>Figured this out –</p>
<p>sequence 1 - sort -&gt; pad and pack -&gt;process using RNN -&gt; unpack -&gt;unsort<br>
sequence 2  -  sort -&gt; pad and pack -&gt;process using RNN -&gt; unpack -&gt;unsort</p>
<p>Do whatever you wanted to do with the unsorted outputs (combine, attend-- whatever)</p>

miguelvr: <p>I think you are better off just by keeping it sorted throughout the training, as sorting operations are usually faster on the CPU (therefore it is better to do the sorting at the data loading stage).</p>

danish: <p><a class="mention" href="/u/miguelvr">@miguelvr</a> Thanks - the reason I am unsorting is that both my sequences are “paired” but pass through different RNNs. If I dont unsort, I lose the pairing (because packing was done independently).</p>

nitish: <p><a class="mention" href="/u/aron-bordin">@aron-bordin</a> : I have a couple of questions. To confirm, <code>dict_index</code> is sorted and <code>original_index</code> is the inverse of the sorting permutation? Is that correct and how do you get the inverse of the sorting permutation from <code>torch.sort</code>?</p>