Question about: Packed RNN with DataParallel

username: stephenrawls
cooked: <p>Hi,</p>
<p>I am building an OCR model in PyTorch and running into an issue with DataParallel and Packed RNN sequences.</p>
<p>(1) My code seems pretty straightforward:</p>
<pre><code class="lang-auto">def forward(self, input, input_lengths):
    packed_input = nn.utils.rnn.pack_padded_sequence(input, input_lengths)
    packed_output = self.lstm(packed_input)
    output, _ = nn.utils.rnn.pad_packed_sequence(packed_output)
    return output
</code></pre>
<p>It works fine on a single GPU, but when I try to use <code>DataParallel</code> to run on multiple GPUs I run into a problem because the unpacked tensors on each GPU are of differing number of time steps, depending on the maximum value of the chunk of the <code>input_lengths</code> array that got passed to each GPU.</p>
<p>I am resolving the problem by doing this:</p>
<pre><code class="lang-auto">padded_output = Variable(torch.zeros(max_T, output.size()[1], output.size()[2]))
padded_output[:input_lengths.max(), :, :] = output
return padded_output
</code></pre>
<p>I just want to check with folks who are more familiar with PyTorch to see if this is the appropriate way to resolve this problem, or if there is a better way?</p>
<p>(2) Another issue, that is easily corrected by a small change to <code>DataParallel</code>, is that my model takes in an image tensor of dimension <code>BCHW</code> and outputs a set of LSTM outputs of dimension <code>TBF</code> – i.e. when I scatter inputs to the various GPUs I want to split on the batch dimension of my input vector, 0, and when I gather outputs from the various GPUs I want to stack them on the batch dimension of my output vector, 1.</p>
<p>I have modified <code>DataParallel</code> locally so that I can pass in different <code>in_dim</code> and <code>out_dim</code> parameters as opposed to a single <code>dim</code> parameter. Is there any interest in making such a change in the official repository?</p>
<p>Regards,<br>
Stephen</p>

username: apaszke
cooked: <p>I’m afraid that packed sequences aren’t compatible with DataParallel! It should raise an error, because I think that right now it computes sth invalid. A fix would be to do <code>pack_padded_sequence</code> inside the DataParallel part (so instead of wrapping <code>self.lstm</code> you’d wrap a module that does <code>pack</code> + <code>self.lstm</code> + <code>pad</code>).</p>

username: stephenrawls
cooked: <p>Hi <a class="mention" href="/u/apaszke">@apaszke</a>, Thanks for the quick reply.</p>
<p>Sorry for not being clearer, I think I am already doing what you are describing–I have a custom class that inherits from nn.Module, and it is this custom class that I wrap with <code>DataParallel</code>. Then inside the <code>forward()</code> method of my custom class, I do the pack + lstm + unpack steps.</p>
<p>But the problem is that if I pass in a <code>100x2xF</code> array with <code>input_lengths = [100, 50]</code>, what happens is that GPU 1 unpacks the LSTM output into a <code>100x1xF</code> tensor, and GPU 2 unpacks the LSTM output into a <code>50x1xF</code> tensor, so that when <code>DataParallel</code> tries to gather them together along the batch dimension, it complains about the sizes not matching up.  (Thus the fix in my code to add padding to the output tensors on all GPUs, e.g. in this example I would pad GPU 2’s output to be <code>100x1xF</code>).</p>

username: apaszke
cooked: <p>I see. Yeah this solution should work correctly.</p>

username: ecolss
cooked: <p><a class="mention" href="/u/apaszke">@apaszke</a> <a class="mention" href="/u/stephenrawls">@stephenrawls</a></p>
<p>I have a naive question here, in order to make the above <code>forward</code> function work on GPU,   the whole model and input have to be moved to GPU by calling <code>.cuda()</code>, right? But since <code>pack_padded_sequence</code> needs a seq lengths param (which is typically a <code>list</code> object if on CPU), so do I have to convert this list param into a cuda array?</p>
<p>I’m having issues when doing this conversion, like <code>LongTensor(input_lengths).cuda()</code> and pass it to <code>pack_padded_sequence</code>.</p>

username: stephenrawls
cooked: <p>Hi <a class="mention" href="/u/ecolss">@ecolss</a>. Yes to your first question, to move to GPU you need to call <code>model = model.cuda()</code>  and <code>input = input.cuda()</code>.</p>
<p>No to your second question–the <code>pack_padded_sequence()</code> function expects a Python iterable object living on the CPU, so you should not call <code>.cuda()</code> on the <code>input_lengths</code> param.</p>
<p>Stephen</p>

username: ecolss
cooked: <p><a class="mention" href="/u/stephenrawls">@stephenrawls</a></p>
<p>OH, yes, I just found the root cause of my problem, thanks</p>

username: apaszke
cooked: <p>Also, the list of lengths is of the same size as the batches, so you can pass it as an input to the special module, and DataParallel should be able to slice it properly.</p>

username: agadetsky
cooked: <p><a class="mention" href="/u/stephenrawls">@stephenrawls</a> Hello, I did the same trick with variable sequence lengths, thank you for the solution. But the question is about why you don’t place padded_output on GPU using .cuda(). Without this option I can’t get it work.<br>
Also, I wanted to ask you what loss function do you use? I use CrossEntropy with zero weight for 0-th class, because this is class for padding. And it seems to work.<br>
<a class="mention" href="/u/apaszke">@apaszke</a> Hello, too) Will someone improve pack_padded_sequence() function for multigpu without such tricks? Maybe someone can add good example with variable sequence length rnnlms in multigpu mode. I saw many questions about this.</p>

username: Ravi_Teja
cooked: <p>I have a bit of confusion here.<br>
Can I create a variable at run-time and call .cuda on it while working on multiple GPUS.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/092c6cec78c07c11d842e8fb69a59dbb3eae8970.png" data-download-href="https://discuss.pytorch.org/uploads/default/092c6cec78c07c11d842e8fb69a59dbb3eae8970" title="34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/092c6cec78c07c11d842e8fb69a59dbb3eae8970_2_690x421.png" alt="34 PM" data-base62-sha1="1j9tEWrpDCuNoWUR444eBJYstHi" width="690" height="421" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/092c6cec78c07c11d842e8fb69a59dbb3eae8970_2_690x421.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/092c6cec78c07c11d842e8fb69a59dbb3eae8970_2_1035x631.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/092c6cec78c07c11d842e8fb69a59dbb3eae8970_2_1380x842.png 2x" data-dominant-color="1B1F2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">34 PM</span><span class="informations">2360×1442 500 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For the above code to work on multi GPU</p>
<ol>
<li>I need to call h_0.cuda(), c_0.cuda() in the forward function?</li>
<li>Can I just call .cuda() on the wcembeds variable which is created during run-time and expect it to work?</li>
</ol>
<p>From the above discussion my lengths tensor which is not a variable should not be converted to .cuda(). This means<br>
3. inputs —&gt; inputs.cuda()<br>
4. lengths → lengths <span class="hashtag-raw">#no</span> cuda transformation?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5039e65f332ae1d7ddbb8a2477ca0990e59c2f07.png" data-download-href="https://discuss.pytorch.org/uploads/default/5039e65f332ae1d7ddbb8a2477ca0990e59c2f07" title="46 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5039e65f332ae1d7ddbb8a2477ca0990e59c2f07_2_689x210.png" alt="46 PM" data-base62-sha1="brIflGS0ncU0NFMBLtw65hKNHIX" width="689" height="210" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5039e65f332ae1d7ddbb8a2477ca0990e59c2f07_2_689x210.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5039e65f332ae1d7ddbb8a2477ca0990e59c2f07_2_1033x315.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/5039e65f332ae1d7ddbb8a2477ca0990e59c2f07.png 2x" data-dominant-color="202230"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">46 PM</span><span class="informations">1374×420 49.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
5. Do I need to convert my original_indices tensor to .cuda()?</p>
<p>Sorry, for being very naive. I will have to spend some time understanding how stuff works on GPU, but, for the time-being could you guys help?</p>

username: Ravi_Teja
cooked: <p>The above code is for creating character level embeddings for each word in the input. The concatenation of the last hidden vector of the forward and backward RNN gives the resultant embedding for each word.</p>
<p>Also,<br>
self.char_embed = nn.Embedding(self.cvocab_size, self.cembed_dim, padding_idx=self.padding_idx)</p>
<p>I hope I am clear.</p>

username: tengerye
cooked: <p><a class="mention" href="/u/apaszke">@apaszke</a> Hi, do you happen to know the source code that we can reference please?</p>

username: mortazavi
cooked: <p>PyTorch 1.5 has completely fixed this issue . . . seamlessly . . . no more gerrymandering being required. I confirmed this today, in a project I’m working involving bidir GRUS on speech mfccs.</p>