How to process variable length sequence of images with CNN

yanweiw: <p>Hi -</p>
<p>I have images in sequences of variable length. I am trying to first process each image with a CNN to get a feature representation. Once I have variable-length sequences of features, I will process each sequence through an LSTM. I know that I can pad the variable-length sequence of feature vectors with zeros and create a packed sequence with pack_padded_sequence() before sending it to LSTM. But how can I do the same to my data before sending it to CNN? I do not want to pad the image sequence with zeros and let the CNN waste computation on processing images of all zeros. Any thoughts?</p>
<p>The way I am currently hacking it is that: I am still padding the image sequence. Once I get a PackedSequence object using pack_padded_sequence(), I can call the ‘forward function’ of my CNN model on the ‘data’ attribute of the PackedSequence object. I can then manually construct a new PackedSequence object with the output from my CNN model, effectively replacing the data field of the old PackedSequence object. Finally, I send the new PackedSequence to the downstream LSTM. Is this an okay workaround?</p>
<p>Thanks!</p>

ptrblck: <p>I’m not sure what an image sequence is. Could you post the shape of your current input and also show how you are currently padding it, please?</p>

yanweiw: <p>Sure, each data sequence is of format <strong>[image_0, action_1, image_1, action_2, image_2, action_3, image_3…]</strong>, and the task is to predict action sequence <strong>[action_1, action_,…, action_k]</strong> given image sequence <strong>[image_0, image_1,…, image_k]</strong>. The image sequence of a variable length <strong>k+1</strong> in a batch, so I pad each sequence with zero images until sequence length is <strong>max_seq_len</strong>. The batched input is thus of shape <strong>(B, max_seq_len, C, H, W)</strong>.</p>
<p>My network uses a <strong>CNN model</strong> to embed each <strong>image</strong> into a feature vector <strong>state</strong>, and then uses a <strong>LSTM model</strong> to predict the action sequence from the state sequence. To avoid embedding all the zero images that are just padding,</p>
<ol>
<li>I use <code>pack_padded_sequence(images, image_seq_lens, batch_first=True, enforce_sorted=False)</code> to produce <strong>packed_images</strong>.</li>
<li>Run the <strong>CNN</strong> on <strong>packed_images.data</strong> to get <strong>packed_states_data</strong>.</li>
<li>Instantiate (a hacked advised against)  <code>packed_states = PackedSequence(packed_states_data, packed_images.batch_sizes, packed_images.sorted_indices, packed_images.unsorted_indices)</code>
</li>
<li>Send <strong>packed_states</strong> to the <strong>LSTM</strong> to predict <strong>packed_actions</strong>
</li>
<li>Unpack actions with <code>actions, action_seq_len = pad_packed_sequence(packed_actions, batch_first=True, total_length=self.max_seq_len-1)</code>
</li>
</ol>
<p>To calculate the loss between predicted <strong>actions</strong> and ground truth <strong>actions_gt</strong>, I also pack <strong>actions_gt</strong> into <strong>packed_actions_gt</strong>. I use <code>assert (packed_actions.sorted_indices == packed_actions_gt.sorted_indices).all()</code> to make sure they are sorted in the same way before packing. Then I compute the loss on <strong>packed_actions.data</strong> and <strong>packed_actions_gt.data</strong>.</p>
<p>My questions are that 1) are these valid operations? 2) is there going to be a performance issue (longer time) with numerous packing and padding back and forth on multiple GPUs? Right now my batched input live on GPU devices and these hacks happen there. Thank you for your time!</p>

ptrblck: <ol>
<li>
<p>I think your general approach is correct. Alternatively to the padding you could also try to create batches with the same <code>image_seq_lens</code>, but depending on your data distribution this might not be easy.</p>
</li>
<li>
<p>You should see a slightly lower performance for the packing and unpacking, but if there is no way around this, you wouldn’t have a baseline to compare against.</p>
</li>
</ol>
<p>One thing I’m a bit worried about is the usage of the <code>.data</code> attribute.<br>
You’ve mentioned that you are using <code>packed_actions.data and packed_actions_gt.data</code> to compute the loss, which would detach these tensors (especially the model output) and you should see an error.<br>
Could you remove the <code>.data</code> usage and try to calculate the loss directly using these tensors?</p>

yanweiw: <p>But <strong>packed_actions</strong> and <strong>packed_actions_gt</strong> are <strong>PackedSequence</strong> objects, how can I compute a loss on them?</p>
<p>Right now there is no syntax error, and model seems to be able to train a little bit. I originally thought that <strong>packed_actions.data</strong> only access the data field of PackedSequence, and <strong>packed_actions.data.data</strong> will detach the tensor from the graph.</p>
<p>If <strong>packed_actions.data</strong> will detach the wrapped tensor, my approach outlined above will not work. Is there a way to access the tensors that <strong>PackedSequence</strong> holds while leave the tensors in the computation graph?</p>
<p>Thank you!</p>

ptrblck: <p>Yeah, I think you are right and the <code>.data</code> attribute is used in the <code>PackedSequence</code>, so skip my comment on the usage of <code>.data</code> in this context.</p>

abhiray92: <p>Hi <a class="mention" href="/u/yanweiw">@yanweiw</a> can we have the full code that you are using to check the condition of the missing sequence and then padding the same to that sequence and unpacking the same at the LSTM end?<br>
Thanks</p>

3nomis: <p>The Idea behind this is to avoid useless convolutions on padded frames right? How do you ensure the convolutions are done correctly on the <code>packed_images.data</code>?</p>

Messiah: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Is this approach indeed feasible? I am facing a similar situation where I am accessing the <code>.data</code> attribute of PackedSequence objects a few times. To elaborate, I am passing the <code>.data</code> tensor of a PackedSequence object of image data to a CNN and then concatenating its output with another <code>.data</code> tensor. I then (similar to this post) create a new <code>PackedSequence</code> object of this combined data using the <code>batch_sizes</code> attribute of either of the two initial <code>PackedSequence</code> objects, before finally passing it through an LSTM layer.</p>
<p>Referring to <a href="https://discuss.pytorch.org/t/accessing-data-in-packedsequence-when-batch-first-true/28752/2" class="inline-onebox">Accessing `.data` in PackedSequence when `batch_first=True` - #2 by albanD</a>, it seems like accessing <code>.data</code> is advised against and can break parts of the computation graph. Might you have any new opinions regarding this approach?</p>

ptrblck: <p>No, please disregard my comment about the <code>.data</code> attribute in this thread.<br>
I wanted to warn against using the <code>tensor.data</code> attribute directly, as its usage is deprecated and could break things. However, the <code>PackedSequence</code> apparently also has a <code>.data</code> attribute (I wish the name would be different <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"> ), which is fine to access.</p>