How to define and train CNN LSTM many to one?

odats: <p>What is the right way to define and train in batches simple LSTM (many to one) with CNN inputs?<br>
NOTE: train both networks LSTM and CNN</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce3f42f3ff85b59760fd416309cae8d976310f1f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ce3f42f3ff85b59760fd416309cae8d976310f1f" title="IMG_4567"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce3f42f3ff85b59760fd416309cae8d976310f1f_2_574x500.jpeg" alt="IMG_4567" data-base62-sha1="tqxTyIgCaVLwCHI1ckolKv1pcv5" width="574" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce3f42f3ff85b59760fd416309cae8d976310f1f_2_574x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce3f42f3ff85b59760fd416309cae8d976310f1f_2_861x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce3f42f3ff85b59760fd416309cae8d976310f1f_2_1148x1000.jpeg 2x" data-dominant-color="A5937C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_4567</span><span class="informations">3013×2624 1.15 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vdw: <p>I’ve neber tried it, and I’m probably a bit naive, but should this pseudo code work:</p>
<pre><code>cnn = MyCnnModel()
lstm = MyLstmModel()

# Go through each image sequence in the dataset
for img_seq_batch in dataset:
    # Initialize hidden state of LSTM
    hidden = init_hidden()
    output, hidden = forward(img_seq_batch, hidden)
    # Push last hidden state trough additional layers
    # Calculate loss
    # Backprop

def forward(img_seq_batch, hidden):
    # Loop over each image in the sequences
    for img_batch in img_seq_batch:
        # Push batch of images through CNN
        encoded = cnn(img_batch)
        # Push CNN output through LSTM (just one time step)
        output, hidden = lstm(encoded, hidden)
    # Return the last output/hidden of the LSTM
    return output, hidden</code></pre>

odats: <p>Yes, it should work and it was my first idea.</p>
<aside class="quote no-group" data-username="vdw" data-post="2" data-topic="74829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>for img_batch in img_seq_batch:</p>
</blockquote>
</aside>
<p>The question is can we skip this loop and do in one transaction? If no, then I accept this answer.</p>

vdw: <p>Hm, not sure why the loop would be a problem. The heaving lifting is still done under the hood by PyTorch. I would also assume that when you give a sequence the <code>nn.LSTM</code> that internally it also just loops over each item.</p>
<p>Maybe another alternative – again, I haven’t tried it:</p>
<ul>
<li>Treat a sequence of images as a batch of images and push it through the CNN</li>
<li>Treat the batch of encoded images again as a sequence and push it in one go through the LSTM (no loop required)</li>
</ul>
<p>Not idea how to extend it to support batches of images sequences, or if it possible at all.</p>

odats: <p>Thanks, I did in this way. It works fine.</p>