Variable input size to LSTM

Anshumaan_Dash: <p>Hi Everyone,</p>
<p>I am new to using LSTMs. I have the following requirements:</p>
<p>Input to lstm: [30, 16, 2]<br>
Output from lstm: [256, 1]</p>
<p>Currently, as per the documentation, the input can be of a specific length, say n. That is one dimensional. How to have a 3d input?</p>
<p>E.g. (  <em>batch size</em>  ,  <em>sequence length</em>  ,  <em>input dimension</em>  ) : I want the “Input dimension” as [30,16,2]</p>
<p>Also, in this case, what exactly is the “Sequence length”?</p>
<p>Thanks</p>

matthias.l: <p>When <code>LSTM</code> in contrast to <code>LSTMCell</code> consumes a sequence.</p>
<p>In this small gist I demonstrate it’s usage with batches:</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7#file-bieber_lstm-py-L139" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7#file-bieber_lstm-py-L139" target="_blank" rel="nofollow noopener">https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7#file-bieber_lstm-py-L139</a></h4>
<h5>bieber_lstm.py</h5>
<pre><code class="Python">#!/usr/bin/env python

import torch
import torch.nn as nn
import torch.nn.utils.rnn as rnn
import torch
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
from torch.nn import functional as F
</code></pre>
This file has been truncated. <a href="https://gist.github.com/lochbrunner/d9c77c3fbfd5301154c070561e294ed7#file-bieber_lstm-py-L139" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Anshumaan_Dash: <p>Thanks for the reply. However, I want the input itself to be 3d, not including the batch dimension.</p>
<p>Hope you can help me with that.</p>

matthias.l: <p>Could you flatten your input and split the output of the LSTM?<br>
I might be wrong, but LSTM has no interaction between the values in one sequence item.</p>

Anshumaan_Dash: <p>Sorry, but the requirement is that the input needs to be fed in as 30 samples of [16, 2] matrices, and the context vector after encoding needs to be [256, 1]. Cann’t change that.</p>
<p>If it’s not possible in torch, gotta do it in tensorflow.</p>

matthias.l: <p>You could use <a href="https://pytorch.org/docs/stable/torch.html#torch.flatten" rel="nofollow noopener">flatten</a> or <code>view</code> on your input.</p>
<p>Sorry I don’t get it it. The sequence (and batch size) of input in output stays the same.<br>
You can modify the shape with <code>view</code> if it makes sense for you.</p>
<pre><code class="lang-python">lstm = nn.LSTM(32,256,1)
# x of dimension 16x2

y, hc = lstm(torch.view(1, 32), hc)
y = y.view(256,1)
</code></pre>

Anshumaan_Dash: <p>Sure, I’ll give it a try.</p>
<p>256 is the sequence length, right? It’s the same as the output. Can u please explain what it signifies.</p>
<p>Thanks.</p>

matthias.l: <p>LSTM is used for sequence input, which is a tensor of variable length. (you can pad it, such that all sequences have the same fixed length).</p>
<p>In your example I can not see the sequence length. Is it <code>30</code>, <code>16</code> or <code>256</code>.</p>
<p>Sorry I am confused.</p>

Anshumaan_Dash: <p>Okay. Let me explain. I need an LSTM to accept 30 samples of (16,2) pose keypoints (they are coordinates on an image, 30 such images) and give out a single vector of length 256.</p>
<p>Hence, I need a single input batch of 30 samples, each with 16 keypoints, which are coordinates.</p>
<p>Hope it makes things clear.</p>

matthias.l: <p>I am not sure if LSTM is the correct module for something like that directly.<br>
But maybe the last output of the LSTM contains the information enough to compute the “single vector of length 256”. You could try <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_select" rel="nofollow noopener">index_select</a> and having several linear modules in order to map the sizes of the vectors.</p>