Confusion about 3D LSTM input

username: Raphikowski
cooked: <p>Hi all,</p>
<p>I want to build a simple LSTM model and am a bit confused about the 3D input dimensions. According to the pytorch documentation the 3 dimensions represent (seq_len, batch, input_size). However, I cannot figure out why I need both the sequence length and the batch size here. Say, I have a 5 dimensional timeseries, that is, 5 feature dimensions. The timeseries has a total length of, say, 1000, with a batch size of 100. So I have read in some other threads, that in this case my input tensor would be (10, 100, 5) (I think). But does that mean that I feed my entire timeseries in at once in that 3D tensor?</p>
<p>Is there a certain reason why the LSTM would take the whole timeseries at once in a 3D tensor?</p>
<p>What would I be doing if I just want to do the splitting in batches in the train function as usual, i.e., that I iterate over the dataloader and sample batches of size 100 which I feed into the forward method of the LSTM?</p>
<p>Best wishes and thank you in advance!</p>

username: JuanFMontesinos
cooked: <p>If you check the sourcecode it just does iteratively run the lstm for input 1, adjust hidden state and process input 2 and so on.</p>

username: Raphikowski
cooked: <p>but what is then the difference between seq_len and batch? Why do we need that distinction?</p>

username: JuanFMontesinos
cooked: <p>Because a single sample is a sequence, eg, a sentence to be translated. However, you don’t usually train with a single sample but several, a batch to smooth gradients and to obtain generalizable normalization statistics.</p>

username: Raphikowski
cooked: <p>To me it still seems that one of these 3 dimensions is redundant. Because if I use, say, a 5 dimensional timeseries and I want to train it in a batched version, then I split that into junks. If the total timeseries had 1000 timepoints and my batches have size 100, then I would only require input tensors of (100, 5) as I understand it… But I would feed 10 of these tensors during one epoch to the LSTM</p>

username: JuanFMontesinos
cooked: <p>You may have to google how RNN works but basically you have TIMExBATCHxN_FEATURES.<br>
If you have a sample that sample can be size 100 or size 1000. It doesn’t mean that you split your timeseries into samples. Each sample is a totally different thing.</p>
<p>You are also forgetting that your sample may have different feature lengths, eg, a video is a HxW feature space, a sentence may be a onehotvector of Q elements.<br>
Also, length T is not fixed for each sample, how may have a video of 10 seconds and another one of 4.</p>
<p>Practical examples,<br>
you encode a word dictionary in a one hot vector of 100 elements. Thus your n_features is 100.<br>
Now you have a book, each sentence has a different length.<br>
for sentence one you may have 5 words --&gt; 5x100<br>
Sentence 2–&gt; 9 words --&gt; 9x100<br>
and so on up to sentence 4<br>
If you get a batch of sentences you will have Tx[S1…S4]x100 but T is different for each sample.</p>
<p>For a video you have have different lengths also.<br>
You n_features can be HxW<br>
Each video contains TxFPS frames<br>
if you have 5 videos you would get [TxFPS,5,HxW] features</p>

username: Raphikowski
cooked: <p>Okay so maybe my confusion comes about because I am looking at a special case. In my case I just have a long timeseries that represents a long trajectory of some object. I dont have any predefined division of that timeseries into subsections like e.g. sentences of a long text. So I can choose that division myself and I want all subsection to be of same length. And that length is my batchsize. I.e., I have a 5D timeseries of 1000 recordings and instead of plugging it in as a whole I will draw randomly samples of size 100 (=batchsize) and plug it into the RNN. In that case what would be the respective seq_len, batch, input_size?</p>

username: JuanFMontesinos
cooked: <p>Soo let’s say that you are using absolute cartessian coords. Your 5D tensor is BATCH,T,X,Y,Z.<br>
From here onwards let’s consider a sample as one segment of your long timeseries.</p>
<p>In order to feed this into an RNN you have to consider that each position of each sample at each time t is defined by [Xt,Yt,Zt], these would be your features. Whether these features are adequate or not to represent your data in this problem is not my concern.</p>
<p>Let me mention that you could consider the whole trajectory as a single sample. Assuming you want to split them. Following your case your input should be T,BATCH_SIZE,[X,Y,Z]<br>
Here</p>
<pre><code class="lang-auto">T --&gt; arbitrary time resolution chosen by you
N_SAMPLES = Timeseries_length/ T
BATCH_SIZE &lt;= N_SAMPLES 
</code></pre>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="nofollow noopener">LSTM docs</a><br>
Realize your <strong>input_size</strong> is 3 [XYZ] at the time of instantiating the RNN module.</p>
<p>What is the proper T? You have to struggle to discover it. Think that an RNN expects to encode the information of T events into the hidden state. The larger your sequence is the more parameter the hidden state needs to encode that information.  There is also a limit on how many information the hidden state can encode efficiently. (Note that translators work well for short sentences but tend to fail more for larger)</p>
<p>Realize that you don’t have specify T to the RNN instance, T can be variable even if you fix it. You have to specify input_size, which is 3 assuming spatial coordinates, but it could be whatever length according to the representation you use.</p>
<p>The hidden state can also have an arbitrary size, it’s a hyperparameter as it’s a representation of your data.</p>
<p>The last but not the least is you don’t really need to precompute splits of a fixed length, you can take random crops of variable length in order to make your network more robust.</p>
<p>Hope it helps!</p>

username: Raphikowski
cooked: <p>Yes, thanks a lot for your explanation!</p>