LSTM doesn't train

Floriane_S: <p>Hello everybody,</p>
<p>I learned Keras and now i will learn PyTorch, I am a beginner. I tried to use a LSTM (both in keras and PyTorch), and the one of PyTorch doesn’t train. I know approximately how the loss and the accuracy must be with Keras, and here, they doesn’t change during the epoch. So i did the assumption that my PyTorch code is not good. I juste want to use one LSTM layer with 256 filters and one linear layer.<br>
This is my PyTorch code :</p><aside class="onebox githubgist">
  <header class="source">

      <a href="https://gist.github.com/floschreiber/e1a60027606f2f771a05987defd5e955" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/floschreiber/e1a60027606f2f771a05987defd5e955" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/floschreiber/e1a60027606f2f771a05987defd5e955</a></h4>

  <h5>gistfile1.txt</h5>
  <pre><code class="Text">from multiprocessing import cpu_count
from pathlib import Path
from dataset_redshift import dataset_redshift
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
import torch
from torch import nn</code></pre>
    This file has been truncated. <a href="https://gist.github.com/floschreiber/e1a60027606f2f771a05987defd5e955" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I want to do some classifications, I have 3 classes, and the data are 1D time series with 17908 values. In the train, there are 14001 time series. The shape of the matrix is 14001x17908.</p>
<p>Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

ptrblck: <p>Based on your description and this code:</p>
<pre><code class="lang-python">X_valid=np.reshape(X_valid, (X_valid.shape[0], 1,X_valid.shape[1]))
X_train=np.reshape(X_train, (X_train.shape[0],1,X_train.shape[1]))  
</code></pre>
<p>as well as the usage of <code>batch_first=True</code> in the <code>nn.LSTM</code> module the input data should have a shape of <code>[batch_size=14001, seq_len=1, nb_features=17908]</code>.<br>
If that’s correct, note that you would only be using a single time step so that the <code>nn.LSTM</code> module might not be really useful.<br>
Could you explain the approach you’ve used in Keras/TF, i.e. which shapes (in particular sequence lengths) were used there?</p>
<p>PS: unrelated to this issue, but it seems you would like to transform the numpy arrays to tensors here:</p>
<pre><code class="lang-python">X_train, X_valid = [torch.tensor(arr, dtype=torch.float32) for arr in (X_train, X_valid)]
y_train, y_valid = [torch.tensor(arr, dtype=torch.long) for arr in (target_train, target_valid)]
</code></pre>
<p>If so, you could use:</p>
<pre><code class="lang-python">X_train = torch.from_numpy(X_train).float()
X_valid = ...
y_train = torch.from_numpy(y_train).long()
y_valid = ...
</code></pre>

Floriane_S: <p>thanks for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> !!</p>
<p>The keras approch I used :</p><aside class="onebox githubgist">
  <header class="source">

      <a href="https://gist.github.com/floschreiber/27582a6dad78da200d4d61e126608143" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/floschreiber/27582a6dad78da200d4d61e126608143" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/floschreiber/27582a6dad78da200d4d61e126608143</a></h4>

  <h5>LSTM_redshift</h5>
  <pre><code class="">import numpy as np
import matplotlib.pyplot as plt
import collections
from sklearn.model_selection import train_test_split
from keras.layers import Input, Dense, Dropout, Conv1D, MaxPooling1D,Flatten, LSTM
from keras.models import Model, Sequential
from keras import metrics
import keras
import matplotlib.pyplot as plt
</code></pre>
    This file has been truncated. <a href="https://gist.github.com/floschreiber/27582a6dad78da200d4d61e126608143" target="_blank" rel="noopener nofollow ugc">show original</a>
  <h5>main</h5>
  <pre><code class="">from dataset_redshift import dataset_redshift
from MLP_redshift import MLP_redshift
from CNN1D_redshift import CNN1D_redshift
from LSTM_redshift import LSTM_redshift

#from visu_data import visu_data
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import collections</code></pre>
    This file has been truncated. <a href="https://gist.github.com/floschreiber/27582a6dad78da200d4d61e126608143" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>i think i use the same sequence lengths…</p>
<p>thanks !!</p>
<p>thanks for the advice for the tensors transformation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Floriane_S: <p>In fact it is not really “time series” but it is probability density function (in log)  so it is complicated to have several sequence lengths, i think…</p>

ptrblck: <p>Thanks for the update.<br>
Based on the <a href="https://keras.io/api/layers/recurrent_layers/lstm/">Keras LSTM docs</a> it seems that the input should have the shape:</p>
<blockquote>
<p><strong>inputs</strong>: A 3D tensor with shape <code>[batch, timesteps, feature]</code>.</p>
</blockquote>
<p>The linked Keras implementation uses tensors as:</p>
<pre data-code-wrap="python"><code class="lang-python">X_test=np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1))
X_train=np.reshape(X_train, (X_train.shape[0],X_train.shape[1],1))	
X_validation=np.reshape(X_validation, (X_validation.shape[0],X_validation.shape[1],1))	
</code></pre>
<p>so it seems that the feature dimension is set to one and the temporal dimension is large.<br>
If I’m not misunderstanding the Keras docs or the posted Keras code, I guess this would be the main difference between both codes.</p>

Sandipan_Majhi: <p>Refer the docs of torch LSTM how the input data should be arranged. It is an excellent piece of documentation. Kind of felt like the input data is not arranged properly in the code. Definitely use the <code>batch_first = True</code> option in the lstm. Makes life a lot easier.</p>
<p><a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwj816OSnorxAhU54XMBHZMWBowQFjAAegQIAhAD&amp;url=https%3A%2F%2Fpytorch.org%2Fdocs%2Fstable%2Fgenerated%2Ftorch.nn.LSTM.html&amp;usg=AOvVaw3XGXwzPFs5Ktx9YTpWYj9A" rel="noopener nofollow ugc">Docs : LSTM_Pytorch</a></p>