LSTM model not learning

alexmulo: <p>Hello everyone,</p>
<p>I did some research but I couldn’t find any solutions at the moment.</p>
<p>I am trying to make categorical prediction of a time series dataset. I have a train dataset with the follow size:<br>
torch.Size([3749, 1, 62]): No. of samples, windows of 1 day, 62 features<br>
labels:<br>
torch.Size([3749]) with category 0,1,2</p>
<p>This is my model:</p>
<pre><code class="lang-auto">
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim) 
        
    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim)
        out, (hn, cn) = self.lstm(x, (h0, c0))
        out = self.fc(out) 
        out = F.softmax(out,dim=1)
        return out
</code></pre>
<p>This is the train loop:</p>
<pre><code class="lang-auto">
input_dim = 62
hidden_dim = 80
num_layers = 1
output_dim = 1
num_epochs = 10

model = LSTM(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers)
criterion = torch.nn.MSELoss()
optimiser = torch.optim.SGD(model.parameters(), lr=0.01)

hist = np.zeros(10)

lstm = []

for t in range(num_epochs):
    optimiser.zero_grad()
    y_train_pred = model(X_train)    

    loss = criterion(y_train_pred, y_train.reshape(y_train.shape[0],1,1))
    print("Epoch ", t, "MSE: ", loss.item())
    hist[t] = loss.item() 

    loss.backward()
    optimiser.step()
</code></pre>
<p>This is the output:</p>
<pre><code class="lang-auto">Epoch  0 MSE:  0.23072819411754608
Epoch  1 MSE:  0.23072819411754608
Epoch  2 MSE:  0.23072819411754608
Epoch  3 MSE:  0.23072819411754608
Epoch  4 MSE:  0.23072819411754608
Epoch  5 MSE:  0.23072819411754608
Epoch  6 MSE:  0.23072819411754608
Epoch  7 MSE:  0.23072819411754608
Epoch  8 MSE:  0.23072819411754608
Epoch  9 MSE:  0.23072819411754608
</code></pre>
<p>As you can see the model does not learn and I happen the same even with more epochs.</p>
<p>Do you have any ideas?</p>
<p>Thanks</p>

Dwight_Foster: <p>This is happening because in your training loop you are only looping over your epochs and not your dataset. Because of this X_train never changes so your model predicts the same thing every time and the loss will not change. You need to also loop over your dataset by doing something like this</p>
<pre><code class="lang-auto">for t in range(num_epochs):
    for batch_idx, (inputs, targets) in enumerate(dataloader):
         optimiser.zero_grad()
         y_train_pred = model(inputs)    

         loss = criterion(y_train_pred, targets.reshape(targets.shape[0],1,1))
         loss.backward()
        optimiser.step()
    print("Epoch ", t, "MSE: ", loss.item())
    hist[t] = loss.item()
</code></pre>
<p>Whatever you dataset is you need to loop through it so you get different batches.</p>

alexmulo: <p>many thanks for your reply. This is also something I have been trying:</p>
<pre><code class="lang-auto">lstm = []

for t in range(num_epochs):
    for inputs, targets in zip(X_train,y_train):
         optimiser.zero_grad()
         y_train_pred = model(inputs)    

         loss = criterion(y_train_pred, targets.reshape(targets.shape[0],1,1))
         loss.backward()
         optimiser.step()
    print("Epoch ", t, "MSE: ", loss.item())
    hist[t] = loss.item()
</code></pre>
<p>However using my actual dataset I got this error:<br>
<code>RuntimeError: input must have 3 dimensions, got 2</code></p>
<p>I should reshape my dataset to avoid this error?</p>

Dwight_Foster: <p>Yes in your training loop you can just say</p>
<pre><code class="lang-auto">input = input.unsqueeze(1)
target = target.unsqueeze(1) 
</code></pre>
<p>and that should help.</p>

alexmulo: <p>Hi,<br>
thanks for your reply.</p>
<p>I solved using TensorDataset and DataLoader:</p>
<pre><code class="lang-auto">train_tensor = torch.utils.data.TensorDataset(X_train,y_train)
train_loader = torch.utils.data.DataLoader(dataset = train_tensor, batch_size = 1)
</code></pre>
<p>Now the loop looks like this:</p>
<pre><code class="lang-auto">hist = np.zeros(10)
lstm = []
for t in range(num_epochs):
    for batch_idx, (inputs, targets) in enumerate(train_loader):
         #model.zero_grad()
         optimiser.zero_grad()
         y_train_pred = model(inputs)    

         loss = criterion(y_train_pred.reshape(1), targets.reshape(1))
         loss.backward()
         optimiser.step()
    print("Epoch ", t, "MSE: ", loss.item())
    hist[t] = loss.item()
</code></pre>
<p>However, the result looks a bit different but it still does not learn:</p>
<pre><code class="lang-auto">Epoch  0 MSE:  1.0
Epoch  1 MSE:  1.0
Epoch  2 MSE:  1.0
Epoch  3 MSE:  1.0
Epoch  4 MSE:  1.0
Epoch  5 MSE:  1.0
Epoch  6 MSE:  1.0
Epoch  7 MSE:  1.0
Epoch  8 MSE:  1.0
Epoch  9 MSE:  1.0
</code></pre>
<p>I can only imagine that dataset shape is not appropriate or something in the model.<br>
Do you have any idea?</p>
<p>Thanks</p>

Dwight_Foster: <p>Why are you reshaping the input to the loss function? Also why is your batch_size only 1?</p>

alexmulo: <p>Hi,<br>
I solved the problem. As I mentioned at the very beginning this i classification problem with multiple classes. This is a time series where each step is one day.</p>
<p>I changed everything in this way and it works quite well:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers)
        self.fc = nn.Linear(hidden_dim, output_dim) 
        
    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim)
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc(out[-1]) 
        #out = F.softmax(out,dim=1)
        return out

input_dim = 62
hidden_dim = 80
num_layers = 1
output_dim = 3 #No of classes
num_epochs = 10

model = LSTM(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers)
criterion = torch.nn.CrossEntropyLoss()
optimiser = torch.optim.Adam(model.parameters(), lr=0.01)

hist = np.zeros(10)
lstm = []
for t in range(num_epochs):
    for batch_idx, (inputs, targets) in enumerate(train_loader):
         #model.zero_grad()
         optimiser.zero_grad()
         y_train_pred = model(inputs)    

         loss = criterion(y_train_pred, targets.long())
         loss.backward()
         optimiser.step()
    print("Epoch ", t, "MSE: ", loss.item())
    hist[t] = loss.item()
</code></pre>
<p>Now I need to find out how to make the prediction to know the predicted category.</p>