LSTM model does not change predictions

Ale_DZ: <p>I am implementing in PyTorch an LSTM model to predict if the closing value of a stock will go up or down in the next 5 and 10 minutes.<br>
Specifically, I am using 24 years of 5 minute data with 19 features, divided in chunks of one week per forecast (using 7 different stocks)<br>
The problem I’m facing is the fact that, no matter what, the LSTM model seems to predict values around one specific value to always minimize the loss, which it does not go down very much.</p>
<p>I pre-prepare the inputs and targets in torch.tensors with [batch_size, sequence_len, features] (which in my case is [32, 2016, 19]), normalize them between 0 and 1 and feed them to my LSTM model which is structured like this:</p>
<pre><code class="lang-auto">class MultiInputOutputLSTM(nn.Module):
  def __init__(self, input_size, hidden_size, num_layers, output_size, dropout, lr, batch_size):
    super(MultiInputOutputLSTM, self).__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.dropout = dropout
    self.batch_size = batch_size
    self.loss_list = []
    self.accuracy = 0
    self.predictions_list = [0]
    
    
    self.lstm = nn.LSTM(input_size = self.input_size, hidden_size = self.hidden_size, num_layers = self.num_layers, dropout = self.dropout,  batch_first=True)
    self.fc = nn.Linear(hidden_size, output_size, bias=True)
    self.sigmoid = nn.Sigmoid()
    self.criterion = nn.BCEWithLogitsLoss()
    self.optimizer = torch.optim.RMSprop(self.parameters(), lr = lr, alpha=0.9, weight_decay=1e-4,  momentum=0.5)
    self.scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(self.optimizer, 'min')

  def forward(self, x):
    
    h0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
    c0 = torch.zeros(self.num_layers, self.batch_size, self.hidden_size)
    lstm_out, _ = self.lstm(x, (h0, c0))
    output = self.fc(lstm_out[:, -1, :])

    return output

  def train_step(self ,x, y):
        self.train()
        predictions_1 = torch.round(self.forward(x))
        predictions = self.forward(x)
        if (predictions_1.detach().cpu().numpy()[0] == y.detach().cpu().numpy()[0]).all():
          self.accuracy += 1
        self.predictions_list.append(predictions.detach().cpu().numpy()[0][0])
        penalty = torch.mean((predictions-0.5)**2)
        loss = self.criterion(predictions, y) + penalty
        self.scheduler.step(loss)
        self.optimizer.zero_grad()
        self.loss_list.append(loss.item())
        mean_loss = sum(self.loss_list)/len(self.loss_list)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.parameters(), max_norm=6)
        self.optimizer.step()
        return loss.item(), mean_loss, self.accuracy, predictions.detach().cpu().numpy()[0], y.detach().cpu().numpy()[0]
  
  def test_step(self, inputs, targets):
    self.eval()
    with torch.no_grad():
      outputs = torch.round(self.forward(inputs))
      outputs = outputs.detach().cpu().numpy()[0]
      targets = targets.detach().cpu().numpy()[0]
      if (outputs == targets).all():
        accuracy = 1
      else:
        accuracy = 0
    return accuracy, outputs, targets
</code></pre>
<p>The targets are 1 if the price goes up, 0 else wise.<br>
The hyper parameters are:</p>
<pre><code class="lang-auto">input_size = 19
hidden_size = 3
num_layers = 5
output_size = 2
lr = 0.001
num_epochs = 5
batch_size = 32
dropout = 0.3
</code></pre>
<p>The model creates predictions at the start and then they will change ± 1e-3, 1e-4, so basically the same through the training.<br>
The train-test split is 85-15%.</p>
<p>Here is a list of what I have tried:<br>
lowering or increasing learning rate (from 0.00001 to 0.1), output size (with more and less forecasts), batch_size (from 1 to 256), num_layers (1-5), input_size (with 1, 2, 3… 19 features), dropout (0 to 0.5) and num_epochs (from 1 to 100).</p>
<p>I have tried Adam optimiser, then SDG, then RMSProp optimiser changing alpha, momentum and weight_decay.</p>
<p>Also tried BCELoss (with the sigmoid activation layer in self.forward()), L1Loss, MSELoss, CrossEntropyLoss before settling to a loss that penalises values too close to 0.5 (that does not help very much, but at this point I’ll try anything.<br>
The mean loss goes down from 0.924 to 0.889.</p>
<p>The accuracy is, as you can imagine, very low (from 5.56% to 11.67%): the only reason that it is not zero is because of pure luck.</p>
<p>I am also trying to forecast the difference between last item of closing column of the data frame and the future next two values, but… with the same problems.</p>
<pre><code class="lang-auto">Ticker - Df - Loss - Mean Loss - Predictions - Targets - Epoch
AAPL 1/120 0.808  0.808 [0.5642182  0.11684255] [1. 1.] 1/50 
AAPL 2/120 0.828  0.818 [0.5629723  0.12899561] [1. 1.] 1/50 
AAPL 3/120 0.834  0.823 [0.5472506  0.13749745] [1. 1.] 1/50 
AAPL 4/120 0.839  0.827 [0.52973044 0.13656825] [0. 0.] 1/50 
AAPL 5/120 0.821  0.826 [0.52082217 0.13681749] [0. 0.] 1/50 
AAPL 6/120 0.804  0.822 [0.5117941  0.14727135] [1. 1.] 1/50 
AAPL 7/120 0.779  0.816 [0.49826628 0.1547919 ] [0. 0.] 1/50 
AAPL 8/120 0.845  0.820 [0.49666688 0.1573844 ] [0. 0.] 1/50 
…
AAPL 1/120 0.762  0.751 [0.4200741  0.24348469] [1. 1.] 50/50 
AAPL 2/120 0.787  0.751 [0.43581426 0.23458275] [1. 1.] 50/50 
AAPL 3/120 0.797  0.751 [0.43012726 0.23912352] [1. 1.] 50/50 
AAPL 4/120 0.805  0.751 [0.43174705 0.23598525] [0. 0.] 50/50 
AAPL 5/120 0.794  0.751 [0.41290796 0.24462458] [0. 0.] 50/50 
AAPL 6/120 0.777  0.751 [0.42214283 0.2398287 ] [1. 1.] 50/50 
AAPL 7/120 0.747  0.751 [0.42418063 0.24077193] [0. 0.] 50/50 
AAPL 8/120 0.822  0.751 [0.44875318 0.23495609] [0. 0.] 50/50
...
</code></pre>
<p>I got the data from <a href="http://finam.ru" rel="noopener nofollow ugc">finam.ru</a>: if you want to download it be careful as the more you go back in time the more the data gets corrupted and requires some processing.<br>
With very little changes I saw that the predictions seem always to shift towards 0.5: a recurrent “anomaly” (i do not know if this is supposed to happen or not) is that the predictions are correct or very close, but the loss doesn’t go down or it just goes up.<br>
How the loss happens seems random and the results do not have any pattern.<br>
Any help is appreciated.<br>
Thank you</p>

soulitzer: <p>Some sanity checks:</p>
<aside class="quote no-group" data-username="Ale_DZ" data-post="1" data-topic="215006">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ale_dz/48/74527_2.png" class="avatar"> Ale_DZ:</div>
<blockquote>
<p>The accuracy is, as you can imagine, very low (from 5.56% to 11.67%): the only reason that it is not zero is because of pure luck.</p>
</blockquote>
</aside>
<p>Is this value better than a random guess? Which I’d imagine to be 1/2^num_outputs based on your criteria for setting accuracy=1.</p>
<p>Another thing to do is training on a single sample and see if you can overfit/ make training loss go to zero</p>
<blockquote>
<p>LSTM model to predict if the closing value of a stock will go up or down in the next 5 and 10 minutes.</p>
</blockquote>
<p>Do you mean to predict whether the closing price is higher or lower than the price of the stock 10 minutes before closing?</p>

Ale_DZ: <p>No, in fact a random guess would have a better chance of being correct.<br>
In the main script I was training with the full dataframe for 50 epochs (passing one week-long dataframe each time ) with the results that you can see, now I’m trying training with each dataframe 50 times. Even if i’m setting a loss that penalises heavily every number close to 0.5 (L1Loss + custom penalty) still I’m getting numbers in the range 0.30-0.35, without them ever going up or down (and loss has the same pattern): it seems that the LSTM model is trying its best to not learn, or to get to a local minimum of the loss and then stop trying.<br>
Someone suggested in another forum that it might be vanishing gradient and it’s somewhat correct, all the grad norm are in the range 0.1 - 0.0001 range but I really do not know how to change this.<br>
Lastly: given this one-week-long dataframe, I want to predict if in the 5 and 10 minutes from the last time of the dataframe I’m training with the price will go up ( which is 1 in the predictions ) or go down/stay the same (which is 0 in the predictions).<br>
Thank you for the reply</p>