Time-series multistep prediction LSTM Model (Recursive prediction)

username: 20am847
cooked: <p>Hello,<br>
I can’t believe how long it took me to get an LSTM to work in PyTorch<br>
and<br>
Still I can’t believe I have not done my work in Pytorch though.</p>
<p>My final goal is make time-series prediction LSTM model</p>
<ul>
<li>not just one step prediction but Multistep prediction model</li>
<li>So it should successfully predict <strong>Recursive Prediction</strong></li>
<li>The dataset is Power Consumption dataset (<a href="http://www.cs.ucr.edu/~eamonn/discords/" class="inline-onebox-loading" rel="noopener nofollow ugc">http://www.cs.ucr.edu/~eamonn/discords/</a>)<br>
<img src="/uploads/default/original/3X/4/4/44051543fccd36307d522556fdebc7d1a9d5b452.png" alt="112" data-base62-sha1="9HJmdKj5VgiSp1t4CVPQYc9oF1w" width="500" height="212"></li>
<li>So I can implement it to the Anomaly Detection (My final final final Goal)(<a href="https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-56.pdf" class="inline-onebox-loading" rel="noopener nofollow ugc">https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-56.pdf</a>)</li>
</ul>
<p>I created my custom dataset referenced by <a href="https://discuss.pytorch.org/t/understanding-lstm-input/31110" class="inline-onebox-loading">https://discuss.pytorch.org/t/understanding-lstm-input/31110</a><br>
and make LSTM model using scheduled sampling -  <a href="https://arxiv.org/abs/1506.03099" class="inline-onebox-loading" rel="noopener nofollow ugc">https://arxiv.org/abs/1506.03099</a><br>
decay schedules is Linear method.</p>
<p>The problem is, rather than making multistep predictions, <strong>my process has been blocked from the training stage</strong>. I plotted the prediction value obtained from the training, the result was such a  mess just a MESS.<br>
<img src="/uploads/default/original/3X/a/a/aa959e3922bc852e00dcb0ceed98aeaf365501f5.png" alt="1113" data-base62-sha1="ol3IXD5QzV2YUqigkRK6iqpx54x" width="630" height="308"></p>
<p>I referenced many git hub codes including official torch github (<a href="https://github.com/pytorch/examples/tree/master/time_sequence_prediction" class="inline-onebox-loading" rel="noopener nofollow ugc">https://github.com/pytorch/examples/tree/master/time_sequence_prediction</a>, <a href="https://github.com/chickenbestlover/RNN-Time-series-Anomaly-Detection" class="inline-onebox-loading" rel="noopener nofollow ugc">https://github.com/chickenbestlover/RNN-Time-series-Anomaly-Detection</a>)  to solve this problem and have seen a lot of topic related to LSTM, but I could not solve my problem. Even I sent an e-mail to the author of the git hub code, but he also said that he did not complete the multistep prediction for power demand dataset.</p>
<p>From <a href="http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A1149130&amp;dswid=3859" class="inline-onebox-loading" rel="noopener nofollow ugc">http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A1149130&amp;dswid=3859</a>,<br>
the <strong>Stateful LSTM method should be used</strong> because the dataset such as power demand has a long weekly cycle. So I tried various attempts to maintain the hidden state and cell state as 0 which means not to initialize it.</p>
<p>I’ve been on this for over a month and I’m so exhausted. I want to upload all the codes in my mind, but because it gets too long, I upload the code of the training part and the lstm model code first. In addition, if the other parts of code are needed, I can upload more. I need the help from anyone who know about lstm well.</p>
<blockquote>
<p>def train_data(model, lookahead, hidden_, cell_, dataloader, epoch, batch_size, device):<br>
model.train()</p>
<pre><code>total_loss = 0
pred_list = []
realY_list = []

optimizer = optim.Adam(model.parameters(), lr=learning_rate)

for iteration, (X, y) in enumerate(dataloader):

    iter_pred = [] # get output from teacher forcing or free learning train
    optimizer.zero_grad()

    X = X.transpose(0, 1).float().to(device)
    y_true = y.float().to(device)

    model_out, hidden_, cell_ = model(X, hidden_, cell_)

    iter_pred.append(model_out)

    for i in range(lookahead-1):
        '''For scheduled sampling (Teacher Forcing or Free Learning)'''
        '''Select T or F'''
        population, weights = get_population_weight(args, epoch)
        teacher_or_free = choice(population, weights)
        # print("Teacher or Free =&gt;", teacher_or_free)

        if teacher_or_free == "T":
            y_teacher = y_true[:, i].reshape(1, batch_size, 1)
            model_out, hidden_, cell_ = model(y_teacher, hidden_, cell_)
            iter_pred.append(model_out)

        elif teacher_or_free == "F":
            y_free = model_out.reshape(1, batch_size, 1)
            model_out, hidden_, cell_ = model(y_free, hidden_, cell_)
            iter_pred.append(model_out)

    y_pred = torch.cat(iter_pred, dim=0)
    y_true = y_true.view(-1)

    loss = loss_function(y_pred, y_true)
    loss.backward()
    optimizer.step()

    total_loss += loss

    pred_list.append(y_pred)
    realY_list.append(y_true)

print(epoch, "epoch")
print("loss =", total_loss)

return realY_list, pred_list, y_pred, hidden_, cell_
</code></pre>
<p>class LSTM(nn.Module):</p>
<pre><code>def __init__(self, input_dim, hidden_dim, output_dim, batch_size, num_layers, dropout, use_bn, device):
    super(LSTM, self).__init__()
    self.input_dim = input_dim
    self.hidden_dim = hidden_dim
    self.output_dim = output_dim
    self.batch_size = batch_size
    self.num_layers = num_layers
    self.device = device

    self.dropout = dropout
    self.use_bn = use_bn

    # Define the LSTM layer
    self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)

    # Define the output layer
    self.regressor = self.make_regressor()

def make_regressor(self):
    layers = []
    if self.use_bn:
        layers.append(nn.BatchNorm1d(self.hidden_dim))
    layers.append(nn.Dropout(self.dropout))

    layers.append(nn.Linear(self.hidden_dim, self.hidden_dim // 2))
    layers.append(nn.ReLU())
    layers.append(nn.Linear(self.hidden_dim // 2, self.output_dim))
    regressor = nn.Sequential(*layers)
    return regressor

def forward(self, lstm_input, hidden, cell):
    hidden_ = hidden.detach()
    cell_ = cell.detach()

    y_pred, (hidden_out, cell_out) = self.lstm(lstm_input, (hidden_, cell_))
    lstm_out = y_pred
    y_pred = self.regressor(lstm_out[-1].view(self.batch_size, -1))

    return y_pred, hidden_out, cell_out
</code></pre>
</blockquote>
<p>ANY help would be appreciated…Thanks…</p>

username: Andi
cooked: <p>Hi, I have a similar challenge, were You able to get further?<br>
I would like to build a simple functionality similar to fv prophet for x steps ahead and with another regressor, but cant find any working examples with stable refeed prognosis back to LSTM…<br>
Some examples start with encoder decoder, but I would like to try LSTM…<br>
Just asked also in Forum for examples.<br>
br Andy</p>