Making Predictions with Time Series Classification model

noah14: <p><strong>Problem</strong></p>
<p>I am a total beginner with pytorch and machine learning in general.</p>
<p>I trained a time series classification model for predicting if a mountainbiker is in the air (jumping) or not, based on X-,Y- and Z - acceleration data of the biker.</p>
<p>I used a tutorial from Venelin Valkov as a template, where he predicted the surfaces robots where standing on, based on acceleration data from the robots.</p>
<p>As i followed his instructions with my own dataset, i am pretty confident, that my model makes sense and looking at the results and a confusion matrix it performs pretty well.</p>
<p>Now i want to evaluate the model by predicting the airtime of whole runs of a biker. Every run was filmed, so i could compare the predicted airtime with the ground truth from the video.<br>
So i would like to upload a csv file containing the data of one run and try my model on it, to get predictions of every sample if its “Airtime” or “No Airtime”.</p>
<p>I have converted the csv to a tensor and i know that I have to use model.eval and torch.no_grad for inferences, but up to now I did not succeed.</p>
<p>So my question is, if it is possible to run my model on a single csv file (not a whole dataset as in training) and make predictions and what is the easiest way to do so?</p>

blueeagle: <p>It is of course possible to make individual predictions with your trained model, but how exactly you should do this depends, for example, on your used data representation and model architecture.</p>
<aside class="quote no-group" data-username="noah14" data-post="1" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bcef8e/48.png" class="avatar"> noah14:</div>
<blockquote>
<p>I have converted the csv to a tensor</p>
</blockquote>
</aside>
<p>I assume your tensor (lets call it <code>x</code>) is of shape <code>[B,F]</code>, where each row represents one sample you want to classify (so B samples in total).<br>
Then you should be able to make a prediction for each of the samples in your tensor with</p>
<pre><code class="lang-auto">predictions = model(x)
</code></pre>
<p>Please let me know if I misunderstood your problem. If so, maybe you could provide some of your code to make the question more clear.</p>

noah14: <p>Thanks for the quick response.</p>
<p>The input needed for my model to make predictions seems to be of shape [batch size, length, features], so i expanded my data to a 3d tensor</p>
<pre><code class="lang-auto">df_test_conts = test_data[['x_test','y_test','z_test']]
test_conts = df_test_conts.astype(np.float32)
test_conts = torch.tensor(test_conts.values).cuda()
test_conts.unsqueeze_(-1)
test_conts = test_conts.transpose(1, 0)
test_conts = test_conts.transpose(2, 0)
test_conts = test_conts.expand(1,2817,3)
</code></pre>
<p>So i end up with a tensor with one batch and length of 2817 (test.csv has 2817 samples).</p>
<p>Then i tried to make predictions like this:</p>
<pre><code class="lang-auto"># make predictions
with torch.no_grad():
  model.eval()
  
  prediction = model(test_conts)
</code></pre>
<p>But the prediction i receive is now a tuple, not as expected a tensor, and looks like this.</p>
<pre><code class="lang-auto">(0, tensor([[-5.3364,  4.6702]], device='cuda:0'))
</code></pre>
<p>I tried then to convert the prediction back to a pandas dataframe, but this is not possible with a tuple. I am also not quite sure if the prediction did what i wanted it to do, which is to predict every sample if it is “air” or “noair”.</p>
<p>Sorry if my I am making stupid mistakes or explaining badly, but I am really new to this topic.</p>

blueeagle: <p>I think it would be helpful if you could provide the implementation of your model.<br>
Moreover I am not quite sure what you are doing here:</p>
<aside class="quote no-group" data-username="noah14" data-post="3" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bcef8e/48.png" class="avatar"> noah14:</div>
<blockquote>
<pre><code class="lang-auto">test_conts.unsqueeze_(-1)
test_conts = test_conts.transpose(1, 0)
test_conts = test_conts.transpose(2, 0)
</code></pre>
</blockquote>
</aside>
<p>In my opinion it seems that you could replace that by simply doing:</p>
<pre><code class="lang-auto">test_conts.unsqueeze_(0)
</code></pre>
<p>Could you also provide the shape of <code>test_conts</code> after this line of code?</p>

noah14: <aside class="quote no-group" data-username="blueeagle" data-post="4" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blueeagle/48/47881_2.png" class="avatar"> blueeagle:</div>
<blockquote>
<p><code>test_conts.unsqueeze_(0)</code></p>
</blockquote>
</aside>
<p>Yeah, you are right, this does the same thing, my bad.</p>
<p>This is the shape of test_conts after adding the fake batch dimension:</p>
<pre><code class="lang-auto">test_conts.size()
torch.Size([1, 2817, 3])
</code></pre>
<p>Here is the Implementation of my model:</p>
<pre><code class="lang-auto">class SequenceModel(nn.Module):

  def __init__(self, n_features, n_classes, n_hidden=256, n_layers=3):
    super().__init__()

    self.lstm = nn.LSTM(
        input_size = n_features,
        hidden_size = n_hidden,
        num_layers = n_layers,
        batch_first = True,
        dropout = 0.75

    )

    self.classifier = nn.Linear(n_hidden, n_classes)

  def forward(self,x):
    self.lstm.flatten_parameters()
    _, (hidden,_) = self.lstm(x)

    out = hidden [-1]
    return self.classifier(out)


class StatePredictor(pl.LightningModule):

  def __init__(self, n_features: int, n_classes: int):
    super().__init__()
    self.model = SequenceModel(n_features, n_classes)
    self.criterion = nn.CrossEntropyLoss()

  def forward (self, x, labels = None):
    output = self.model(x)
    loss = 0
    if labels is not None:
      loss = self.criterion(output, labels)
    return loss, output

  def training_step(self, batch, batch_idx):
    sequences = batch["sequence"]
    labels = batch["label"]
    loss, outputs = self(sequences, labels)
    predictions = torch.argmax(outputs, dim=1)
    step_accuracy = accuracy(predictions, labels)

    self.log("train_loss", loss, prog_bar=True, logger=True)
    self.log("train_accuracy", step_accuracy, prog_bar=True, logger=True)
    return {"loss": loss, "accuracy": step_accuracy}

  def validation_step(self, batch, batch_idx):
    sequences = batch["sequence"]
    labels = batch["label"]
    loss, outputs = self(sequences, labels)
    predictions = torch.argmax(outputs, dim=1)
    step_accuracy = accuracy(predictions, labels)

    self.log("val_loss", loss, prog_bar=True, logger=True)
    self.log("val_accuracy", step_accuracy, prog_bar=True, logger=True)
    return {"loss": loss, "accuracy": step_accuracy}
  
  def test_step(self, batch, batch_idx):
    sequences = batch["sequence"]
    labels = batch["label"]
    loss, outputs = self(sequences, labels)
    predictions = torch.argmax(outputs, dim=1)
    step_accuracy = accuracy(predictions, labels)

    self.log("test_loss", loss, prog_bar=True, logger=True)
    self.log("test_accuracy", step_accuracy, prog_bar=True, logger=True)
    return {"loss": loss, "accuracy": step_accuracy}


  def configure_optimizers(self):
    return optim.Adam(self.parameters(), lr=0.0001)

model = StatePredictor(
    n_features=len(FEATURE_COLUMNS),
    n_classes=len(label_encoder.classes_)
    )
</code></pre>

blueeagle: <p>The model implementation looks fine to me.<br>
If I run your code with</p>
<pre><code class="lang-auto">test_conts = torch.rand((1, 2817, 3))
model = SequenceModel(3, 2, 256, 3)
prediction = model(test_conts)
</code></pre>
<p>I receive a prediction of shape <code>[1,2]</code>, so everything seems to work fine here.</p>
<aside class="quote no-group" data-username="noah14" data-post="3" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bcef8e/48.png" class="avatar"> noah14:</div>
<blockquote>
<p><code>(0, tensor([[-5.3364,  4.6702]], device='cuda:0'))</code></p>
</blockquote>
</aside>
<p>Given you use the model implementation above, your model shouldn’t be able to return a tuple at all.<br>
Maybe you did change your model implementation in the meantime?</p>

noah14: <aside class="quote no-group" data-username="blueeagle" data-post="6" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blueeagle/48/47881_2.png" class="avatar"> blueeagle:</div>
<blockquote>
<pre><code class="lang-auto">test_conts = torch.rand((1, 2817, 3))
model = SequenceModel(3, 2, 256, 3)
prediction = model(test_conts)
</code></pre>
</blockquote>
</aside>
<p>I get the same shape with this code, so no i don´t think i changed my model implementation.</p>
<p>I think the cause of this Output</p>
<aside class="quote no-group quote-modified" data-username="noah14" data-post="3" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/bcef8e/48.png" class="avatar"> noah14:</div>
<blockquote>
<p>(0, tensor([[-5.3364, 4.6702]], device=‘cuda:0’))</p>
</blockquote>
</aside>
<p>was that in training i had Sequences with each 30 samples. Within a sequence all samples were classified 0 or 1, respectively “No Air” or “Air”. And the model tries to find out which sequences of my “Training-Dataset” are 0 or 1, which worked pretty good.</p>
<p>So when feeding new unseen data with X samples and no sequencing, apparently the model tries to find out if the data as a whole is 0 or 1 (0 in this case). But i want my model to predict the label of every line/sample of the data now and I am not figuring out how to do so. I hope I made my problem more clear with this.</p>

blueeagle: <p>Ok now I see. As far as I understood you trained your model on a sequence2one task i.e. there was ONE label associated with a whole sequence. Now you want to use this trained model in a sequence2sequence task, and to be more precise you want to classify each element in the sequence whether it is „No Air“ or „Air“.<br>
Is that right?</p>
<p>The implementation of your model, that you posted above, is however only suitable for sequence2one tasks. The LSTM cell takes one element of the sequence (x_i) as input at a time, does some calculations based on this input and two „hidden states“ (h_i and c_i), updating the hidden states and generating an output variable (y_i). The updated hidden states are then used together with the next sequence element (x_i+1) as input in the next time step.You can find a illustration of the concept here <a href="https://i.stack.imgur.com/SjnTl.png" rel="noopener nofollow ugc">PyTorch LSTM Cell</a> (taken from this <a href="https://stackoverflow.com/questions/48302810/whats-the-difference-between-hidden-and-output-in-pytorch-lstm" rel="noopener nofollow ugc">post</a>)</p>
<p>In your implementation you are only taking the final hidden state h_N of the LSTM and pass it to a classifier. This implies the reasonable assumption that the final hidden state contains some information about the whole sequence. The classifier finally predicts „No Air“ or „Air“ from this final hidden state. As you see, this will only work for sequence2one tasks, as your classifier can only predict 1 label for the whole sequence.</p>
<p>If you want to use your LSTM for a sequence2sequence task I am seeing 2 options:</p>
<ol>
<li>You change your implementation to not only return the last hidden_state h_i but also the generated outputs y_i. You could try using your trained classifier to predict „No Air“ / „Air“ from every y_i, and assign the prediction to the input of this tilmestep (x_i) or even reducing the number of hidden_features to the number of your classes, applying a softmax to the outputs y_i and taking that as predictions. However, I am not sure if this will give you reasonable results.</li>
<li>You classify „No Air“ / „Air“ from every intermediate hidden state „h_i“. However I am not aware of a method to store the hidden_states for all timesteps during forwardpass, so I would assume you would have to write a custom loop over the LSTM cell in order to get these values</li>
</ol>
<p>I hope I was finally able to solve your question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

noah14: <p>Thanks a lot for this thorough explanation, now i understand my model a lot better <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="blueeagle" data-post="8" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blueeagle/48/47881_2.png" class="avatar"> blueeagle:</div>
<blockquote>
<p>If you want to use your LSTM for a sequence2sequence task I am seeing 2 options:</p>
</blockquote>
</aside>
<p>I did not succeed with these 2 suggestions, but i managed to do predictions of single runs by sequencing them like the training data set, but making each sample a own sequence. (e.g. 2817 sequences in a file with 2817 samples). I also provided the Y-Data containing the Ground Truth and the corresponding sequence(“sample”)-number. So the same procedure as in training.</p>
<p>The prediction is now working but not as efficient as in training. In training I am receiving an accuracy of 95%. Now with the prediction I am predicting 95% of NOAIR correctly, but only 77% of AIR.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b04b55d71075363d09c978da7cd603d023acda95.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b04b55d71075363d09c978da7cd603d023acda95" title=""><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04b55d71075363d09c978da7cd603d023acda95_2_500x375.jpeg" alt="" data-base62-sha1="p9zrhdndV3opC3s3OyACn1xqhTv" width="500" height="375" role="presentation" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04b55d71075363d09c978da7cd603d023acda95_2_500x375.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04b55d71075363d09c978da7cd603d023acda95_2_750x562.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/0/b04b55d71075363d09c978da7cd603d023acda95.jpeg 2x" data-dominant-color="C3CEDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">805×603 14.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Ground Truth is 224 Samples AIR and 2593 Samples NOAIR</p>
<p>I have tried the following to improve the performance:</p>
<ul>
<li>
<p>Changing the number of n_hidden</p>
</li>
<li>
<p>Changing the batch size</p>
</li>
<li>
<p>Changing the dropout</p>
</li>
<li>
<p>Changing the learning rate</p>
</li>
</ul>
<p>But this is the best result i could achieve. Are there any other ways to improve the performance of a LSTM-Model? Or do you think the Training-Dataset is not sufficiently enough for this task (about 50000 samples)?</p>
<p>I am sorry for asking so many questions, but i could not find a solution elsewhere <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

blueeagle: <p>If I got it right, you are now making your predictions based on only one time-step of the time-series. Therefore your model is not able to exploit timely dependencies.<br>
If you want to improve your results I would suggest to try one of these options again:</p>
<aside class="quote no-group" data-username="blueeagle" data-post="8" data-topic="147322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blueeagle/48/47881_2.png" class="avatar"> blueeagle:</div>
<blockquote>
<p>If you want to use your LSTM for a sequence2sequence task I am seeing 2 options:</p>
</blockquote>
</aside>
<p>Apart from that the problem that you describe consists in the accuracy on the test set being significantly lower than on the Training set. Lower Test-Accuracy is probably caused by one of the following reasons:</p>
<ol>
<li>Overfitting on the training data<br>
You can check for that by using a validation set and overcome it by implementing early stopping during training</li>
<li>your test data has a different underlying distribution than your training data<br>
Was it generated by  another data generation process? This could for example be the case if the time-series was recorded by a bike-rider not in the training set or on a hill that was not present in the training-set.  Are you using a single time-series as test-set? Try to increase the test-set size, e.g. use 60% of your data for training, 20% for validation and 20% for testing. That should help making your evaluation more robust.</li>
</ol>