Time-series prediction day-by-day with Conv1D

JDE65: <p>Hello<br>
I developed a standard Conv1D model in Pytorch to predict time series with classification (4 classes).<br>
I gathered a train set (5000 data) and a test set (1000 data). The model predicts daily data by batches and is quite efficient.</p>
<p>As the results were satisfactory, I then moved to the next step :</p>
<ol>
<li>I trained my model</li>
<li>I saved the model</li>
<li>I used the trained model on daily new data (bringing data for prediction day-by-day, instead of by batch size).<br>
The results were very deceptive (if not catastrophic).</li>
</ol>
<p>Therefore, I checked what happen and tried to save my trained model, to clean the GPU cache, to re-load the model and to apply it to the test-set.</p>
<p>Herewith the original code :</p>
<pre><code>    train_dl, test_dl = get_data_loaderRN(X_train, y_train, X_tests, y_tests, batch_size) 
    model = Conv_1D(input_shape, nb_classes, num_features, seq_length, batch_size, iter_model, iter_pre, dropout)
    model = model.double()
    model.cuda()
    criterion = nn.CrossEntropyLoss()  
    optimizer = torch.optim.Adam(model.parameters(), lr= learning)        
    y_predtrain, running_loss = train_Conv_1D(model, criterion, optimizer, epochs, learning, verbose, train_dl)
    acc_train = accuracy_score(y_train, y_predtrain)
    y_predsubt, y_pred_proba, running_loss = eval_model_a(model, nb_classes, criterion, test_dl)
    acc_test = accuracy_score(y_tests, y_predsubt)
</code></pre>
<p>And the code with save and re-load model :</p>
<pre><code>    train_dl, test_dl = get_data_loaderRN(X_train, y_train, X_tests, y_tests, batch_size) 
    model = Conv_1D(input_shape, nb_classes, num_features, seq_length, batch_size, iter_model, iter_pre, dropout)
    model = model.double()
    model.cuda()
    criterion = nn.CrossEntropyLoss()  
    optimizer = torch.optim.Adam(model.parameters(), lr= learning)        
    y_predtrain, running_loss = train_Conv_1D(model, criterion, optimizer, epochs, learning, verbose, train_dl)
    acc_train = accuracy_score(y_train, y_predtrain)
    ## Save model and clean GPU cache + re-seed
    torch.save(model, model_path)
    torch.cuda.empty_cache()
    torch.manual_seed(my_seed)
    torch.cuda.manual_seed(my_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    ## Reload model and use it to predict
    model = torch.load(model_path)
    model.to(device)
    y_predsubt, y_pred_proba, running_loss = eval_model_a(model, nb_classes, criterion, test_dl)
    acc_test = accuracy_score(y_tests, y_predsubt)
</code></pre>
<p>The Save / Clean / Load steps are there to simulate what happened with the model used for daily prediction… and the results are far from those with the original model.</p>
<p>How could I address this problem ?<br>
Thanks in advance for your support.<br>
Best,</p>
<p>NB1. The output is double “y_predsubt” recors the predicted class and “y_pred_proba” records the likelihood associated by the model to each of the 4 classes.</p>
<p>NB2. I did the same with a LSTM model and get the same issue <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>If I understand the issue correctly, you are seeing a performance drop after loading a saved model and evaluating it on the same dataset?<br>
If so, are you calling <code>model.eval()</code> while the evaluation is performed? If that’s the case, could you pass a static input to the model before saving and after loading (e.g. <code>torch.ones</code>) and compare the outputs?</p>

JDE65: <p>Hello ptrblck,<br>
Thanks so much.<br>
That is exactly my problem. The model does not predict the same results if :<br>
A. I divide my train and test sets and train + model.eval() it by batches<br>
B. I divide my train and test sets and train by batches and create a loop to test it with model.eval() day-by-day<br>
C. I divide my train and test sets and train by batches, save it, re-load it and test it with model.eval()</p>
<p>BUT… I didn’t use model.eval() whose utility I discovered thanks to you, hence the instability.</p>
<p>I also faced another source of instability, even with model.eval(), and that comes from the dropout in LSTM. Removing it made the results stable, but less appealing.</p>
<p>The next task is to get back the quality of results I had before introducing model.eval() while keeping the reproducibility even in daily predictions.<br>
THX again !</p>