Improving the Performance of the model

Swamy: <p>Hi,<br>
I have been trying to train a LSTM on customer sequence model to predict the next event that is likely to happen.<br>
The event space consists of 2983 events.  To begin with, I have made a dataset that the most frequent 100 events form the labels for training and the last 10 events before the label as the attributes( or event sequence).<br>
For example the event sequence in a row might be like</p>
<pre><code class="lang-auto">* Attributes  :  [[2,3,5,7,42,432,4,3,1,5],
                       [243,343,52,7,423,432,42,31,112,532]]
* Labels  :  [[2],
                 [1]]
</code></pre>
<p>Encoding  : One-Hot encoding  (For  Attributes)<br>
Raw Values (For Labels)<br>
<span class="hashtag-raw">#Imports</span> and defaults :</p>
<pre><code class="lang-auto">import pandas as pd
import numpy as np
import torch 
import torch.nn as nn
import ast
from sklearn.model_selection import train_test_split 
from datetime import datetime
import torch.nn.functional as F
from torch.optim.lr_scheduler import ReduceLROnPlateau, CyclicLR
from torch.utils import data
import matplotlib.pyplot as plt
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score, classification_report

torch.multiprocessing.set_sharing_strategy('file_system')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
</code></pre>
<p><span class="hashtag-raw">#HyperParameters</span> :</p>
<pre><code class="lang-auto"> sequence_length = 10
input_size = 2983
hidden_size = 1024
num_layers = 2
num_classes = 100
batch_size = 50
num_epochs = 10
learning_rate = 1
</code></pre>
<p><span class="hashtag-raw">#Model</span> :</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers        
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional  = True)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):

        h0 = torch.randn(self.num_layers, x.size(0), self.hidden_size).to(device) 
        c0 = torch.randn(self.num_layers, x.size(0), self.hidden_size).to(device)
        out, _ = self.lstm(x, (h0, c0))
        out_ = self.fc(out[:, -1, :])
        return out_

model = RNN(input_size, hidden_size, num_layers, num_classes).cuda()
</code></pre>
<p><span class="hashtag-raw">#Loss</span>  and optimizer and Scheduler :</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum= 0.1)
scheduler = CyclicLR(optimizer, mode = 'exp_range',gamma  =0.9999, base_lr=  1e-5, max_lr = 1)
</code></pre>
<p><span class="hashtag-raw">#DataLoader</span> :</p>
<pre><code class="lang-auto">class Dataset_1(data.Dataset):
    def __init__(self, list_IDs, labels, n_uniq):
        self.labels = labels
        self.list_IDs = list_IDs
        self.n_uniq = n_uniq
    
    def __len__(self):
        return len(self.list_IDs)
    
    def __getitem__(self, index):
        X = self.list_IDs[index]
        X = F.one_hot(torch.from_numpy(X).to(torch.int64), num_classes = self.n_uniq)
        y = self.labels[index]
        return X, y
</code></pre>
<p><span class="hashtag-raw">#Train</span> and validation generators:</p>
<pre><code class="lang-auto">params = {'batch_size': 50,
          'shuffle': True,
          'num_workers': 30}

training_set = Dataset_1(x_train, y_train, input_size )
training_generator = data.DataLoader(training_set, **params)


validation_set = Dataset_1(x_test, y_test, input_size)
validation_generator = data.DataLoader(validation_set, **params)
</code></pre>
<p><span class="hashtag-raw">#Training</span>  :</p>
<pre><code class="lang-auto">for epoch in range(10):
    epoch_time = datetime.now()    
    # Training
    print('Training Start')
    counter = 0 
    for local_batch, local_labels in training_generator:
        # Transfer to GPU
        counter +=1
        #local_batch, local_labels = (copy.deepcopy(local_batch), copy.deepcopy(local_labels))
        local_batch, local_labels = local_batch.to(device).float(), local_labels.to(device).long()

        outputs = model(local_batch)
        loss= criterion(outputs,local_labels)
        #loss = criterion(local_labels,outputs, input_lengths, target_lengths)
        optimizer.zero_grad()
        
        loss.backward()
        optimizer.step()
        if counter%5000 == 0: 
            print(f"Counter : {counter} || Loss : {loss.item()}\n LR : {optimizer.state_dict()['param_groups'][0]['lr']}")

        if counter%50 == 0: 
            lst_loss.append(loss.item())
            lst_lr.append(optimizer.state_dict()['param_groups'][0]['lr'])
            scheduler.step()

            
            
    print(f"Epoch time : {datetime.now()-epoch_time} || Loss : {loss.item()}")
    
    with torch.no_grad():
        correct = 0
        total = 0
        pred = []
        orig =[]
        for local_batch, local_labels in validation_generator:

            outputs = model(local_batch.to(device).float())
            _, predicted = torch.max(outputs.data, 1)
            total += local_labels.squeeze().size(0)
            correct += (predicted == local_labels.squeeze().to(device).long()).sum().item()
            orig.append(local_labels.squeeze().cpu().numpy())
            pred.append(predicted.cpu().numpy())
            if total &gt;100000:
                print(f'Epoch : {epoch}')
                print('Test Accuracy of the model : {} %'.format(100 * correct / total))
                pred = np.array(pred).reshape((1,-1)).squeeze()
                orig =  np.array(orig).reshape((1,-1)).squeeze()
                s = classification_report(orig,pred)
                df_eval  = pd.DataFrame({'Labels' :s.split()[4:504:5],'precision' : s.split()[5:504:5] ,
                            'recall' :s.split()[6:504:5], 'f-1 score' : s.split()[7:504:5], 'support' : s.split()[8:504:5]})
                torch.save({'model_state_dict' : model.state_dict(),
                           'optimizer_state_dict' : optimizer.state_dict(),
                            'loss' : loss.item(), 'df_eval' : df_eval,
                            'accuracy' : accuracy_score(orig,pred),
                            'epoch' : epoch
                           }, f'./models/epoch_test/model_top_100_LR_0.01_REDUCELR_layers_2_epoch_{epoch}_accur_{int(accuracy_score(orig,pred)*100)}.pth')
                break
</code></pre>
<p>Below is the models Loss and Learning rate :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bfeac8746ad6a787e7e142167f1e9467efbb47a.png" data-download-href="https://discuss.pytorch.org/uploads/default/2bfeac8746ad6a787e7e142167f1e9467efbb47a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfeac8746ad6a787e7e142167f1e9467efbb47a_2_690x349.png" alt="image" data-base62-sha1="6hcaomk5GUB0Nvn0jo7DrEYKTRU" width="690" height="349" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfeac8746ad6a787e7e142167f1e9467efbb47a_2_690x349.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfeac8746ad6a787e7e142167f1e9467efbb47a_2_1035x523.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bfeac8746ad6a787e7e142167f1e9467efbb47a.png 2x" data-dominant-color="D9E7F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1170×592 51.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have tried many variations of the model :<br>
Num of layers from 1-20<br>
hidden_size 128,1024</p>
<p>The dataset I used have 200M rows fro training and since the dataset is huge, with the current infra (GPU - 8GB, Memory- 700GB) that we have here it takes 15Hrs to complete 1 epoch.<br>
The accuracy of the model in general fluctuates between 31 and 34.<br>
Questions :</p>
<ol>
<li>
<p>When the number of nodes in each layer is 128 the amount of GPU used was aroung 1GB and though increasing the batch size helps in more utilization the loss was even bigger with each iteration.<br>
What is the range of batch sizes that I needs to look for? Is there anything in specific that is making the model take so much of time or is the time seems normal for the dataset?</p>
</li>
<li>
<p>Is there any thing that I am doing that is weird and not intended to use it the way I am doing (updating LR after 1000 batches rather than for each epoch or LR range - 1e-5 to 1) ?</p>
</li>
<li>
<p>Why isn’t the loss value decreasing any further for any of the learning rates?</p>
</li>
<li>
<p>I would like to expand the model to all of the 2983 events and predict n events in the future. Is there any better algorithms and techniques that would help in achieving those?</p>
</li>
</ol>
<p>Thanks for the help and for the resources that are being provided to make easy for the beginners to use deeplearning.</p>
<p>Thanks,<br>
Swamy</p>