Adding Validation Loss to the model

Aneeq_Bokhari: <p>Hi everyone.<br>
For my project , i’m  trying to predict the ratings that a user will give to an unseen movie, based on the ratings he gave to other movies. I’m using the movielens dataset.The Main folder, which is ml-100k contains informations about 100 000 movies.To create the recommendation systems, the model ‘<strong>Autoencoder</strong>’ is being used. I’m using <strong>Google Colab</strong> for coding implementation.</p>
<p>So far I managed to get the training loss values and plot them on the graph but I’m struggling to add the validation loss  for this project.</p>
<p>Here’s my code.</p>
<blockquote>
<p>r_cols = [‘user_id’, ‘movie_id’, ‘rating’, ‘timestamp’] <span class="hashtag-raw">#Create</span> each column</p>
<p>user_ratings = pd.read_csv(‘u.data’, sep=‘\t’, names=r_cols, header = None, engine = ‘python’, encoding = ‘latin-1’) <span class="hashtag-raw">#Read</span> the file</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Preparing</span> Training Set and Test Set<br>
train_data, test_data = train_test_split(user_ratings , test_size=0.20, random_state=42)</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Converting</span> the training and test sets into numpy arrays<br>
train_data = np.array(train_data, dtype = ‘int’)<br>
test_data = np.array(test_data, dtype = ‘int’)</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#We</span>’ll need the number of users and the number of movies to build our recommendation system. Since id_users and id_movies start at index 1, the number of users represent the maximum value of id_user. Similarly, the number of movies represent the maximum value of id_movie. However, since the data is divided into training and test set, the maximum value of id_user/id_movie is either in the training_set or in the test_set.</p>
<p><span class="hashtag-raw">#Getting</span> the number of users and movies<br>
nb_users = int(max(max(train_data[:, 0]), max(train_data[:, 0])))<br>
nb_movies = int(max(max(test_data[:, 1]), max(test_data[:, 1])))</p>
<p>print(“Number of users: {}”.format(nb_users))<br>
print(“Number of movies: {}”.format(nb_movies))</p>
<p><span class="hashtag-raw">#Number</span> of users: 943<br>
<span class="hashtag-raw">#Number</span> of movies: 1682</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#In</span> order to build the autoencoder, we need a specific data structure. In our case, we will create a list of lists, expected by Pytorch. Each list of list will contain the ratings that a specific user gave to the movies. If a user didn’t rate a movie, we’ll just add a 0 for that observation. We will define a function which will create this list of list for us.</p>
<p>def convert(data):<br>
# Initializing an empty list that will take the list of ratings given by a specific user<br>
new_data = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
# Looping over all the users<br>
for id_users in range(1, nb_users + 1):<br>
# We get the id of the movies rated by the current user<br>
id_movies = data[:, 1][data[:, 0] == id_users]<br>
# We get the id of the ratings given by the current_user<br>
id_ratings = data[:, 2][data[:, 0] == id_users]<br>
ratings = np.zeros(nb_movies)<br>
# For movies rated by the current user, we replace 0 with the rating<br>
# The first element of ratings is at index 0. However, id_movies start at index 1.<br>
# Therefore, ratings[id_movies - 1] will correspond to the location of the movie we’re considering<br>
ratings[id_movies - 1] = id_ratings<br>
new_data.append(list(ratings))<br>
return new_data</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Applying</span> the convert function to the training and test set.<br>
train_data = convert(train_data)<br>
test_data = convert(test_data)</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Convert</span> the data into Torch tensors<br>
train_data = torch.FloatTensor(train_data)<br>
test_data = torch.FloatTensor(test_data)</p>
</blockquote>
<blockquote>
<p>batch_size  = 200</p>
<p>‘’’ Dataset Class’‘’</p>
<p>class DatasetR(Dataset):</p>
<pre><code>def __init__(self, train_data, nb_users, transform=None):

    super(DatasetR, self).__init__()

    self.train_data = train_data

    self.nb_users = nb_users

def __len__(self):

    return self.nb_users

def __getitem__(self, idx):

    #print(idx)

    sample = self.train_data[idx]

    #sample = torch.Tensor(sample)

    return sample
</code></pre>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Creating</span> the architecture of the Neural Network We’ll create a stacked autoencoder. This stacked autoencoder will get one input layer, two encoding layers and two decoding layers. As a reminder, for an autoencoder, the number of nodes of the output layer should equal the number of nodes of the input layer.</p>
<p>class SAE(nn.Module):</p>
<pre><code>#Initializing the class

def __init__(self, ):

    # making the class get all the functions from the parent class nn.Module

    super(SAE, self).__init__()

    # Creating the first encoding layer. The number of input corresponds to the number of movies

    #  Decide to encode it into 20 outputs

    self.fc1 = nn.Linear(nb_movies, 20)

    # Creating the second encoding layer. From 20 inputs to 10 outputs

    self.fc2 = nn.Linear(20, 10)

    # Creating the first decoding layer. From 10 inputs to 20 outputs

    self.fc3 = nn.Linear(10, 20)

    # Creating the second hidden layer. From 20 inputs to nb_movies outputs

    self.fc4 = nn.Linear(20, nb_movies)

    # Creating the activation fucntion which will fire up specific neurons 

    self.activation = nn.Sigmoid()

    

    # Creating the function for forward propagation

def forward(self, x):

    # x = self.do1(self.bn1(self.activation(self.fc1(x))))

    # x = self.do2(self.bn2(self.activation(self.fc2(x))))

    # x = self.do3(self.bn3(self.activation(self.fc3(x))))

    x = self.activation(self.fc1(x))

    x = self.activation(self.fc2(x))

    x = self.activation(self.fc3(x))

    # With autoencoder, we don't need an activation function for the last decoding part

    x = self.fc4(x)

    return x
</code></pre>
<p><span class="hashtag-raw">#Creating</span> an instance of our SAE class</p>
<p>sae = SAE()</p>
<p>dataset = DatasetR(train_data = train_data, nb_users = nb_users)</p>
<p>train_loader = torch.utils.data.DataLoader(dataset,  batch_size = batch_size, shuffle=True, num_workers=4, drop_last=True)</p>
<p>datasetTest = DatasetR(train_data = test_data, nb_users = nb_users)</p>
<p>test_loader = torch.utils.data.DataLoader(datasetTest,  batch_size = batch_size, shuffle=True, num_workers=4, drop_last=True)</p>
<p>criterion = nn.MSELoss()</p>
<p><span class="hashtag-raw">#Defining</span> the algorithm used to minimize the loss function. In this case, we’ll use RMSprop</p>
<p>optimizer = optim.RMSprop(sae.parameters(), lr = 0.01, weight_decay = 0.5)</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Setting</span> the number of epochs</p>
<p>losses = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>nb_epoch = 200</p>
<p><span class="hashtag-raw">#Iterating</span> over each epoch</p>
<p>for epoch in range(1, nb_epoch + 1):</p>
<pre><code>#sae.train()

# Initializing the train_loss which will be updated

train_loss = 0

# Initializing a counter

s = 0.

#Iterating over each user

#for id_user in range(nb_users):

for batch_idx, (sample) in enumerate(train_loader):

    # The input corresponds to the ratings given by the current user for each movie

    input = Variable(sample)

    target = input.clone()

    # We don't consider movies NOT rated by the current user. So we specify a conditional statement

    if torch.sum(target.data &gt; 0) &gt; 0:

        # We use our SAE to get the output from the 

        #print('input:  '+ str(input.shape))

        output = sae(input)

        #print(output.shape)

        target.require_grad = False

        output[target == 0] = 0

        # Defining our loss function, comparing the output with the target

        loss = criterion(output, target)

        mean_corrector = nb_movies/float(torch.sum(target.data &gt; 0) + 1e-10)

        # Computing the gradients necessary to adjust the weights

        loss.backward()

        # Updating the train_loss

        train_loss += np.sqrt(loss.data*mean_corrector)

        s += 1.

        # Updating the weights of the neural network

        optimizer.step()

epoch_loss = train_loss / len(train_loader)

losses.append(epoch_loss)      

print('epoch: '+str(epoch)+' loss: '+str(train_loss/s))
</code></pre>
</blockquote>
<blockquote>
<p>plt.plot(losses)</p>
<p>plt.xlabel("Number of Epoch ")</p>
<p>plt.ylabel(" Training Loss")</p>
<p>plt.title(“Autoencoder : Loss vs Number of Epoch”)</p>
<p>plt.show()</p>
<p>After a training of 200 epochs, we get an overall lost of 0.0837. This means that for the training_set, we have : <strong>predicted_rating - 0.0837&lt;= real_rating &lt;= predicted_rating + 0.0837</strong></p>
</blockquote>
<p>Now I’m stuck on implementing the validation loss part.<br>
Can anyone help me with this?<br>
Thanks</p>

ptrblck: <p>You could reuse the training loop and just remove the <code>backward</code> and <code>optimizer.step()</code> calls.<br>
Also, you could wrap the validation/test loop into a <code>with torch.no_grad()</code> block to save some memory.</p>
<p>Also note that <code>Variables</code> are deprecated since PyTorch <code>0.4</code> and you can use tensors now.</p>

Aneeq_Bokhari: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for the reply.<br>
I modified the code  below</p>
<pre><code class="lang-auto"># Setting the number of epochs

losses = []

tlosses = []

nb_epoch = 200

# Iterating over each epoch

for epoch in range(1, nb_epoch + 1):

    #sae.train()

    # Initializing the train_loss which will be updated

    train_loss = 0

    test_loss = 0

    # Initializing a counter

    s = 0.

    # Iterating over each user

    #for id_user in range(nb_users):

    for batch_idx, (sample) in enumerate(train_loader):

        # The input corresponds to the ratings given by the current user for each movie

        input = Variable(sample)

        target = input.clone()

        # We don't consider movies NOT rated by the current user. So we specify a conditional statement

        if torch.sum(target.data &gt; 0) &gt; 0:

            # We use our SAE to get the output from the 

            #print('input:  '+ str(input.shape))

            output = sae(input)

            #print(output.shape)

            target.require_grad = False

            output[target == 0] = 0

            # Defining our loss function, comparing the output with the target

            loss = criterion(output, target)

            mean_corrector = nb_movies/float(torch.sum(target.data &gt; 0) + 1e-10)

            # Computing the gradients necessary to adjust the weights

            loss.backward()

            # Updating the train_loss

            train_loss += np.sqrt(loss.data*mean_corrector)

            s += 1.

            # Updating the weights of the neural network

            optimizer.step()

            

    with torch.no_grad():

      for batch_idx, (sample) in enumerate(test_loader):

          # The input corresponds to the ratings given by the current user for each movie

          input = Variable(sample)

          target = input.clone()

          # We don't consider movies NOT rated by the current user. So we specify a conditional statement

          if torch.sum(target.data &gt; 0) &gt; 0:

              # We use our SAE to get the output from the 

              #print('input:  '+ str(input.shape))

              output = sae(input)

              #print(output.shape)

              target.require_grad = False

              output[target == 0] = 0

              # Defining our loss function, comparing the output with the target

              loss = criterion(output, target)

              mean_corrector = nb_movies/float(torch.sum(target.data &gt; 0) + 1e-10)

              # Computing the gradients necessary to adjust the weights

              # Updating the train_loss

              test_loss += np.sqrt(loss.data*mean_corrector)

              s += 1.

      

      epoch_loss = train_loss / len(train_loader)

      epoch_loss1 = test_loss / len(test_loader)

      losses.append(epoch_loss)

      tlosses.append(epoch_loss1)      

      print('epoch: '+str(epoch)+' Training loss: '+str(train_loss/s))

      print('epoch: '+str(epoch)+' Test loss: '+str(test_loss/s))



</code></pre>
<p>When I plot the training loss and Validation loss graphs .They look basically the same.</p>
<pre><code class="lang-auto">epoch: 1 Training loss: tensor(0.1306)
epoch: 1 Test loss: tensor(0.1298)
epoch: 2 Training loss: tensor(0.1296)
epoch: 2 Test loss: tensor(0.1285)
epoch: 3 Training loss: tensor(0.1289)
epoch: 3 Test loss: tensor(0.1275)
epoch: 4 Training loss: tensor(0.1277)
epoch: 4 Test loss: tensor(0.1263)
epoch: 5 Training loss: tensor(0.1265)
epoch: 5 Test loss: tensor(0.1252)
.........
.......
.......
epoch: 195 Training loss: tensor(0.0419)
epoch: 195 Test loss: tensor(0.0428)
epoch: 196 Training loss: tensor(0.0420)
epoch: 196 Test loss: tensor(0.0426)
epoch: 197 Training loss: tensor(0.0417)
epoch: 197 Test loss: tensor(0.0423)
epoch: 198 Training loss: tensor(0.0418)
epoch: 198 Test loss: tensor(0.0423)
epoch: 199 Training loss: tensor(0.0417)
epoch: 199 Test loss: tensor(0.0421)
epoch: 200 Training loss: tensor(0.0418)
epoch: 200 Test loss: tensor(0.0425)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04fd91065e80cdbce93f08ed65d61aab06a9aeaa.png" alt="image" data-base62-sha1="I9aLm6eHrGCggMIUHOgF96C9lw" width="398" height="278"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a07f26c9a0a2e2ca08e1645685f3e437814bd95.png" alt="image" data-base62-sha1="1qJNF4JfNLYt0ZOXq5MdI3hDm17" width="398" height="278"></p>
<p>I am new to Pytorch, can you(or someone) please have a look if I am doing it right?<br>
Is the model fine (Not overfitting or undefitting)?<br>
Thanks.</p>

ptrblck: <p>The loss curves look alright.</p>
<p>I’m not familiar with your code and cannot comment on the correctness of some operations such as <code>output[target == 0] = 0</code> etc., but it looks generally alright (besides the usage of <code>Variable</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>

Aneeq_Bokhari: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for your feedback.</p>

Aneeq_Bokhari: <p>Hi everyone</p>
<p>I have a a question. This is related to this post.</p>
<p>I split the dataset into training(80%) set and testing set(20%). My loss function is MSE.  When I plot Training Loss curve and Validation curve, the loss curves, look fine. Its shows minimal gap between them.(Look at the pictures above)<br>
But when I changed my loss function to RMSE and plotted the loss curves. There is a huge gap between training loss curve and validation loss curve.(epoch: 200 training loss: 0.0757. Test loss: 0.1079)</p>
<p>In my code, I only changed the loss function part(MSE to RMSE). I applied the Regularization techniques such as Batch Normalization and Dropout but still there is a big gap between the curves.<br>
I’m new to deep learning, but do you know whats the reason why there is a huge gap between the curves when applying RMSE?<br>
Is it something to do  with the Evaluation metric or something wrong in the coding part?<br>
Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cf7179f9f6a2a8293a8fae125dddc00e6c29ffb.png" data-download-href="https://discuss.pytorch.org/uploads/default/5cf7179f9f6a2a8293a8fae125dddc00e6c29ffb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cf7179f9f6a2a8293a8fae125dddc00e6c29ffb_2_517x138.png" alt="image" data-base62-sha1="dgpjQguAlvHutIwEliC1PNXPWob" width="517" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cf7179f9f6a2a8293a8fae125dddc00e6c29ffb_2_517x138.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cf7179f9f6a2a8293a8fae125dddc00e6c29ffb_2_775x207.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5cf7179f9f6a2a8293a8fae125dddc00e6c29ffb_2_1034x276.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1470×395 52.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>