Getting the validation loss while training

username: smu226
cooked: <p>Hello! I want to get the validation loss after each epoch while training. I have this piece of code:</p>
<pre><code class="lang-auto">def train(model, train_loader, optimizer):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        scheduler.step()            
        data, target = data.cuda(), target.cuda()
        optimizer.zero_grad()
        output = model(data)

        loss = torch.sqrt(F.mse_loss(output, target))
        loss.backward()
        optimizer.step()

        loss_validation = torch.sqrt(F.mse_loss(model(factors_val), product_val))

        if batch_idx%100 == 0:
            print(loss)
    return(loss, loss_validation)
</code></pre>
<p>I have a batch size of 512 and 300 epoch. The test size has 250000 inputs and the validation set has 20000. The NN is a simple feed forward fully connected with 8 hidden layers. If I don’t use <code>loss_validation = torch.sqrt(F.mse_loss(model(factors_val), product_val))</code> the code works fine. However, if I use that line, I am getting a <code>CUDA out of memory</code> message after epoch 44. The way I go through the epochs is this:</p>
<pre><code class="lang-auto">for epoch in range(1, epochs ):
    losses = losses + [train(model, train_loader, optimizer)]
</code></pre>
<p>All I want to do is to save the training and validation set after each epoch so I can plot it. Can someone tell me what am I doing wrong and how can I fix it? I am really new to pytorch so it is probably something stupid and obvious, but I really need some help. Thank you!</p>

username: JuanFMontesinos
cooked: <p>Hi there,<br>
PyTorch dynamically generates the computational graph which represents the neural network. In short, PyTorch does not know that your validation set is a validation set.</p>
<p>In order not to compute the backward over the validation set you need to use</p>
<pre><code class="lang-auto">with torch.no_grad():
      validation_operations ...
</code></pre>
<p>It means torch shouldn’t keep tracking of gradients for those operations.</p>
<p>You also may want to use model.eval() which turns your model into evaluation mode, deactivating batch normalization layers nor dropout.</p>
<p>remember to set model.train() before computing training operations.</p>
<p>Edit,<br>
When you attach loss to the list, you are also picking the history of the loss, not only the numerical data. That’s why you get out of memory after certain amonut of iterations.</p>
<p>Use  <code>loss += loss.item()</code> to catch just the number.<br>
Regards</p>

username: Vaibhav_Sundharam
cooked: <p>Hi,<br>
I just wanted to know if it is mandatory to use model.eval() if we are calculating validation loss after each epoch?</p>

username: a_d
cooked: <p>Yes, when calling .eval(), your dropout layers(if you have any) are deactivated in a manner of speaking, as it should be the case in the testing phase, so yes the .eval() must be called…</p>

username: MaKaNu
cooked: <p>is  model.eval() setting the modell in an equal state comparable with “with torch.no_grad()”?</p>

username: ptrblck
cooked: <p>No, <code>model.eval()</code> and <code>torch.no_grad()</code> will not have the same effect.<br>
While <code>model.eval()</code> changes the behavior of some layers (e.g. dropout will be disabled and the running stats of batchnorm layers will be used), <code>with torch.no_grad()</code> disables the gradient calculation.</p>

username: Aioku_Takume
cooked: <p>Hi I am using torch.nn.LSTM for time series.  I wonder if I dont have dropout and batch norm , is it necessary to use model.train() and model.eval() ?</p>

username: shartzog
cooked: <p><a class="mention" href="/u/aioku_takume">@Aioku_Takume</a><br>
Not sure it’s <em>mandatory</em>, but I’m pretty sure it’s a best practice. In fact, I wonder if it might reduce computational overhead if you do call vs not? Time it and see? I’m interested to see how that would turn out. If it saves time, I can’t think of any reason why you wouldn’t call it…</p>
<p>Any experts want to weigh in?? I’m new to this stuff myself. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: yuri
cooked: <p>Hi,<br>
isn’t eval also meant not to run backpropagation and only run the feed forward algortihm of the model?<br>
in other words, torch.no_grad() will also be launched once you call model.eval()<br>
thx</p>

username: ptrblck
cooked: <p>This would be the common use case, yes.<br>
However, <code>model.eval()</code> changes the behavior of some modules during training and validation, while <code>torch.no_grad()</code> disables the gradient calculation, and some use cases treat these two options independently.<br>
E.g. you might want to leave dropout layers enabled during validation to and create multiple (noisy) predictions for the same input samples etc.</p>

username: Wenuka
cooked: <p>Hi,<br>
Sorry if this is a lame question.<br>
But what happens to the forward pass value when we do a validation forward pass?<br>
To elaborate my question, because we need the forward pass value when updating the weights, and we only make gradients zero (before the optimizer.step()), can validation forward pass value mess up with the weight update? or the model parameters only remember the last forward pass value so the validation forward pass is forgotten when we do the update (because we do the train forward pass later)?</p>

username: ptrblck
cooked: <p>If you are not wrapping the validation forward pass into <code>with torch.no_grad()</code>, the computation graph will be created, which would be needed to calculate the gradients. Unless you are calling <code>backward()</code> on the output or loss, no gradients will be calculated and the model will thus not be changed. As long as you hold a valid reference to the model output the computation graph will be kept alive.</p>

username: Navya
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85b396552e24338d83b39a10b0982200f131ac68.png" alt="loss plot" data-base62-sha1="j4MgA543giC1qPDdOVqp6t2jTQc" width="640" height="460"></p>
<p>why is validation loss so low in the first epoch and it doesn’t change after that?<br>
Please help me with this.</p>
<p>Thank you</p>