Validation loss not changing when adding dropout

Carlos_Giudice: <p>Hello!!</p>
<p>As the title says, when I add dropout to a model, the validation loss seems to stop changing (worth noting that the model finely overfits and the validation loss changes when no dropout is used). I’m using pytorch lightning, which is supposed to turn off dropout when running the validation_step.</p>
<p>I’m not sure about how to debug this or what to try next. I’ll leave the code I’m using and the results in hopes someone helps me find the problem.</p>
<h3><a name="model-definition-1" class="anchor" href="#model-definition-1"></a>Model definition</h3>
<pre><code class="lang-auto">    def __init__(self, input_shape, dropout_p=0, *args, **kwargs):
        
        ...

        conv3 = torch.nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3))

        if not dropout_p:
            regularization = ('batch_norm', torch.nn.BatchNorm2d(128))
        else:
            regularization = ('dropout', torch.nn.Dropout2d(dropout_p))

        layers = OrderedDict([
            ('conv', conv), ('relu', torch.nn.ReLU()), 
                ('pool',torch.nn.MaxPool2d((2, 2))), regularization,
            ('flatten', torch.nn.Flatten(start_dim=1)),
            ('output', CatDogOutput(460800))
        ])
        self.model = torch.nn.Sequential(layers)
</code></pre>
<h3><a name="validation-step-2" class="anchor" href="#validation-step-2"></a>Validation step</h3>
<pre><code class="lang-auto">    def validation_step(self, batch, batch_idx):
        target, pred_target, bbox, pred_bbox, classification_loss, bbox_loss, loss = self._shared_step(batch)
        imgs = batch[0]
        sel_idx = np.random.randint(len(imgs))
        return {"img": imgs[sel_idx], "sel_idx": sel_idx, "target": target,
                "pred_target": pred_target, "pred_bbox": pred_bbox,
                "classification_loss": classification_loss, "bbox_loss": bbox_loss, "loss": loss}

    def _shared_step(self, batch):
        img, target, bbox = batch
        model_input = self.preprocess_img(img)
        pred_target, pred_bbox = self.forward_pass(model_input)
        # adds one dimension to target, just the way torch likes it
        target = target.unsqueeze(1)

        classification_loss = F.binary_cross_entropy(pred_target, target)
        bbox_loss = F.mse_loss(pred_bbox, bbox)
        loss = classification_loss + self.hparams.bbox_alpha * bbox_loss

        return target.int(), pred_target, bbox, pred_bbox, classification_loss, bbox_loss, loss

    def forward_pass(self, img):
        output = self.model(img)
        # import ipdb; ipdb.set_trace()
        return output

</code></pre>
<h3><a name="loss-behaviour-3" class="anchor" href="#loss-behaviour-3"></a>Loss behaviour</h3>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbb6e65f216ba89161bcc53ad7d342e7107057a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/bbb6e65f216ba89161bcc53ad7d342e7107057a9" title="CNN_classifier_3_layers_with_max_pooling_dropout_0.5_batch_size_82_lr_2e-4_losses"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbb6e65f216ba89161bcc53ad7d342e7107057a9_2_617x500.png" alt="CNN_classifier_3_layers_with_max_pooling_dropout_0.5_batch_size_82_lr_2e-4_losses" data-base62-sha1="qMB9bBdOnTDpgu9x9DRc5Qij6OJ" width="617" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbb6e65f216ba89161bcc53ad7d342e7107057a9_2_617x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbb6e65f216ba89161bcc53ad7d342e7107057a9_2_925x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbb6e65f216ba89161bcc53ad7d342e7107057a9.png 2x" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CNN_classifier_3_layers_with_max_pooling_dropout_0.5_batch_size_82_lr_2e-4_losses</span><span class="informations">1046×847 76.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>