Constant loss and accuracy

username: Meghna_Kapoor
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52e344316afed1fe0e9ca16fffbc5c8920489c86.png" data-download-href="https://discuss.pytorch.org/uploads/default/52e344316afed1fe0e9ca16fffbc5c8920489c86" title="Screenshot from 2022-08-19 15-18-02"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52e344316afed1fe0e9ca16fffbc5c8920489c86_2_690x317.png" alt="Screenshot from 2022-08-19 15-18-02" data-base62-sha1="bPg4fuZ2i9oTIM89QVCaxLglnXE" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52e344316afed1fe0e9ca16fffbc5c8920489c86_2_690x317.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52e344316afed1fe0e9ca16fffbc5c8920489c86_2_1035x475.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52e344316afed1fe0e9ca16fffbc5c8920489c86_2_1380x634.png 2x" data-dominant-color="340E28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-08-19 15-18-02</span><span class="informations">1917×881 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My loss and accuracy is constant. My forward function is:<br>
def forward(self, inp):<br>
# Preprocessing<br>
out = self.conv3d_1a_7x7(inp)<br>
skip1 = out<br>
out = self.maxPool3d_2a_3x3(out)<br>
out = self.dropout(out)<br>
out = self.conv3d_2b_1x1(out)<br>
out = self.conv3d_2c_3x3(out)<br>
out = self.maxPool3d_3a_3x3(out)<br>
out = self.dropout(out)<br>
out = self.mixed_3b(out)<br>
skip2 = out<br>
out = self.mixed_3c(out)<br>
out = self.maxPool3d_4a_3x3(out)<br>
out = self.dropout(out)<br>
out = self.mixed_4b(out)<br>
out = self.mixed_4c(out)<br>
out = self.dropout(out)<br>
out = self.mixed_4d(out)<br>
skip3 = out<br>
out = self.dropout(out)<br>
out = self.mixed_4e(out)<br>
out = self.mixed_4f(out)<br>
out = self.maxPool3d_5a_2x2(out)<br>
out = self.dropout(out)<br>
out = self.mixed_5b(out)<br>
out = self.mixed_5c(out)<br>
out = self.dropout(out)<br>
out = self.tconv6(out, skip1,skip2,skip3)<br>
out = self.sigmoid(out)<br>
print(“Before permutation”, out.shape)<br>
out = out.permute(0,1,3,4,2)<br>
out_logits = out<br>
return out, out_logits</p>
<p>My train function is:<br>
misc,out_logits[stream] = models<a>stream</a><br>
gt = gt.to(device,  dtype=torch.float)<br>
gt = gt.squeeze(1)<br>
gt = gt.squeeze(1)<br>
out_softmax = torch.nn.functional.softmax(out_logits[stream], 1).requires_grad_()<br>
val, preds = torch.max(out_logits[stream].data, 1)<br>
preds = preds.to(device, dtype=torch.float)<br>
gt = torch.round(gt)<br>
gt_avg = torch.mean(gt)<br>
gt[gt&gt;gt_avg] = 1<br>
gt[gt&lt;=gt_avg] = 0<br>
val_avg = torch.mean(val)<br>
val[val&gt;val_avg] = 1<br>
val[val&lt;=val_avg] = 0<br>
out_logits[stream] = out_logits[stream].squeeze(1)<br>
losses[stream] = criterion(val.cpu(), gt.cpu()).requires_grad_()</p>
<h1><a name="backward-optimize-only-if-in-training-phase-1" class="anchor" href="#backward-optimize-only-if-in-training-phase-1"></a>backward + optimize only if in training phase</h1>
<p>if phase == ‘train’:<br>
optimizers[stream].zero_grad()<br>
losses[stream].backward(retain_graph=True)<br>
optimizers[stream].step()<br>
gt_c = gt.squeeze(1)<br>
running_losses[stream] += losses[stream].item() * data[stream].shape[0]<br>
print(“Current Loss is”, running_losses[stream])<br>
running_corrects[stream] += torch.sum(val.cpu() == gt_c.data.cpu()).item()<br>
correct_t = torch.sum(preds==gt_c).item()<br>
total_t = gt_c.shape[0]*gt_c.shape[1]*gt_c.shape[2]<em>gt_c.shape[3]<br>
acc_epc = 100</em>correct_t/total_t<br>
for scheduler in schedulers.values():<br>
scheduler.step()</p>
<p>The loss function is defined as:<br>
criterion = torch.nn.BCEWithLogitsLoss()</p>
<p>I am trying for binary images by converting ground truth and output of model. But model is not converging. I have tried with different optimizers but still no improvement. Currently its SGD<br>
optimizers[stream] = optim.SGD(filter(lambda p: p.requires_grad, models[stream].parameters()), lr=1e-8,<br>
momentum=2, nesterov=True)</p>

username: ptrblck
cooked: <p>It seems you are detaching the computation graph at a few points and are trying to fix it by calling <code>requires_grad_()</code> on the already detached tensor, which won’t re-attach it:</p>
<pre><code class="lang-python">out_softmax = torch.nn.functional.softmax(out_logits[stream], 1).requires_grad_()
...
losses[stream] = criterion(val.cpu(), gt.cpu()).requires_grad_()
</code></pre>
<p>Make sure you are using differentiable operations (the output tensor should have a valid <code>.grad_fn</code>) and don’t explicitly detach tensors from the computation graph.</p>

username: Meghna_Kapoor
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="159559">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Make sure you are using differentiable operations (the output tensor should have a valid <code>.grad_fn</code>) and don’t explicitly detach tensors from the computation graph.</p>
</blockquote>
</aside>
<p>Hi, Thanks for the suggestion. I haven’t detached the tensor and my out_logits do have grad_fn. By using requires_grad I am just making sure the loss is updated. By using out_logits instead of preds does change a loss a bit but still far away from convergence.</p>

username: ptrblck
cooked: <p>In that case remove the <code>.requires_grad_()</code> calls as they are not needed and check if you would be running into any errors. Generally, don’t set the <code>requires_grad</code> attribute on any tensor which should already be attached to a computation graph as this would only mask errors.</p>