How can the model updates weights under conditions that loss=nan in some batches?

Chu-i_Yang: <p>Hi,<br>
I have a custom loss function which contains log(y) in it.<br>
y is one of the model outputs and I don’t want to restrict y to be positive through any activation function.</p>
<p>So, it is possible that loss returns ‘nan’ because of log(y) for y &lt; 0.</p>
<p>Somehow, my model returns ‘nan’ in some batches but it can keep on training until it converges.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60bddb905166bb278a8fc883b19191a33d717288.png" alt="截圖 2021-03-18 下午5.45.45" data-base62-sha1="dNOBwX7TOLiYO5KiHOcFNuWauYU" width="534" height="415"></p>
<p>I wonder how the model updates its weights when the loss returns ‘nan’…?</p>
<p>Thank you</p>

ptrblck: <p>I don’t know how the model is updated exactly, but you could check all <code>.grad</code> attributes after the <code>loss.backward()</code> call and see, if there are <code>nan</code> values.<br>
Generally, a <code>nan</code> loss could break your model as seen here:</p>
<pre><code class="lang-python">torch.manual_seed(2809)

x = torch.randn(10, requires_grad=True)
optimizer = torch.optim.SGD([x], lr=1.)

y = torch.log(x)
y = y * y
loss = y.mean()
print(loss)
&gt; tensor(nan, grad_fn=&lt;MeanBackward0&gt;)

loss.backward()
print(x.grad)
&gt; tensor([    nan, -0.0501,     nan, -0.0420,     nan,     nan,     nan,     nan,
        -0.2245,     nan])

optimizer.step()
print(x)
&gt; tensor([   nan, 0.8653,    nan, 0.8805,    nan,    nan,    nan,    nan, 0.7679,
           nan], requires_grad=True)
</code></pre>
<p>Are you sure the <code>optimizer.step()</code> is indeed performed or does your code have specific guards to avoid it?</p>

Chu-i_Yang: <p>Thank you so much for your reply!<br>
Your sample code gave me inspiration for the following toy model</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(2810)
# input
x = torch.randn(5, requires_grad=True)

# model
class simple(torch.nn.Module):
    def __init__(self):
        super(simple, self).__init__()
        self.layer = torch.nn.Linear(5, 1, bias=False)
    def forward(self, in_):
        return self.layer(in_)
model = simple()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
model.zero_grad()

y = torch.log(model(x))
loss = y

print('input x: ', x)
&gt; input x: tensor([-0.6195,  1.2090,  0.7255,  0.3041, -1.8422], requires_grad=True)
print('layer weights: ', model.layer.weight.data)
&gt; layer weights: tensor([[ 0.0566,  0.3471,  0.0256, -0.3857,  0.2482]])
print('loss: ', loss)
&gt; loss: tensor([nan], grad_fn=&lt;LogBackward&gt;)
</code></pre>
<p>then I calculated gradients and backpropagated</p>
<pre><code class="lang-auto">loss.backward()
print('layer weights: ', model.layer.weight.data)
&gt; layer weights: tensor([[ 0.0566,  0.3471,  0.0256, -0.3857,  0.2482]])
print('layer weights grad: ', model.layer.weight.grad)
&gt; layer weights grad: tensor([[ 3.6150, -7.0544, -4.2334, -1.7744, 10.7490]])

optimizer.step()
print('layer weights: ', model.layer.weight.data)
&gt; layer weights: tensor([[ 0.0530,  0.3542,  0.0298, -0.3840,  0.2374]])
</code></pre>
<p>The model could obtain gradients of the layer weights and update the weights even if the loss is ‘nan’!<br>
I think the optimizer has another strategy to update the model’s weights when receiving a ‘nan’ loss,<br>
but I’m still looking for the answer…</p>
<p>Thank you so much again for your kind reply, you really saved my time!</p>