Gradient value is nan

saumya0303: <p>Hi team,<br>
Please follow the below code,</p>
<p><strong>x.requires_grad = True</strong><br>
<strong>loss.backward()</strong><br>
<strong>print(x.grad)</strong></p>
<p>output:-<br>
tensor([ 1.0545e-05,  9.5438e-06, -8.3444e-06,  …,         nan,<br>
nan,         nan])<br>
how to resolve this nan problem as i am unable to find the range of x.grad.<br>
Please help me to resolve this issue.</p>

TinfoilHat0: <p>Perhaps this is due to exploding gradients? I’d recommend you to first try gradient clipping and see how the training goes.</p>

saumya0303: <p>Thanks for the answer. Actually I am trying to perform an adversarial attack where I don’t have to perform any training. The strange thing happening is when I calculate my gradients over an original  input I get  <strong>tensor([0., 0., 0.,  …, nan, nan, nan])</strong>  as result but if I made very small changes to my input the gradients turn out to perfect in the range of  <strong>tensor(0.0580) and tensor(-0.0501).</strong>.<br>
Please help me to figure out this issue?</p>

ptrblck: <p>You could add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your script to get an error with a stack trace, which should point to the operation, which created the NaNs and which should help debugging the issue.</p>

szahan: <p>Hey, I am also having nan issues with the gradient. I tried gradient clipping, converted my relu functions to LeakyReLU. But no progress. Any suggestions would be great. Thanks.</p>

ptrblck: <p>Assuming that the forward pass does not create invalid outputs, you could register hooks to the parameters of the model and print the gradients during the backward pass in order to isolate which gradient gets the first invalid value.<br>
This could make it easier to debug the issue further and check the operations used to create this gradient (e.g. are you dividing by a small number?).</p>

Alibay_Alili: <p>Hey, any idea why does gradient become nan? I mean which mathematical operation cause it?</p>

ptrblck: <p>Invalid outputs can create NaN gradients:</p>
<pre><code class="lang-python">x = torch.randn(1, requires_grad=True)
y = x / 0.
y = y / y
y.backward()
print(x.grad)
# tensor([nan])
</code></pre>

Alibay_Alili: <p>Yes, that is true. But my case is different. In my case, y is a valid output. But when I call <code>y.backward()</code>, one of the components of gradient is NaN, it traces back to the input during BP and most parameters become NaN.<br>
I think that, since one of the components of gradient is NaN, so partial deriviative is NaN. So relative change produce NaN. Any idea how it can produce NaN? Maybe my conclusion is wrong <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_hand_over_mouth.png?v=12" title=":face_with_hand_over_mouth:" class="emoji" alt=":face_with_hand_over_mouth:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Use <code>torch.autograd.detect_anomaly</code> to check which layer is creating the invalid gradients, then check its operations and inputs.</p>

ptrblck: <aside class="quote no-group" data-username="Alibay_Alili" data-post="9" data-topic="91663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alibay_alili/48/18355_2.png" class="avatar"> Alibay_Alili:</div>
<blockquote>
<p>But when I call <code>y.backward()</code>, one of the components of gradient is NaN, it traces back to the input during BP and most parameters become NaN.</p>
</blockquote>
</aside>
<p>I’m rereading this part and am unsure how to understand it. Are you already seeing invalid values in <code>y</code> before calling <code>backward</code>? Or do you see the first invalid gradient somewhere later in the model?</p>
<p>Also, once you’ve narrowed down the layer or parameter where the first NaN is created, check if something could overflow (and then create NaNs somehow).</p>

Alibay_Alili: <p>Sorry, I just focused on NaN using <code>torch.isnan(tensor).any()</code>. Actually, I must also take into account <code>torch.isinf(tensor).any()</code>. Before loss is NaN, there is actually <code>float('infinity')</code> :</p>
<pre><code class="lang-auto">for images, targets in dataloader['train']:
        images, targets= images.to(device),  targets.to(device)
        outputs = model(images) # some elements is infinity
        loss = cross_entropy(outputs, targets) # loss is NaN
       .........
</code></pre>
<p>Simple test:</p>
<pre><code class="lang-auto">input_ = torch.tensor([[1, float('infinity'), 6],
                     [3, 5, 2],
                      [10, 12, 4]])
criterion = nn.CrossEntropyLoss()
loss = criterion(input_, torch.tensor([0, 2, 1])) # loss is NaN
</code></pre>

ptrblck: <p>If you training diverges and the output overflows, the Inf can easily become a NaN value, so you can also check the tensors to be valid via <code>torch.isfinite</code>.</p>

hellohawaii: <p>FYI, <code>torch.autograd.set_detect_anomaly(True)</code> do not work with pytorch_geometric (at least I think so).</p>
<p>I am using pytorch_geometric. I got a message that <code>gcnnorm</code> raises an exception.</p>
<pre><code class="lang-auto">def gcn_norm(edge_index, edge_weight=None, num_nodes=None, improved=False,
             add_self_loops=True, flow="source_to_target", dtype=None):

    fill_value = 2. if improved else 1.

    if isinstance(edge_index, SparseTensor):
        assert edge_index.size(0) == edge_index.size(1)

        adj_t = edge_index

        if not adj_t.has_value():
            adj_t = adj_t.fill_value(1., dtype=dtype)
        if add_self_loops:
            adj_t = torch_sparse.fill_diag(adj_t, fill_value)

        deg = torch_sparse.sum(adj_t, dim=1)
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0.)
        adj_t = torch_sparse.mul(adj_t, deg_inv_sqrt.view(-1, 1))
        adj_t = torch_sparse.mul(adj_t, deg_inv_sqrt.view(1, -1))

        return adj_t

    if is_torch_sparse_tensor(edge_index):
        assert edge_index.size(0) == edge_index.size(1)

        if edge_index.layout == torch.sparse_csc:
            raise NotImplementedError("Sparse CSC matrices are not yet "
                                      "supported in 'gcn_norm'")

        adj_t = edge_index
        if add_self_loops:
            adj_t, _ = add_self_loops_fn(adj_t, None, fill_value, num_nodes)

        edge_index, value = to_edge_index(adj_t)
        col, row = edge_index[0], edge_index[1]

        deg = scatter(value, col, 0, dim_size=num_nodes, reduce='sum')
        deg_inv_sqrt = deg.pow_(-0.5)
        deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0)
        value = deg_inv_sqrt[row] * value * deg_inv_sqrt[col]

        return set_sparse_value(adj_t, value), None

    assert flow in ['source_to_target', 'target_to_source']
    num_nodes = maybe_num_nodes(edge_index, num_nodes)

    if add_self_loops:
        edge_index, edge_weight = add_remaining_self_loops(
            edge_index, edge_weight, fill_value, num_nodes)

    if edge_weight is None:
        edge_weight = torch.ones((edge_index.size(1), ), dtype=dtype,
                                 device=edge_index.device)

    row, col = edge_index[0], edge_index[1]
    idx = col if flow == 'source_to_target' else row
    deg = scatter(edge_weight, idx, dim=0, dim_size=num_nodes, reduce='sum')
    deg_inv_sqrt = deg.pow_(-0.5)
    deg_inv_sqrt.masked_fill_(deg_inv_sqrt == float('inf'), 0)
    edge_weight = deg_inv_sqrt[row] * edge_weight * deg_inv_sqrt[col]

    return edge_index, edge_weight
</code></pre>
<p>I was told that the last 4 line <code> deg_inv_sqrt = deg.pow_(-0.5)</code> cause the nan. However, in gcnnorm implementation, the nan is masked out an do not influence the training. But pytorch still raise an exception.</p>

ptrblck: <aside class="quote no-group" data-username="hellohawaii" data-post="14" data-topic="91663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hellohawaii/48/42794_2.png" class="avatar"> hellohawaii:</div>
<blockquote>
<p>However, in gcnnorm implementation, the nan is masked out an do not influence the training.</p>
</blockquote>
</aside>
<p>I don’t see where this would happen as you are masking <code>Inf</code> values, not <code>NaN</code>s. Even if you would mask the <code>NaN</code>, the operation creating the <code>NaN</code> is still used, anomaly detection raises a proper error as expected, and the backward pass will contain invalid gradients as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(-1., requires_grad=True)
deg = x * 2

deg_inv_sqrt = deg.pow_(-0.5)
deg_inv_sqrt = torch.nan_to_num(deg_inv_sqrt, nan=0.)

deg_inv_sqrt.mean().backward()
print(x.grad)
# tensor(nan)
</code></pre>