Handling None Labels in Dataset Leads to NaN Gradients and Predictions

username: lpolisi
cooked: <p>Hi everyone,</p>
<p>I’ve encountered an issue while training my model with a dataset that occasionally has samples with <code>None</code> labels. To handle these cases, I set the loss to 0 whenever the label is <code>None</code> by using <code>reduction="none"</code> on the loss function. Here’s a simplified version of my approach:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import optim, nn
from torch.utils.data import DataLoader

# Dummy data
x = torch.randn(100, 10)
y = torch.randn(100, 1)

# Set random values to NaN
nan_indexes = torch.randperm(100)[:10]
y[nan_indexes] = float("nan")

dl = DataLoader(list(zip(x, y)), batch_size=10)

# Model, loss function, and optimizer
model = nn.Sequential(nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 1))
criterion = nn.MSELoss(reduction="none")
optimizer = optim.Adam(model.parameters(), lr=0.001)

num_epochs = 20

for epoch in range(num_epochs):
    for x, y in dl:
        optimizer.zero_grad()
        outputs = model(x)
        loss = criterion(outputs, y)

        # Handle None labels
        nan_indexes = torch.isnan(y).nonzero()
        loss[nan_indexes] = 0
        loss = loss.sum() / (loss.shape[0] - nan_indexes.shape[0])  # Calculate the correct average loss

        loss.backward()
        optimizer.step()
        print(loss.item())
</code></pre>
<p>The issue arises after a few steps: the gradient norms become NaN and loss is 0. Eventually, the model’s predictions also yield NaN and then the training crashes.</p>
<p>Here’s what I’ve tried:</p>
<ul>
<li>Verified that only the None labels are handled with the zero loss.</li>
<li>Checked that the data inputs do not contain NaN values.</li>
</ul>
<p>Despite these checks, the problem persists. Has anyone faced a similar issue or have any suggestions on how to properly handle None labels in the dataset without leading to NaN gradients?</p>
<p>Thanks in advance for your help!</p>
<p>Edit: It seems that if I zero out the label before calculating the output, I don’t get NaN. Is this the correct procedure if I want to ignore samples on the fly?</p>

username: KFrank
cooked: <p>Hi Lpolisi!</p>
<aside class="quote no-group" data-username="lpolisi" data-post="1" data-topic="206144" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lpolisi/48/63030_2.png" class="avatar"> lpolisi:</div>
<blockquote>
<p>I’ve encountered an issue while training my model with a dataset that occasionally has samples with <code>None</code> labels.</p>
</blockquote>
</aside>
<p>You talk about labels with <code>None</code> values, but, in your example, you have<br>
labels (your <code>y</code>) with <code>nan</code> values.  <code>None</code> and <code>nan</code> are different.  Following<br>
your example, I will talk about <code>nan</code>s, as they are the cause of your issue.</p>
<aside class="quote no-group">
<blockquote>
<p>The issue arises after a few steps: the gradient norms become NaN</p>
</blockquote>
</aside>
<p>Yes.  <code>nan</code>s, by design,  pollute everything.  (For example, <code>0 * nan = nan</code>.)<br>
So even though you zero out the <code>nan</code>s in <code>loss</code>, you still have <code>nan</code>s in the<br>
computation graph and they pollute the gradients when you backpropagate.</p>
<aside class="quote no-group">
<blockquote>
<p>any suggestions on how to properly handle None labels in the dataset without leading to NaN gradients?</p>
<p>Edit: It seems that if I zero out the label before calculating the output, I don’t get NaN. Is this the correct procedure if I want to ignore samples on the fly?</p>
</blockquote>
</aside>
<p>Yes, this is the correct approach.  Get rid of the <code>nan</code>s before you perform<br>
any computation that mixes them with a <code>requires_grad = True</code> tensor<br>
and you will avoid having any <code>nan</code>s lurking in your computation graph that<br>
would just waiting to pollute your gradients when you backpropagate.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.3.1
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.arange (5.0, requires_grad = True)
&gt;&gt;&gt; y = torch.ones (5)
&gt;&gt;&gt; y[1:3] = float ('nan')
&gt;&gt;&gt;
&gt;&gt;&gt; nan_indices = torch.isnan (y).nonzero()
&gt;&gt;&gt; y
tensor([1., nan, nan, 1., 1.])
&gt;&gt;&gt; nan_indices
tensor([[1],
        [2]])
&gt;&gt;&gt;
&gt;&gt;&gt; loss_unr = (t - y)**2
&gt;&gt;&gt; loss_unr                      # damage is done, need to get rid of nans first
tensor([1., nan, nan, 4., 9.], grad_fn=&lt;PowBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; # get rid of nans
&gt;&gt;&gt; y[nan_indices] = 99.0
&gt;&gt;&gt; loss_unr = (t - y)**2         # no nans hiding in computation graph
&gt;&gt;&gt; loss_unr                      # no nans
tensor([1.0000e+00, 9.6040e+03, 9.4090e+03, 4.0000e+00, 9.0000e+00],
       grad_fn=&lt;PowBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; loss_unr[nan_indices] = 0.0   # zero out bogus values
&gt;&gt;&gt; loss = loss_unr.sum()
&gt;&gt;&gt; loss                          # no nans
tensor(14., grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; t.grad                        # grad is not polluted with nans
tensor([-2., -0., -0.,  4.,  6.])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>