Why my train and validation loss is as nan?

L_Z: <p>I have added in more data to train the model, but I am not sure the reason why the train and validation loss are as nan which means approaching to infinity. And then the the validation accuracy does not update. Any possible solutions to this?</p>

ptrblck: <aside class="quote no-group" data-username="L_Z" data-post="1" data-topic="206578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l_z/48/71230_2.png" class="avatar"> L_Z:</div>
<blockquote>
<p>train and validation loss are as nan which means approaching to infinity</p>
</blockquote>
</aside>
<p>A NaN loss is not the same as Inf, as the former might be caused by e.g. an invalid input while the latter can be caused if the training diverges.<br>
Could you check if the newly added data contains any NaNs or Infs?</p>

L_Z: <p>Thank you for your response. So I have two models, and I was passing in the same set of the data to them as well as same way of loading and preprocess the data. But then, one can train normally, the second model gives my both train and val loss as nan. And I was getting an error message while the training continues, UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at …/aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)<br>
return F.conv1d(input, weight, bias, self.stride,<br>
is this the reason? Or does that mean there are something wrong with my second model’s implementation? Any suggestions on how to debug that?</p>

ptrblck: <p>Are you able to reproduce the same warning in running the <code>nn.Conv1d</code> layer alone? If so, could you post a minimal and executable code snippet reproducing the warning?<br>
In case you are using an older PyTorch release, could you also update to the latest stable or nightly release?</p>

L_Z: <p>I will try update to the newest version of PyTorch first. But I am not sure if there is a specific version required for cuda 11.8 and cu121? And right now my PyTorch version is as 2.3.0.</p>

ptrblck: <p>For any “newer” GPUs, we recommend using 12.1 or 12.4 (e.g. anything Ampere+). The CUDA 11.8 builds are kept alive for users, who could not update their driver in the last ~1.5 years to support CUDA 12.x.</p>

L_Z: <p>Disregard of the error message for now since the model still trains. Do you have any suggestions on how to debug the reason behind getting nan or determine the potential causes? With only 1dCNN, the train loss increases in the second epoch, and stops updating. After I added in the customized activation function, I am getting NaNs. So my only guessing is that my activation function leads to NaN in my data?</p>

ptrblck: <aside class="quote no-group" data-username="L_Z" data-post="7" data-topic="206578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l_z/48/71230_2.png" class="avatar"> L_Z:</div>
<blockquote>
<p>After I added in the customized activation function, I am getting NaNs. So my only guessing is that my activation function leads to NaN in my data?</p>
</blockquote>
</aside>
<p>This could also be the case. What is your custom activation function doing?</p>

L_Z: <p>Essentially, I am trying to implement the model in this research paper: <a href="https://ieeexplore.ieee.org/document/9760210/authors#authors" class="inline-onebox" rel="noopener nofollow ugc">Residential Appliance Identification Using 1-D Convolutional Neural Network Based on Multiscale Sinusoidal Initializers | IEEE Journals &amp; Magazine | IEEE Xplore</a>. They have introduced an activation function called SL-ReLU which does not set negative values to zero. But I am actually not sure if I have implemented correctly, so after conv1d, I set the SK as the same shape as the output of conv1d. and then this is the activation function code:</p>
<h1><a name="sl-relu-activation-function-1" class="anchor" href="#sl-relu-activation-function-1"></a>SL-ReLU activation function</h1>
<p>class SLReLU(nn.Module):<br>
def <strong>init</strong>(self, initial_delta=1.0):<br>
super(SLReLU, self).<strong>init</strong>()<br>
self.delta = nn.Parameter(torch.tensor(initial_delta, dtype=torch.float32))  # Learnable parameter δ</p>
<p>def forward(self, x, skli):<br>
# x: input tensor<br>
# skli: sinusoidal kernel values, same shape as x</p>
<pre><code>   # Conditions for SL-ReLU
   negative_mask = x &lt;= 0
   relu_mask = (x &gt; 0) &amp; (x &lt;= skli)
   log_mask = x &gt; skli
  
   # Apply the conditions
   negative_part = x / (1 + x.abs())
   relu_part = F.relu(x)
   log_part = torch.log(self.delta * x + 1) + torch.abs(torch.log(self.delta * skli + 1) - skli)
  
   # Combine the parts
   output = negative_part * negative_mask + relu_part * relu_mask + log_part * log_mask
  
   return output
</code></pre>

ptrblck: <p>Did you make sure the <code>torch.log</code> cannot create <code>NaN</code> outputs by e.g. clipping <code>self.delta</code> or the input?<br>
Here is a small example using a negative <code>delta</code> (which is trainable according to your code):</p>
<pre data-code-wrap="python"><code class="lang-python">delta = torch.tensor(-5., requires_grad=True)
x = torch.tensor(1.)

out = torch.log(delta * x + 1)
print(out)
# tensor(nan, grad_fn=&lt;LogBackward0&gt;)
</code></pre>

L_Z: <p>Oh, I did not. How can I avoid this? So if my values passing to torch.log() are negative, then NaN will be outputted. So how should I clip self.data or the input?<br>
This case has not being introduced in the paper.<br>
Also, I am just wondering do I need to include the dimension for batch_size for the skli? Or just as the shape of (output_channels, sequence_len) is good?</p>

ptrblck: <p>You could <code>clip</code> <code>self.delta</code> and/or <code>x</code> making sure the input to <code>torch.log</code> is not negative, but I’m not familiar with the paper so also don’t know if this is a valid approach.</p>