BatchNorm and ReLU

divyesh_rajpura: <pre><code class="lang-auto">Model(
  (multi_tdnn): MultiTDNN(
    (multi_tdnn): Sequential(
      (0): TDNN(
        (conv): Conv1d(60, 512, kernel_size=(5,), stride=(1,))
        (nonlinearity): ReLU()
        (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): TDNN(
        (conv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), dilation=(2,))
        (nonlinearity): ReLU()
        (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): TDNN(
        (conv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), dilation=(3,))
        (nonlinearity): ReLU()
        (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): TDNN(
        (conv): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
        (nonlinearity): ReLU()
        (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (4): TDNN(
        (conv): Conv1d(512, 1500, kernel_size=(1,), stride=(1,))
        (nonlinearity): ReLU()
        (bn): BatchNorm1d(1500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
  )
  (stats_pool): StatsPool()
  (linear1): Linear(in_features=3000, out_features=512, bias=True)
  (linear2): Linear(in_features=512, out_features=1739, bias=True)
  (nonlinearity): ReLU()
)
</code></pre>
<p>I am using above architecture for my experimentation. But, it gives me NaN vales. From my analysis, I think ReLU and BatchNorm in TDNN layer is causing problem.</p>
<p><strong>Conv1d --&gt; ReLU --&gt; BatchNorm1d                        &lt;== Gives NaN</strong><br>
<strong>Conv1d --&gt; Tanh --&gt; BatchNorm1d                         &lt;== Working Perfectly</strong><br>
<strong>Conv1d --&gt; ReLU --&gt; Dropout                                 &lt;== Gives NaN</strong><br>
<strong>Conv1d --&gt; Tanh --&gt; Dropout                                  &lt;== Working Perfectly</strong><br>
<strong>Conv1d --&gt; ReLU --&gt; BatchNorm1d --&gt; Dropout    &lt;== Working Perfectly</strong></p>
<p>Can someone please guide me on this issue?</p>

dpernes: <p>ReLU might be zeroing many of your activations, leading to very low variance on the activations which in turn leads to NaN’s (due to 0 by 0 division). Check the way you are scaling the input data and/or try another activation function (e.g. leaky relu).</p>

divyesh_rajpura: <p><a class="mention" href="/u/dpernes">@dpernes</a>, Thank you for your reply.<br>
I have also think of these that zeros by ReLU can lead to division by zero. But then I have checked the PyTorch implementation of <a href="https://pytorch.org/docs/stable/nn.html#batchnorm1d" rel="nofollow noopener">BatchNorm1d</a>, and I can see that they have added eps to variance to overcome this.</p>

ptrblck: <p>Could you give us some information about your setup?<br>
I.e. which PyTorch, CUDA, cudnn versions are you using?<br>
Are you seeing these NaNs on the CPU, GPU or both?<br>
Are you able to reproduce the NaNs quickly (in a couple of iterations)?</p>

divyesh_rajpura: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply.</p>
<p>Below are details about configuration.<br>
PyTorch --&gt; 1.4.0<br>
CUDA --&gt; 10.2</p>
<p>I am not able to locate CuDNN. If CuDNN comes inbuilt with the CUDA, then it might be installed, but I haven’t installed it myself.</p>
<p>Yes, I have checked on both GPU and CPU I am getting NaN just after 5 to 10 iterations (i.e. 5 to 10 batches).</p>

ptrblck: <p>Could you check the inputs for NaNs and Infs, please?<br>
I assume the NaNs are returned during training?<br>
Could you store the inputs with the model’s <code>state_dict</code>, so that we could reproduce it in a single iteration?</p>

braindotai: <p>Might sound silly but, try reduce learning rate. And also try using gradient clipping as well -</p>
<pre><code class="lang-python">loss.backward()

torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

optimizer.step()
</code></pre>

divyesh_rajpura: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thank you for reply.</p>
<blockquote>
<p>Could you check the inputs for NaNs and Infs, please?<br>
I assume the NaNs are returned during training?</p>
</blockquote>
<p>Yes, NaN coming during training. Currently, I have already trained my model with <strong>Conv1d → ReLU → BatchNorm → Dropout</strong> setup for TDNN block for 6 epochs without any problem. I have also successfully trained another LSTM based architecture on same data. So I believe there will be no NaN and Inf.</p>
<blockquote>
<p>Could you store the inputs with the model’s  <code>state_dict</code> , so that we could reproduce it in a single iteration?</p>
</blockquote>
<p>You want script and sample data from my side to reproduce the same by you, am I correct?</p>

ptrblck: <aside class="quote no-group" data-username="divyesh_rajpura" data-post="8" data-topic="76563">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/divyesh_rajpura/48/12485_2.png" class="avatar"> divyesh_rajpura:</div>
<blockquote>
<p>You want script and sample data from my side to reproduce the same by you, am I correct?</p>
</blockquote>
</aside>
<p>Yes, that is correct.<br>
We are seeing more NaN issues recently, which could be solved so far, but it’s still a red flag if these issues are posted frequently, so that I would like to make sure we are not missing an internal bug.</p>

divyesh_rajpura: <p>Sure. I will get back to you with scripts and data asap. Thank You.</p>

divyesh_rajpura: <p><a class="mention" href="/u/braindotai">@braindotai</a>, Thank You for suggestion.<br>
I tried <code>clip_grad_norm</code>, but it doesn’t solve the problem.</p>

braindotai: <p>My pleasure, another suggestion though - make sure you have normalized the data.</p>

divyesh_rajpura: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have create repository on GitHub which consist of script and sample data. <a href="https://github.com/divyeshrajpura4114/smaple_classification" rel="nofollow noopener">code</a></p>
<p>Below are some basic detail about my architecture.<br>
The architecture is for multiclass classification. The full architecture I have already posted.<br>
The architecture consist of TDNN layers which works at frame level (input data is of shape (200, 20)). StatsPool layer computes mean and std and combination of both gives us single 3000 dimensional vector. FInally, three Linear layers where last linear layers have 1739 output as I have that many number of classes. I have used CrossEntropyLoss as criterian.</p>
<p>Please let me know if you have any query regarding architecture and data I have used.</p>

ptrblck: <p>Thanks for the code!<br>
I can reproduce the NaNs and isolated it to an Inf gradient after the <code>std</code> operation.<br>
Not sure, what’s causing it at the moment, but will debug further.</p>

divyesh_rajpura: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thank you for your time.<br>
If you add Dropout layer after BatchNorm1d i.e. (<strong>Conv1d --&gt; ReLU --&gt; BatchNorm1d --&gt; Dropout</strong>), it works like charm.<br>
I have also checked PReLU as activation function (<strong>Conv1d --&gt; PReLU --&gt; BatchNorm1d</strong>), which again works perfectly, only ReLU is causing problem. I have successfully trained model with this setup.</p>
<p>I hope this might help you to debug further.</p>

divyesh_rajpura: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, any updates on this issue?? Thank You.</p>

ptrblck: <p>Thanks for pinging. Not yet, I’ll try to take a look at it later this evening.</p>

divyesh_rajpura: <p>Sure. Please take your time.</p>

tom: <p>From the square root, the derivative of <code>std</code> is infinity at std=0. I would recommend working around it by using a backward hook and where. Or you could spell out the std and regularize before the square root.</p>
<p>Best regards</p>
<p>Thomas</p>

ptrblck: <p><a class="mention" href="/u/tom">@tom</a> is correct. The output of <code>std</code> indeed yields a zero output in the forward pass, which will yield the Inf gradients.</p>
<p>EDIT: <a href="https://github.com/divyeshrajpura4114/smaple_classification/blob/637d0cb01f5f1f1a791c877a01dc24eb77738983/train_dnn_sample.py#L24">This line</a> in particular raises the issue.</p>