Model producing NAN values after few epochs

amangaurav-yalai: <p>Hey I am doing Speech Enhancement and when i start training it ran for few epochs (9)  but at 10th epoch training it gave error Input waveform is NAN . I used Lr = 0.0001 and tried changing it but nothing happening .</p>
<p>Epoch 10 | Train Loss: 0.001552 | SmoothL1: 0.001329 | ERB_spec_loss: 0.010675<br>
[rank0]: Traceback (most recent call last):<br>
[rank0]:   File “/data/aman/scripts/attenuate/attenuate_ddp_train.py”, line 283, in<br>
[rank0]:<br>
[rank0]:   File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1532, in _wrapped_call_impl<br>
[rank0]:     return self._call_impl(*args, **kwargs)<br>
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
[rank0]:   File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1541, in _call_impl<br>
[rank0]:     return forward_call(*args, **kwargs)<br>
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
[rank0]:   File “/data/aman/scripts/attenuate/loss.py”, line 344, in forward<br>
[rank0]:     raise ValueError(“Input waveforms contain NaN or Inf values”)<br>
[rank0]: ValueError: Input waveforms contain NaN or Inf values<br>
[rank1]: Traceback (most recent call last):<br>
[rank1]:   File “/data/aman/scripts/attenuate/attenuate_ddp_train.py”, line 283, in<br>
[rank1]:<br>
[rank1]:   File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1532, in _wrapped_call_impl<br>
[rank1]:     return self._call_impl(*args, **kwargs)<br>
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
[rank1]:   File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1541, in _call_impl<br>
[rank1]:     return forward_call(*args, **kwargs)<br>
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
[rank1]:   File “/data/aman/scripts/attenuate/loss.py”, line 344, in forward<br>
[rank1]:     raise ValueError(“Input waveforms contain NaN or Inf values”)<br>
[rank1]: ValueError: Input waveforms contain NaN or Inf values<br>
E1019 03:37:32.660000 124960817152128 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 375841) of binary: /data/aman/envs/attenuate/bin/python<br>
Traceback (most recent call last):<br>
File “”, line 198, in _run_module_as_main<br>
File “”, line 88, in _run_code<br>
File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/distributed/run.py”, line 883, in<br>
main()<br>
File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 347, in wrapper<br>
return f(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^<br>
File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/distributed/run.py”, line 879, in main<br>
run(args)<br>
File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/distributed/run.py”, line 870, in run<br>
elastic_launch(<br>
File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/distributed/launcher/api.py”, line 132, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/aman/envs/attenuate/lib/python3.12/site-packages/torch/distributed/launcher/api.py”, line 263, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>

nitinnageshkulkarni: <p>Hey, so this is a really common issue that happens to everyone at some point during training. The NaN error appearing at epoch 10 can be actually a sign that something in your numerical computation is becoming unstable.</p>
<p>Your learning rate is already pretty conservative at 0.0001, but that doesn’t necessarily prevent gradients from exploding. Try adding gradient clipping before your optimizer step try this<br>
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)<br>
optimizer.step()<br>
Start with a max_norm of 1.0 and adjust from there if needed.</p>