Repository: open-mmlab/mmdetection

Issue:
[Bug] WIDERFace official config training loss gets NAN
### Prerequisite

- [X] I have searched [Issues](https://github.com/open-mmlab/mmdetection/issues) and [Discussions](https://github.com/open-mmlab/mmdetection/discussions) but cannot get the expected help.
- [X] I have read the [FAQ documentation](https://mmdetection.readthedocs.io/en/latest/faq.html) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version (master)](https://github.com/open-mmlab/mmdetection) or [latest version (3.x)](https://github.com/open-mmlab/mmdetection/tree/dev-3.x).

### Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

### Branch

master branch https://github.com/open-mmlab/mmdetection

### Environment

```
No CUDA runtime is found, using CUDA_HOME='/mnt/lustre/share/cuda-9.0'
sys.platform: linux
Python: 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59) [GCC 7.5.0]
CUDA available: False
GCC: gcc (GCC) 5.4.0
PyTorch: 1.8.1+cuda90.cudnn7.6.5
PyTorch compiling details: PyTorch built with:
  - GCC 5.4
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201307 (a.k.a. OpenMP 4.0)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=9.0, CUDNN_VERSION=7.6.5, CXX_COMPILER=/usr/local/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type
-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -Wno-unused-but-set-variable -Wno-maybe-unini
tialized -fno-math-errno -fno-trapping-math -Werror=format, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 
```

TorchVision: 0.8.0a0+2f40a48
OpenCV: 4.6.0
MMCV: 1.4.2
MMCV Compiler: GCC 5.4
MMCV CUDA Compiler: 9.0
MMDetection: 2.26.0+31c8495


### Reproduces the problem - code sample

-

### Reproduces the problem - command or script

`GPUS_PER_NODE=8 GPUS=8 CPUS_PER_TASK=2 ./tools/slurm_train.sh Test train configs/wider_face/ssd300_wider_face.py work_dirs/ssd300_wider_face
`

### Reproduces the problem - error message

```
2022-12-29 15:58:42,698 - mmdet - INFO - Epoch(val) [3][404]    AP50: 0.0820, mAP: 0.0820                                                                                                  
2022-12-29 15:58:51,194 - mmdet - INFO - Epoch [4][1/55]        lr: 1.990e-03, eta: 3:01:01, time: 7.665, data_time: 6.831, memory: 9269, loss_cls: 7.6470, loss_bbox: 3.2089, loss: 10.8559 
2022-12-29 15:58:55,255 - mmdet - INFO - Epoch [4][2/55]        lr: 2.002e-03, eta: 3:00:15, time: 4.068, data_time: 3.281, memory: 9269, loss_cls: 5.7089, loss_bbox: 2.8437, loss: 8.5526
2022-12-29 15:59:00,380 - mmdet - INFO - Epoch [4][3/55]        lr: 2.014e-03, eta: 2:59:36, time: 5.130, data_time: 4.199, memory: 9269, loss_cls: 4.7298, loss_bbox: 2.7796, loss: 7.5094  
2022-12-29 15:59:12,074 - mmdet - INFO - Epoch [4][4/55]        lr: 2.026e-03, eta: 2:59:43, time: 11.684, data_time: 10.868, memory: 9269, loss_cls: 8.7619, loss_bbox: 3.1394, loss: 11.9013
2022-12-29 15:59:19,008 - mmdet - INFO - Epoch [4][5/55]        lr: 2.038e-03, eta: 2:59:17, time: 6.948, data_time: 5.999, memory: 9269, loss_cls: 9.7502, loss_bbox: 3.8803, loss: 13.6304 
2022-12-29 15:59:31,448 - mmdet - INFO - Epoch [4][6/55]        lr: 2.050e-03, eta: 2:59:28, time: 12.441, data_time: 11.668, memory: 9269, loss_cls: 11.5603, loss_bbox: 3.4775, loss: 15.0378
2022-12-29 15:59:39,742 - mmdet - INFO - Epoch [4][7/55]        lr: 2.062e-03, eta: 2:59:12, time: 8.299, data_time: 7.516, memory: 9269, loss_cls: 11.2196, loss_bbox: 3.4839, loss: 14.7034
2022-12-29 15:59:50,085 - mmdet - INFO - Epoch [4][8/55]        lr: 2.074e-03, eta: 2:59:09, time: 10.333, data_time: 9.528, memory: 9269, loss_cls: 10.2483, loss_bbox: 4.0472, loss: 14.2955
2022-12-29 16:00:01,364 - mmdet - INFO - Epoch [4][9/55]        lr: 2.086e-03, eta: 2:59:12, time: 11.204, data_time: 10.436, memory: 9269, loss_cls: 5.0894, loss_bbox: 4.6891, loss: 9.7784
2022-12-29 16:00:09,459 - mmdet - INFO - Epoch [4][10/55]       lr: 2.098e-03, eta: 2:58:54, time: 8.180, data_time: 7.360, memory: 9269, loss_cls: 27.2303, loss_bbox: 4.1039, loss: 31.3342
2022-12-29 16:00:22,794 - mmdet - INFO - Epoch [4][11/55]       lr: 2.110e-03, eta: 2:59:11, time: 13.336, data_time: 12.508, memory: 9269, loss_cls: 17.0886, loss_bbox: 4.6867, loss: 21.7753
2022-12-29 16:00:28,384 - mmdet - INFO - Epoch [4][12/55]       lr: 2.122e-03, eta: 2:58:37, time: 5.579, data_time: 4.766, memory: 9269, loss_cls: 17.4810, loss_bbox: 4.4995, loss: 21.9805
2022-12-29 16:00:41,585 - mmdet - INFO - Epoch [4][13/55]       lr: 2.134e-03, eta: 2:58:52, time: 13.219, data_time: 12.392, memory: 9269, loss_cls: 19.9781, loss_bbox: 5.2929, loss: 25.2710
2022-12-29 16:00:46,719 - mmdet - INFO - Epoch [4][14/55]       lr: 2.146e-03, eta: 2:58:15, time: 5.120, data_time: 4.322, memory: 9269, loss_cls: 47.5561, loss_bbox: 11.1209, loss: 58.6769              2022-12-29 16:01:00,184 - mmdet - INFO - Epoch [4][15/55]       lr: 2.158e-03, eta: 2:58:32, time: 13.465, data_time: 12.676, memory: 9269, loss_cls: 109.4393, loss_bbox: 9.6013, loss: 119.0406           
2022-12-29 16:01:06,761 - mmdet - INFO - Epoch [4][16/55]       lr: 2.170e-03, eta: 2:58:05, time: 6.563, data_time: 5.801, memory: 9269, loss_cls: 46.4443, loss_bbox: 4.9081, loss: 51.3524               2022-12-29 16:01:20,119 - mmdet - INFO - Epoch [4][17/55]       lr: 2.182e-03, eta: 2:58:20, time: 13.379, data_time: 12.573, memory: 9269, loss_cls: 4942.5396, loss_bbox: 104.6648, loss: 5047.2051       
2022-12-29 16:01:25,744 - mmdet - INFO - Epoch [4][18/55]       lr: 2.194e-03, eta: 2:57:47, time: 5.618, data_time: 4.786, memory: 9269, loss_cls: 9061150126477076856832.0000, loss_bbox: 618768130152895348736.0000, loss: 9679916567780111941632.0000                                                                                                                                                
2022-12-29 16:01:38,900 - mmdet - INFO - Epoch [4][19/55]       lr: 2.206e-03, eta: 2:58:01, time: 13.167, data_time: 12.410, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan       
2022-12-29 16:01:47,495 - mmdet - INFO - Epoch [4][20/55]       lr: 2.218e-03, eta: 2:57:47, time: 8.595, data_time: 7.856, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan           
2022-12-29 16:01:57,279 - mmdet - INFO - Epoch [4][21/55]       lr: 2.230e-03, eta: 2:57:40, time: 9.783, data_time: 9.023, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan         
2022-12-29 16:02:06,307 - mmdet - INFO - Epoch [4][22/55]       lr: 2.242e-03, eta: 2:57:28, time: 9.026, data_time: 8.243, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan           
2022-12-29 16:02:15,655 - mmdet - INFO - Epoch [4][23/55]       lr: 2.254e-03, eta: 2:57:18, time: 9.347, data_time: 8.598, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan         
2022-12-29 16:02:24,530 - mmdet - INFO - Epoch [4][24/55]       lr: 2.266e-03, eta: 2:57:06, time: 8.882, data_time: 8.122, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan           
2022-12-29 16:02:34,867 - mmdet - INFO - Epoch [4][25/55]       lr: 2.278e-03, eta: 2:57:02, time: 10.331, data_time: 9.585, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan        
2022-12-29 16:02:43,227 - mmdet - INFO - Epoch [4][26/55]       lr: 2.290e-03, eta: 2:56:47, time: 8.361, data_time: 7.602, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan           
2022-12-29 16:02:54,472 - mmdet - INFO - Epoch [4][27/55]       lr: 2.302e-03, eta: 2:56:48, time: 11.245, data_time: 10.471, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan       
2022-12-29 16:03:02,513 - mmdet - INFO - Epoch [4][28/55]       lr: 2.314e-03, eta: 2:56:31, time: 8.043, data_time: 7.290, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:03:16,707 - mmdet - INFO - Epoch [4][29/55]       lr: 2.326e-03, eta: 2:56:49, time: 14.195, data_time: 13.455, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:03:21,485 - mmdet - INFO - Epoch [4][30/55]       lr: 2.338e-03, eta: 2:56:13, time: 4.778, data_time: 4.021, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:03:40,089 - mmdet - INFO - Epoch [4][31/55]       lr: 2.350e-03, eta: 2:56:56, time: 18.605, data_time: 17.831, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:03:41,846 - mmdet - INFO - Epoch [4][32/55]       lr: 2.362e-03, eta: 2:56:03, time: 1.725, data_time: 0.818, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:03:58,572 - mmdet - INFO - Epoch [4][33/55]       lr: 2.374e-03, eta: 2:56:35, time: 16.757, data_time: 15.999, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:04:00,291 - mmdet - INFO - Epoch [4][34/55]       lr: 2.386e-03, eta: 2:55:42, time: 1.718, data_time: 0.796, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:04:17,374 - mmdet - INFO - Epoch [4][35/55]       lr: 2.398e-03, eta: 2:56:16, time: 17.080, data_time: 16.316, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan
2022-12-29 16:04:19,015 - mmdet - INFO - Epoch [4][36/55]       lr: 2.410e-03, eta: 2:55:23, time: 1.645, data_time: 0.813, memory: 9269, loss_cls: nan, loss_bbox: nan, loss: nan

```

### Additional information

I used the official ssd300_wider_face.config. At first it runs smoothly, but the loss becomes NAN in the 4th epoch.



Discussion:
Thanks for your report! We will verify and fix it as soon as possible.
I ran into the same issue.
@RangiLyu any idea how to solve this?
The same happened to me. any update?

Resolving patch:
diff --git a/configs/wider_face/ssd300_wider_face.py b/configs/wider_face/ssd300_wider_face.py
--- a/configs/wider_face/ssd300_wider_face.py
+++ b/configs/wider_face/ssd300_wider_face.py
@@ -5,7 +5,7 @@
 model = dict(bbox_head=dict(num_classes=1))
 # optimizer
 optimizer = dict(type='SGD', lr=0.012, momentum=0.9, weight_decay=5e-4)
-optimizer_config = dict()
+optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
 # learning policy
 lr_config = dict(
     policy='step',


Test patch: