Repository: open-mmlab/mmdetection

Issue:
SimOTAAssigner does not support half precision training
Thanks for your error report and we appreciate it a lot.

**Checklist**

1. I have searched related issues but cannot get the expected help.
2. I have read the [FAQ documentation](https://mmdetection.readthedocs.io/en/latest/faq.html) but cannot get the expected help.
3. The bug has not been fixed in the latest version.

**Describe the bug**
A clear and concise description of what the bug is.

Trying to run YOLOX with the SimOTAAssigner in 16-bit precision mode, but it fails with a `RuntimeError: "sqrt" "_vml_cpu" not implemented for 'Half'`.

**Reproduction**

1. What command or script did you run?

```none
A placeholder for the command.
```

2. Did you make any modifications on the code or config? Did you understand what you have modified?
3. What dataset did you use?

**Environment**

1. Please run `python mmdet/utils/collect_env.py` to collect necessary environment information and paste it here.
```
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
sys.platform: linux
Python: 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03) [GCC 9.4.0]
CUDA available: False
GCC: gcc (GCC) 9.4.0
PyTorch: 1.10.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.11.1+cu102
OpenCV: 4.5.3
MMCV: 1.4.6
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.3
MMDetection: 2.22.0+d939bb6
```
2. You may add addition that may be helpful for locating the problem, such as
    - How you installed PyTorch [e.g., pip, conda, source] **conda**
    - Other environment variables that may be related (such as `$PATH`, `$LD_LIBRARY_PATH`, `$PYTHONPATH`, etc.)

**Error traceback**
If applicable, paste the error trackback here.

```python-traceback
File "/home/username/Documents/github/project/project/yoloxnet.py", line 240, in evaluate
  loss_dict: dict = self.yolox_head.loss(
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/mmcv/runner/fp16_utils.py", line 197, in new_func
  return old_func(*args, **kwargs)
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/mmdet/models/dense_heads/yolox_head.py", line 381, in loss
  num_fg_imgs) = multi_apply(
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/mmdet/core/utils/misc.py", line 30, in multi_apply
  return tuple(map(list, zip(*map_results)))
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
  return func(*args, **kwargs)
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/mmdet/models/dense_heads/yolox_head.py", line 461, in _get_target_single
  assign_result = self.assigner.assign(
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/mmdet/core/bbox/assigners/sim_ota_assigner.py", line 85, in assign
  assign_result = self._assign(pred_scores, priors, decoded_bboxes,
File "/home/username/miniconda/envs/project/lib/python3.9/site-packages/mmdet/core/bbox/assigners/sim_ota_assigner.py", line 161, in _assign
  valid_pred_scores.sqrt_(), gt_onehot_label,
RuntimeError: "sqrt" "_vml_cpu" not implemented for 'Half'
```

**Bug fix**
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

Requires modifying the code somewhere here:

https://github.com/open-mmlab/mmdetection/blob/52a32763bef402a5eb684625d974eebe29755351/mmdet/core/bbox/assigners/sim_ota_assigner.py#L160-L162

The `.sqrt_()` doesn't work with `torch.float16` on CPU, and also, the binary_cross_entropy function seems to require float32 inputs.

Related to https://github.com/pytorch/pytorch/issues/50789


Discussion:
This might be true. Could you also try to make it work and create a PR if you have time? It's OK if you are not interested, we will  plan a fix in the future. 

Resolving patch:
diff --git a/mmdet/core/bbox/assigners/sim_ota_assigner.py b/mmdet/core/bbox/assigners/sim_ota_assigner.py
--- a/mmdet/core/bbox/assigners/sim_ota_assigner.py
+++ b/mmdet/core/bbox/assigners/sim_ota_assigner.py
@@ -157,9 +157,12 @@ def _assign(self,
                           num_valid, 1, 1))
 
         valid_pred_scores = valid_pred_scores.unsqueeze(1).repeat(1, num_gt, 1)
-        cls_cost = F.binary_cross_entropy(
-            valid_pred_scores.sqrt_(), gt_onehot_label,
-            reduction='none').sum(-1)
+        cls_cost = (
+            F.binary_cross_entropy(
+                valid_pred_scores.to(dtype=torch.float32).sqrt_(),
+                gt_onehot_label,
+                reduction='none',
+            ).sum(-1).to(dtype=valid_pred_scores.dtype))
 
         cost_matrix = (
             cls_cost * self.cls_weight + iou_cost * self.iou_weight +


Test patch:
diff --git a/tests/test_utils/test_assigner.py b/tests/test_utils/test_assigner.py
--- a/tests/test_utils/test_assigner.py
+++ b/tests/test_utils/test_assigner.py
@@ -11,8 +11,8 @@
 from mmdet.core.bbox.assigners import (ApproxMaxIoUAssigner,
                                        CenterRegionAssigner, HungarianAssigner,
                                        MaskHungarianAssigner, MaxIoUAssigner,
-                                       PointAssigner, TaskAlignedAssigner,
-                                       UniformAssigner)
+                                       PointAssigner, SimOTAAssigner,
+                                       TaskAlignedAssigner, UniformAssigner)
 
 
 def test_max_iou_assigner():
@@ -500,6 +500,21 @@ def test_uniform_assigner_with_empty_boxes():
     assert len(assign_result.gt_inds) == 0
 
 
+def test_sim_ota_assigner():
+    self = SimOTAAssigner(
+        center_radius=2.5, candidate_topk=1, iou_weight=3.0, cls_weight=1.0)
+    pred_scores = torch.FloatTensor([[0.2], [0.8]])
+    priors = torch.Tensor([[0, 12, 23, 34], [4, 5, 6, 7]])
+    decoded_bboxes = torch.Tensor([[[30, 40, 50, 60]], [[4, 5, 6, 7]]])
+    gt_bboxes = torch.Tensor([[23.6667, 23.8757, 238.6326, 151.8874]])
+    gt_labels = torch.LongTensor([2])
+    assign_result = self.assign(pred_scores, priors, decoded_bboxes, gt_bboxes,
+                                gt_labels)
+
+    expected_gt_inds = torch.LongTensor([0, 0])
+    assert torch.all(assign_result.gt_inds == expected_gt_inds)
+
+
 def test_task_aligned_assigner():
     with pytest.raises(AssertionError):
         TaskAlignedAssigner(topk=0)