Repository: open-mmlab/mmdetection

Issue:
[Bug] Inconsistent float precision in training log
### Prerequisite

- [X] I have searched [Issues](https://github.com/open-mmlab/mmdetection/issues) and [Discussions](https://github.com/open-mmlab/mmdetection/discussions) but cannot get the expected help.
- [X] I have read the [FAQ documentation](https://mmdetection.readthedocs.io/en/latest/faq.html) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version (master)](https://github.com/open-mmlab/mmdetection) or [latest version (3.x)](https://github.com/open-mmlab/mmdetection/tree/dev-3.x).

### Task

I have modified the scripts/configs, or I'm working on my own tasks/models/datasets.

### Branch

master branch https://github.com/open-mmlab/mmdetection

### Environment

sys.platform: linux
Python: 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) [GCC 7.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: Tesla V100-SXM2-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.2, V10.2.8
GCC: gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.2
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
  - CuDNN 7.6.5
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.0
OpenCV: 4.5.3
MMCV: 1.5.0
MMCV Compiler: GCC 7.4
MMCV CUDA Compiler: 10.2
MMDetection: 2.21.0+


### Reproduces the problem - code sample

training on any dataset

### Reproduces the problem - command or script

training on any dataset

### Reproduces the problem - error message

2022-12-28 11:15:09,949 - mmdet - INFO -
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.00041
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = **0.00150**
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.00013
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.00000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.00009
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.00101
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.09577
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.09577
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.09577
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.00000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.05205
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.17818

2022-12-28 11:15:10,003 - mmdet - INFO - Exp name: autoassign_r50_fpn_8x2_1x_mae_x-bkpd.py
2022-12-28 11:15:10,003 - mmdet - INFO - Epoch(val) [100][1177] bbox_mAP: 0.0000, bbox_mAP_50: **0.0010**, bbox_mAP_75: 0.0000, bbox_mAP_s: 0.0000, bbox_mAP_m: 0.0000, bbox_mAP_l: 0.0010, bbox_mAP_copypaste: 0.000 0.001 0.000 0.000 0.000 0.001

-------------------------------

I am training on my custom dataset and I notice that the mAPs in log have different float precision, which results in different scores after rounding off.
For example, mAP_50 is 0.00150 before and 0.0010 after.
This is much more severe when the performance of the model is bad. A minor difference in float precision will have more significant difference when selecting the best model.

### Additional information

![图片](https://user-images.githubusercontent.com/13494034/210727720-34bec4c7-8cae-4a76-bc85-a55364d7354b.png)



Discussion:

Resolving patch:
diff --git a/mmdet/datasets/coco.py b/mmdet/datasets/coco.py
--- a/mmdet/datasets/coco.py
+++ b/mmdet/datasets/coco.py
@@ -525,7 +525,7 @@ def evaluate_det_segm(self,
 
                 for item in metric_items:
                     val = float(
-                        f'{cocoEval.stats[coco_metric_names[item]]:.3f}')
+                        f'{cocoEval.stats[coco_metric_names[item]]:.4f}')
                     eval_results[item] = val
             else:
                 cocoEval.evaluate()
@@ -579,13 +579,13 @@ def evaluate_det_segm(self,
                 for metric_item in metric_items:
                     key = f'{metric}_{metric_item}'
                     val = float(
-                        f'{cocoEval.stats[coco_metric_names[metric_item]]:.3f}'
+                        f'{cocoEval.stats[coco_metric_names[metric_item]]:.4f}'
                     )
                     eval_results[key] = val
                 ap = cocoEval.stats[:6]
                 eval_results[f'{metric}_mAP_copypaste'] = (
-                    f'{ap[0]:.3f} {ap[1]:.3f} {ap[2]:.3f} {ap[3]:.3f} '
-                    f'{ap[4]:.3f} {ap[5]:.3f}')
+                    f'{ap[0]:.4f} {ap[1]:.4f} {ap[2]:.4f} {ap[3]:.4f} '
+                    f'{ap[4]:.4f} {ap[5]:.4f}')
 
         return eval_results
 
diff --git a/mmdet/datasets/lvis.py b/mmdet/datasets/lvis.py
--- a/mmdet/datasets/lvis.py
+++ b/mmdet/datasets/lvis.py
@@ -408,7 +408,7 @@ def evaluate(self,
                 lvis_eval.summarize()
                 for k, v in lvis_eval.get_results().items():
                     if k.startswith('AR'):
-                        val = float('{:.3f}'.format(float(v)))
+                        val = float('{:.4f}'.format(float(v)))
                         eval_results[k] = val
             else:
                 lvis_eval.evaluate()
@@ -454,10 +454,10 @@ def evaluate(self,
                 for k, v in lvis_results.items():
                     if k.startswith('AP'):
                         key = '{}_{}'.format(metric, k)
-                        val = float('{:.3f}'.format(float(v)))
+                        val = float('{:.4f}'.format(float(v)))
                         eval_results[key] = val
                 ap_summary = ' '.join([
-                    '{}:{:.3f}'.format(k, float(v))
+                    '{}:{:.4f}'.format(k, float(v))
                     for k, v in lvis_results.items() if k.startswith('AP')
                 ])
                 eval_results['{}_mAP_copypaste'.format(metric)] = ap_summary


Test patch:
diff --git a/tests/test_data/test_datasets/test_panoptic_dataset.py b/tests/test_data/test_datasets/test_panoptic_dataset.py
--- a/tests/test_data/test_datasets/test_panoptic_dataset.py
+++ b/tests/test_data/test_datasets/test_panoptic_dataset.py
@@ -444,13 +444,13 @@ def test_instance_segmentation_evaluation():
 
     # Here is the results for instance segmentation:
     # {
-    #     'segm_mAP': 0.5, 'segm_mAP_50': 0.626, 'segm_mAP_75': 0.5,
+    #     'segm_mAP': 0.5005, 'segm_mAP_50': 0.626, 'segm_mAP_75': 0.5,
     #     'segm_mAP_s': 0.5, 'segm_mAP_m': -1.0, 'segm_mAP_l': -1.0,
     #     'segm_mAP_copypaste': '0.500 0.626 0.500 0.500 -1.000 -1.000',
-    #     'bbox_mAP': 0.564, 'bbox_mAP_50': 0.626, 'bbox_mAP_75': 0.626,
+    #     'bbox_mAP': 0.5636, 'bbox_mAP_50': 0.626, 'bbox_mAP_75': 0.626,
     #     'bbox_mAP_s': 0.564, 'bbox_mAP_m': -1.0, 'bbox_mAP_l': -1.0,
     #     'bbox_mAP_copypaste': '0.564 0.626 0.626 0.564 -1.000 -1.000'
     # }
 
-    assert np.isclose(parsed_results['segm_mAP'], 0.5)
-    assert np.isclose(parsed_results['bbox_mAP'], 0.564)
+    assert np.isclose(parsed_results['segm_mAP'], 0.5005)
+    assert np.isclose(parsed_results['bbox_mAP'], 0.5636)