opened
Mxbonn
username_0
By default COCO doesn't use a single IoU to calculate the mAP but instead averages over a range of IoUs (IoU=0.50:0.95).
The evaluate function also outputs the mAP for IoU 0.5 and 0.75.
However calculating the mAP for IoU 0.5 and 0.75 using the `evaluate` method from the `super` class `CustomDataset`  gives a different result.

**This means that implementing a new dataset will give different evaluation results depending on whether it is implemented as a new subclass of `CustomDataset` or with COCO compatible labels as a `CocoDataset`.**

Minimal example:
```python
import os

import mmdet.datasets
import mmdet.apis
import mmcv
```


```python
checkpoint_path = "checkpoints/ssd300_coco_20200307-a92d2092.pth"
config_path = "configs/ssd/ssd300_coco.py"

cfg = mmcv.Config.fromfile(config_path)
cfg.model.pretrained = None
cfg.data.test.test_mode = True
cfg.test_cfg.max_per_img = 100 # Set to 100 instead of 200 as CustomDataset.evaluate has no maxDets
```


```python
test_dataset = mmdet.datasets.build_dataset(cfg.data.test)
test_dataloader = mmdet.datasets.build_dataloader(
    test_dataset,
    samples_per_gpu=1,
    workers_per_gpu=cfg.data.workers_per_gpu,
    dist=False,
    shuffle=False
)
```

    loading annotations into memory...
    Done (t=0.52s)
    creating index...
    index created!



```python
raw_model = mmdet.models.build_detector(cfg.model, train_cfg=None, test_cfg=cfg.test_cfg)
checkpoint = mmcv.runner.load_checkpoint(raw_model, checkpoint_path, map_location='cpu')
raw_model.CLASSES = test_dataset.CLASSES
model = mmcv.parallel.MMDataParallel(raw_model, device_ids=[0])
```


```python
results = mmdet.apis.single_gpu_test(model, test_dataloader)
```

    [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 39.0 task/s, elapsed: 128s, ETA:     0s


```python
test_dataset.evaluate(results)
```


    Evaluating bbox...
    Loading and preparing results...
    DONE (t=3.77s)
    creating index...
    index created!
    Running per image evaluation...
    Evaluate annotation type *bbox*
    DONE (t=53.15s).
    Accumulating evaluation results...
    DONE (t=7.20s).
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.255
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.436
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.263
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.065
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.277
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.422
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.237
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.347
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.364
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.108
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.401
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.582





    {'bbox_mAP': 0.255,
     'bbox_mAP_50': 0.436,
     'bbox_mAP_75': 0.263,
     'bbox_mAP_s': 0.065,
     'bbox_mAP_m': 0.277,
     'bbox_mAP_l': 0.422,
     'bbox_mAP_copypaste': '0.255 0.436 0.263 0.065 0.277 0.422'}




```python
super(type(test_dataset), test_dataset).evaluate(results, iou_thr=0.5)
```


    +----------------+-------+--------+--------+-------+
    | class          | gts   | dets   | recall | ap    |
    +----------------+-------+--------+--------+-------+
    | person         | 10777 | 151136 | 0.781  | 0.619 |
    | bicycle        | 314   | 1935   | 0.503  | 0.376 |
    | car            | 1918  | 17858  | 0.626  | 0.440 |
    | motorcycle     | 367   | 2282   | 0.703  | 0.571 |
    | airplane       | 143   | 3942   | 0.895  | 0.737 |
    | bus            | 283   | 875    | 0.742  | 0.693 |
    | train          | 190   | 4048   | 0.863  | 0.795 |
    | truck          | 414   | 2610   | 0.693  | 0.431 |
    | boat           | 424   | 8890   | 0.547  | 0.308 |
    | traffic light  | 634   | 12564  | 0.465  | 0.236 |
    | fire hydrant   | 101   | 274    | 0.703  | 0.662 |
    | stop sign      | 75    | 479    | 0.640  | 0.581 |
    | parking meter  | 60    | 147    | 0.583  | 0.498 |
    | bench          | 411   | 12074  | 0.457  | 0.267 |
    | bird           | 427   | 13891  | 0.433  | 0.296 |
    | cat            | 202   | 1221   | 0.931  | 0.851 |
    | dog            | 218   | 1027   | 0.821  | 0.713 |
    | horse          | 272   | 2077   | 0.794  | 0.671 |
    | sheep          | 354   | 5733   | 0.782  | 0.559 |
    | cow            | 372   | 4083   | 0.766  | 0.534 |
    | elephant       | 252   | 3908   | 0.897  | 0.738 |
    | bear           | 71    | 308    | 0.831  | 0.777 |
    | zebra          | 266   | 5540   | 0.880  | 0.797 |
    | giraffe        | 232   | 5620   | 0.901  | 0.812 |
    | backpack       | 371   | 2462   | 0.305  | 0.111 |
    | umbrella       | 407   | 4079   | 0.634  | 0.462 |
    | handbag        | 540   | 4092   | 0.274  | 0.079 |
    | tie            | 252   | 4043   | 0.480  | 0.314 |
    | suitcase       | 299   | 1934   | 0.515  | 0.334 |
    | frisbee        | 115   | 566    | 0.687  | 0.499 |
    | skis           | 240   | 5994   | 0.592  | 0.301 |
    | snowboard      | 69    | 343    | 0.478  | 0.311 |
    | sports ball    | 260   | 4571   | 0.504  | 0.382 |
    | kite           | 327   | 7137   | 0.648  | 0.385 |
    | baseball bat   | 145   | 1113   | 0.441  | 0.283 |
    | baseball glove | 148   | 1161   | 0.480  | 0.296 |
    | skateboard     | 179   | 1872   | 0.721  | 0.559 |
    | surfboard      | 267   | 4453   | 0.592  | 0.413 |
    | tennis racket  | 225   | 2015   | 0.631  | 0.524 |
    | bottle         | 1013  | 7547   | 0.523  | 0.277 |
    | wine glass     | 341   | 1171   | 0.408  | 0.290 |
    | cup            | 895   | 4976   | 0.534  | 0.330 |
    | fork           | 215   | 1029   | 0.414  | 0.228 |
    | knife          | 325   | 1714   | 0.237  | 0.065 |
    | spoon          | 253   | 1237   | 0.249  | 0.088 |
    | bowl           | 623   | 3130   | 0.652  | 0.431 |
    | banana         | 370   | 7206   | 0.635  | 0.300 |
    | apple          | 236   | 2028   | 0.424  | 0.154 |
    | sandwich       | 177   | 1156   | 0.678  | 0.462 |
    | orange         | 285   | 2710   | 0.561  | 0.285 |
    | broccoli       | 312   | 13007  | 0.769  | 0.325 |
    | carrot         | 365   | 9667   | 0.573  | 0.193 |
    | hot dog        | 125   | 470    | 0.480  | 0.363 |
    | pizza          | 284   | 2390   | 0.736  | 0.600 |
    | donut          | 328   | 2525   | 0.649  | 0.389 |
    | cake           | 310   | 1742   | 0.626  | 0.412 |
    | chair          | 1771  | 33923  | 0.537  | 0.279 |
    | couch          | 261   | 1659   | 0.785  | 0.590 |
    | potted plant   | 342   | 12017  | 0.661  | 0.308 |
    | bed            | 163   | 1625   | 0.785  | 0.632 |
    | dining table   | 695   | 8174   | 0.662  | 0.381 |
    | toilet         | 179   | 3298   | 0.866  | 0.730 |
    | tv             | 288   | 2276   | 0.795  | 0.671 |
    | laptop         | 231   | 913    | 0.740  | 0.647 |
    | mouse          | 106   | 627    | 0.708  | 0.557 |
    | remote         | 283   | 2146   | 0.378  | 0.138 |
    | keyboard       | 153   | 1376   | 0.784  | 0.624 |
    | cell phone     | 262   | 2546   | 0.462  | 0.287 |
    | microwave      | 55    | 366    | 0.727  | 0.562 |
    | oven           | 143   | 3618   | 0.783  | 0.523 |
    | toaster        | 9     | 16     | 0.111  | 0.007 |
    | sink           | 225   | 8280   | 0.702  | 0.447 |
    | refrigerator   | 126   | 2632   | 0.794  | 0.599 |
    | book           | 1129  | 21431  | 0.417  | 0.100 |
    | clock          | 267   | 4601   | 0.674  | 0.562 |
    | vase           | 274   | 1612   | 0.504  | 0.315 |
    | scissors       | 36    | 256    | 0.472  | 0.379 |
    | teddy bear     | 190   | 2154   | 0.674  | 0.535 |
    | hair drier     | 11    | 1      | 0.000  | 0.000 |
    | toothbrush     | 57    | 326    | 0.263  | 0.146 |
    +----------------+-------+--------+--------+-------+
    | mAP            |       |        |        | 0.430 |
    +----------------+-------+--------+--------+-------+





    {'mAP': 0.43032437562942505}




```python
super(type(test_dataset), test_dataset).evaluate(results, iou_thr=0.75)
```


    +----------------+-------+--------+--------+-------+
    | class          | gts   | dets   | recall | ap    |
    +----------------+-------+--------+--------+-------+
    | person         | 10777 | 151136 | 0.424  | 0.322 |
    | bicycle        | 314   | 1935   | 0.220  | 0.141 |
    | car            | 1918  | 17858  | 0.301  | 0.209 |
    | motorcycle     | 367   | 2282   | 0.351  | 0.275 |
    | airplane       | 143   | 3942   | 0.664  | 0.538 |
    | bus            | 283   | 875    | 0.625  | 0.589 |
    | train          | 190   | 4048   | 0.679  | 0.623 |
    | truck          | 414   | 2610   | 0.425  | 0.236 |
    | boat           | 424   | 8890   | 0.200  | 0.094 |
    | traffic light  | 634   | 12564  | 0.129  | 0.056 |
    | fire hydrant   | 101   | 274    | 0.594  | 0.532 |
    | stop sign      | 75    | 479    | 0.600  | 0.567 |
    | parking meter  | 60    | 147    | 0.433  | 0.373 |
    | bench          | 411   | 12074  | 0.221  | 0.138 |
    | bird           | 427   | 13891  | 0.267  | 0.194 |
    | cat            | 202   | 1221   | 0.757  | 0.617 |
    | dog            | 218   | 1027   | 0.651  | 0.529 |
    | horse          | 272   | 2077   | 0.540  | 0.418 |
    | sheep          | 354   | 5733   | 0.432  | 0.290 |
    | cow            | 372   | 4083   | 0.462  | 0.333 |
    | elephant       | 252   | 3908   | 0.643  | 0.511 |
    | bear           | 71    | 308    | 0.718  | 0.664 |
    | zebra          | 266   | 5540   | 0.624  | 0.552 |
    | giraffe        | 232   | 5620   | 0.668  | 0.582 |
    | backpack       | 371   | 2462   | 0.127  | 0.029 |
    | umbrella       | 407   | 4079   | 0.327  | 0.218 |
    | handbag        | 540   | 4092   | 0.093  | 0.019 |
    | tie            | 252   | 4043   | 0.183  | 0.118 |
    | suitcase       | 299   | 1934   | 0.261  | 0.143 |
    | frisbee        | 115   | 566    | 0.539  | 0.393 |
    | skis           | 240   | 5994   | 0.133  | 0.046 |
    | snowboard      | 69    | 343    | 0.217  | 0.134 |
    | sports ball    | 260   | 4571   | 0.342  | 0.255 |
    | kite           | 327   | 7137   | 0.321  | 0.175 |
    | baseball bat   | 145   | 1113   | 0.172  | 0.067 |
    | baseball glove | 148   | 1161   | 0.257  | 0.117 |
    | skateboard     | 179   | 1872   | 0.358  | 0.237 |
    | surfboard      | 267   | 4453   | 0.270  | 0.156 |
    | tennis racket  | 225   | 2015   | 0.293  | 0.183 |
    | bottle         | 1013  | 7547   | 0.247  | 0.116 |
    | wine glass     | 341   | 1171   | 0.194  | 0.115 |
    | cup            | 895   | 4976   | 0.345  | 0.214 |
    | fork           | 215   | 1029   | 0.172  | 0.080 |
    | knife          | 325   | 1714   | 0.098  | 0.029 |
    | spoon          | 253   | 1237   | 0.119  | 0.046 |
    | bowl           | 623   | 3130   | 0.456  | 0.293 |
    | banana         | 370   | 7206   | 0.265  | 0.117 |
    | apple          | 236   | 2028   | 0.250  | 0.094 |
    | sandwich       | 177   | 1156   | 0.486  | 0.318 |
    | orange         | 285   | 2710   | 0.411  | 0.210 |
    | broccoli       | 312   | 13007  | 0.308  | 0.108 |
    | carrot         | 365   | 9667   | 0.216  | 0.064 |
    | hot dog        | 125   | 470    | 0.304  | 0.233 |
    | pizza          | 284   | 2390   | 0.521  | 0.434 |
    | donut          | 328   | 2525   | 0.463  | 0.293 |
    | cake           | 310   | 1742   | 0.374  | 0.199 |
    | chair          | 1771  | 33923  | 0.244  | 0.126 |
    | couch          | 261   | 1659   | 0.521  | 0.321 |
    | potted plant   | 342   | 12017  | 0.266  | 0.102 |
    | bed            | 163   | 1625   | 0.534  | 0.405 |
    | dining table   | 695   | 8174   | 0.397  | 0.230 |
    | toilet         | 179   | 3298   | 0.659  | 0.549 |
    | tv             | 288   | 2276   | 0.635  | 0.507 |
    | laptop         | 231   | 913    | 0.576  | 0.499 |
    | mouse          | 106   | 627    | 0.557  | 0.459 |
    | remote         | 283   | 2146   | 0.166  | 0.073 |
    | keyboard       | 153   | 1376   | 0.484  | 0.333 |
    | cell phone     | 262   | 2546   | 0.282  | 0.177 |
    | microwave      | 55    | 366    | 0.545  | 0.438 |
    | oven           | 143   | 3618   | 0.413  | 0.277 |
    | toaster        | 9     | 16     | 0.111  | 0.007 |
    | sink           | 225   | 8280   | 0.351  | 0.179 |
    | refrigerator   | 126   | 2632   | 0.556  | 0.424 |
    | book           | 1129  | 21431  | 0.117  | 0.024 |
    | clock          | 267   | 4601   | 0.468  | 0.339 |
    | vase           | 274   | 1612   | 0.270  | 0.167 |
    | scissors       | 36    | 256    | 0.250  | 0.189 |
    | teddy bear     | 190   | 2154   | 0.374  | 0.240 |
    | hair drier     | 11    | 1      | 0.000  | 0.000 |
    | toothbrush     | 57    | 326    | 0.070  | 0.040 |
    +----------------+-------+--------+--------+-------+
    | mAP            |       |        |        | 0.259 |
    +----------------+-------+--------+--------+-------+


    {'mAP': 0.2592655122280121}
COCO mAP and CustomDataset mAP don't match on single IoU value.
issue
created
ZwwWayne
username_1
The `customdataset` uses VOC-style AP calculation, its implementation when calculating TP and FPs are a little bit different from COCO evaluation.
comment
created
hellock
username_2
@username_0 If you use the official evaluation tool provided by COCO to evaluate the results of Pascal VOC dataset or ImageNet DET dataset, you will also get slightly different mAPs. The implementation of evaluation metrics of different datasets are not exactly the same.
comment
closed
hellock
username_2
issue