How to get average precision for individual class?

username: iamexperimentingnow
cooked: <p>Hi, I’m trying to get the individual class average precision.</p>
<p>Currently, I have trained object detection model using torchvision</p>
<pre><code class="lang-auto">num_classes = 3 # car, person, background
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
</code></pre>
<p>here, when I evaluate the model, i’m getting overall performance of the model.</p>
<pre><code class="lang-auto">IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.657
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.930
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.751
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.512
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.697
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.489
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.710
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.710
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.704
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.734
</code></pre>
<p>but, I’m trying to get individual class performance.<br>
can anyone help me to get average precision for individual category?</p>
<p>I know, detectron2 produces the individual class performance, but for my dataset detectron2 taking much longer time to get trained. So, I’m sticking with torchvision.</p>
<p>Also, I have tried other approach by clubbing(torchvision + Detectron2) them together. eg,</p>
<pre><code class="lang-auto">cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml"))
cfg.DATASETS.TRAIN = ('dataset_train',)
cfg.DATASETS.TEST = ('dataset_val',)   
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.LR_SCHEDULER_NAME = "WarmupCosineLR"
cfg.SOLVER.MAX_ITER = 500
cfg.SOLVER.BASE_LR = 0.005
cfg.SOLVER.MOMENTUM = 0.9
cfg.SOLVER.NESTEROV = False
cfg.SOLVER.STEPS = (400,450)
cfg.SOLVER.WEIGHT_DECAY = 0.0001
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 2
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.TEST.EVAL_PERIOD = 50
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SEED = 42
cfg.VIS_PERIOD = 100
cfg.SOLVER.CHECKPOINT_PERIOD = 500

trainer = MyTrainer(cfg)

from detectron2.config import get_cfg
from detectron2.data import DatasetCatalog, MetadataCatalog, build_detection_test_loader
from detectron2.evaluation import COCOEvaluator, inference_on_dataset
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "./outputs/fasterrcnn_custom_test_1_loss_0.12796187750063837.pth") # this model is trained using torchvision
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.85
trainer = MyTrainer(cfg)
predictor = DefaultPredictor(cfg)
evaluator = COCOEvaluator("dataset_val", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "dataset_val")
inference_on_dataset(trainer.model, val_loader, evaluator)
</code></pre>
<p>when I performed this, I’m getting below answer,</p>
<pre><code class="lang-auto">[11/20 00:19:51 d2.evaluation.coco_evaluation]: Per-category bbox AP: 
| category   | AP    | category   | AP    |
|:-----------|:------|:-----------|:------|
| car        | 0.000 | person     | 0.000 |
OrderedDict([('bbox',
              {'AP': 0.0,
               'AP50': 0.0,
               'AP75': 0.0,
               'APs': 0.0,
               'APm': 0.0,
               'APl': 0.0,
               'AP-car': 0.0,
               'AP-person': 0.0})])
</code></pre>
<p>Also I noted this,</p>
<pre><code class="lang-auto">Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (12, 1024) in the checkpoint but (8, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (12,) in the checkpoint but (8,) in the model! You might want to double check if this is expected.
Some model parameters or buffers are not found in the checkpoint:
</code></pre>
<p>it would be really helpful you anyone can guide me here.</p>
<p>Note: I have tried different github repo’s to get this answer, but I’m trying this approach.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: sidnetopia
cooked: <p>Hello, did you solve this?</p>

username: iamexperimentingnow
cooked: <p><a class="mention" href="/u/sidnetopia">@sidnetopia</a> I have used custom model with custom metrics.</p>

username: Jack_Taylor
cooked: <p>Hi there, how did you find out that the results of the model were an average across the classes?</p>

username: Jack_Taylor
cooked: <p>Never mind, I found this from the COCO website <a href="https://cocodataset.org/#detection-eval" class="inline-onebox" rel="noopener nofollow ugc">COCO - Common Objects in Context</a></p>
<p>“AP is averaged over all categories. Traditionally, this is called “mean average precision” (mAP). We make no distinction between AP and mAP (and likewise AR and mAR) and assume the difference is clear from context.”</p>