Package: nobunaga
Nobunaga: Object Detection AnalyzerNobunaga is an Object Detection Analysis tool in Computer Vision.
Based onTIDE, which is one of the excellent error analysis algorithms,
Nobunaga exports the overall distribution of detection error in the given dataset as well as per-category error.
Towards analyzing miss classification, Nobunaga plots their error in a confusion matrix.
Moreover, it plots detection errors in images one by one with annotation.
We hope it could be beneficial for you to figure out what kind of detection error happens in your models specifically.Installationpipinstallnobunagaorgitclonehttps://github.com/FastAccounting/nobunagacdnobunagapipinstall-e.Analize errorBefore analyzing detection error,
you need to prepare ground truth formated inCOCO object detectionand its corresponding prediction results in JSON and image dataset.Command line is as follows:nobunaga--predcoco_instances_results.json\--gtinstances_val.json\--image_dirpath/to/image_dir\--iou_threshold0.5\--confidence_threshold0.7Then, you can get the below filesError summary similar to TIDEPer-label detection errorClass error matrixError images with ground truth labelThe definition of errors follows TIDE.
However, we remark that Nobunaga counts the number of labels for each error, unlike TIDE.
Therefore, Nobunaga displays the total number of corresponding errors that occurred in the data set. The numbers are different from those in TIDE.Error SummaryThe summary of errors provides an overview of errors in the dataset, similar to TIDE.Per-label detection errorThis section delves into the Error Summary and shows which errors occur for each label in a figure.
The further to the right, the higher the number of errors.
For example, we confirm that the "Person" label has many Dupe errors, followed by Loc, Bkg, and Miss.Class error confusion matrixThe confusion matrix visualizes what label tends to be misclassified as other labels.
We remark that this confusion matrix visualizes only false examples, so correct ones are omitted.Error images with ground truth labelImages containing detection errors are written to the{model_name}_errordirectory for each error.
The pictures show one with the error on the left and another with the ground truth on the right.
Two examples are shown here.
The first image is an example of a Bkg error, which incorrectly detects the background area. The correct image has no annotations.
The second image is an example of a Miss error, indicating that a detection error has been occured.Explanation of each ErrorsClassification Error (Cls)localized correctly but classified incorrectly.
max(IoU) ≥ tf for GT of the incorrect class.Location Error (Loc)Detected by small bounding box below IoU threshold.
Classified correctly but localized incorrectly.
tb ≤ max(IoU) ≤ tf for GT of the correct classBoth Error (Both)Classified incorrectly and localized incorrectly.
tb ≤ max(IoU) ≤ tf for GT of the incorrect classDuplicate Error (Dupe)It would be correct if not for a higher scoring detection.
max(IoU) ≥ tf for GT of the correct class but another higher-scoring detection already matched that GTBackground Error (Bkg)Detected background as foreground.
max(IoU) ≤ tb for all GT.Miss Error (Miss)All undetected ground truths (false negatives) are not already covered by classification or localization error.
In Nobunaga, "not already covered by all other errors" define as Miss Error, and more labels tend to be explained as Miss Error more than other errors.