How to work with object detection datasets in COCO format | by Eric Hofesmann | Towards Data Science

Microsoft's Common Objects in Context dataset (COCO) is the most popular object detection dataset at the moment. It is widely used to benchmark the performance of computer vision methods.
Due to the popularity of the dataset, the format that COCO uses to store annotations is often the go-to format when creating a new custom object detection dataset. While the COCO dataset also supports annotations for other tasks like segmentation, I will leave that to a future blog post. For now, we will focus only on object detection data.
The “COCO format” is a specific JSON structure dictating how labels and metadata are saved for an image dataset.
Many blog posts exist that describe the basic format of COCO, but they often lack detailed examples of loading and working with your COCO formatted data. This post will walk you through:
The COCO file format
Converting an existing dataset to COCO format
Loading a COCO dataset
Visualizing and exploring your dataset
Generating predictions from an object detection model
Evaluating mAP of a model on your COCO dataset
In order to do all of this, I’ll be using the open-source machine learning developer tool, FiftyOne, that I have been working on. It’s designed to let researchers and engineers easily work with and visualize image and video datasets with annotations and model predictions stored in various formats.
You can easily install FiftyOne through pip:
pip install fiftyone
As of 06/29/2021: With support from the COCO team, COCO has been integrated into FiftyOne to make it easy to download and evaluate on the dataset. You can now specify and download the exact subset of the dataset that you want, load your own COCO-formatted data into FiftyOne, and evaluate your models with COCO-style evaluation enhanced by the visualization capabilities of FiftyOne.
See this post or this documentation for more details!
If you are new to the object detection space and are tasked with creating a new object detection dataset, then following the COCO format is a good choice due to its relative simplicity and widespread usage. This section will explain what the file and folder structure of a COCO formatted object detection dataset actually looks like.
At a high level, the COCO format defines exactly how your annotations (bounding boxes, object classes, etc) and image metadata (like height, width, image sources, etc) are stored on disk.
The folder structure of a COCO dataset looks like this:
<dataset_dir>/    data/        <filename0>.<ext>        <filename1>.<ext>        ...    labels.json
The dataset is stored in a directory containing your raw image data and a single json file that contains all of the annotations, metadata, categories, and other information that you could possibly want to store about your dataset. If you have multiple splits of data, they would be stored in different directories with different json files.
If you were to download the COCO dataset from their website, this would be the instances_train2017.json and instances_val2017.json files. (Note: The official test set annotations are unavailable to the public)
{    "info": {        "year": "2021",        "version": "1.0",        "description": "Exported from FiftyOne",        "contributor": "Voxel51",        "url": "https://fiftyone.ai",        "date_created": "2021-01-19T09:48:27"    },    "licenses": [        {          "url": "http://creativecommons.org/licenses/by-nc-sa/2.0/",          "id": 1,          "name": "Attribution-NonCommercial-ShareAlike License"        },        ...       ],    "categories": [        ...        {            "id": 2,            "name": "cat",            "supercategory": "animal"        },        ...    ],    "images": [        {            "id": 0,            "license": 1,            "file_name": "<filename0>.<ext>",            "height": 480,            "width": 640,            "date_captured": null        },        ...    ],    "annotations": [        {            "id": 0,            "image_id": 0,            "category_id": 2,            "bbox": [260, 177, 231, 199],            "segmentation": [...],            "area": 45969,            "iscrowd": 0        },        ...    ]}
Info — Description and versioning information about your dataset.
Licenses — List of licenses with unique IDs to be specified by your images.
Categories — Classification categories each with a unique ID. Optionally associated with a supercategory that can span multiple classes. These categories can be whatever you want, but note that if you’d need to follow the COCO classes if you want to use a model pretrained on COCO out of the box (or follow other dataset categories to use other models).
Images — List of images in your dataset and relevant metadata including unique image ID, filepath, height, width, and optional attributes like license, URL, date captured, etc.
Annotations — List of annotations each with a unique ID and the image ID it relates to. This is where you will store the bounding box information in our case or segmentation/keypoint/other label information for other tasks. This also stores bounding box area and iscrowd indicating a large bounding box surrounding multiple objects of the same category which is used for evaluation.
This section will outline how to take your raw or annotated dataset and convert it to the COCO format depending on what data you currently have and the format it is in.
In this case, you already have a dataset with images and annotations but want to convert it to the COCO format.
If your dataset happens to follow a different common format that is supported by FiftyOne, like CVAT, YOLO, KITTI, Pascal VOC, TF Object detection, or others, then you can load and convert it to COCO format in a single command.
# Convert a COCO detection dataset to CVAT image formatfiftyone convert \    --input-dir /path/to/cvat-image-dataset \    --input-type fiftyone.types.CVATImageDataset \    --output-dir /path/to/coco-detection-dataset \    --output-type fiftyone.types.COCODetectionDataset
If your data is not stored in a supported format, then it is still easy to load it into FiftyOne using Python and export it in COCO format. The idea is to load each image and associated labels as a FiftyOne Sample and add them to a FiftyOne Dataset:
You can then export this dataset in COCO format with one line:
And there you have it! /path/to/coco-detection-dataset now contains your images and labels in COCO format. Check out the next section to see how to easily load it back into Python.
If you only have unlabeled images, then you will first need to generate object labels. You can generate either ground truth labels with an annotation tool or provider (like CVAT, Labelbox, MTurk, or one of many others) or predicted labels with an existing pretrained model.
If, for example, you used CVAT to annotate your raw data, then you can now convert it to COCO format using the FiftyOne command just like in the above section:
# Convert a COCO detection dataset to CVAT image formatfiftyone convert \    --input-dir /path/to/cvat-image-dataset \    --input-type fiftyone.types.CVATImageDataset \    --output-dir /path/to/coco-detection-dataset \    --output-type fiftyone.types.COCODetectionDataset
Alternatively, if you want to use a model to generate predictions, you can load your unlabeled data into FiftyOne and generate predictions with the FiftyOne Model Zoo, then save your dataset in COCO format.
This section assumes that you have gathered images and annotated them, storing your dataset in the COCO format, either following the previous section or manually building the labels JSON through custom scripting.
In order to load your COCO formatted dataset, you could write a parser for the JSON labels file, but really you should just use one of the various tools out there that will load it for you. Two of the best tools for this are the official COCO APIs and FiftyOne.
There are official COCO APIs for Python, Lua, and Matlab. These APIs are commonly used and provide basic functionality to load and compute dataset-wide evaluation on your dataset.
If you are using Python, I would recommend trying out FiftyOne, since it provides similar functionality to the cocoapi, along with a powerful API and GUI designed specifically to make it as easy as possible for you to explore, analyze, and work with your data.
If your dataset correctly follows the COCO format outlined in the previous sections, you can load it into a FiftyOne Dataset in Python with a single command:
Now that your dataset is in Python, you can use the FiftyOne API to easily access all of the different information and labels associated with your data and visualize it in the App.
To visualize your dataset, launch the FiftyOne App:
With the API, you can use aggregations to get statistics about your dataset, like the number of detections for each category:
The primary way of interacting with your dataset is through views. Every query you make will give you a different view into your dataset, like sorting by samples with the most number of objects:
You can also make a view that filters a label field based on a more complex value like small bounding box area:
The FiftyOne Brain contains various methods that allow you to analyze the quality of your ground truth data. For example, you can find the most unique samples in your dataset which can help you get a better idea of what kind of additional data you should add:
Other Brain methods can help you find possible annotation mistakes and identify hard samples you may want to train on. All of these will assist you in training better models since better models generally stem from better data.
The main reason that you want to create a COCO formatted dataset is to use it to train and test models.
Most models these days rely on your data being loaded into Python. Especially if you are using TensorFlow or PyTorch since these libraries are primarily Python-based. Using the COCO API or FiftyOne to get your dataset into Python makes it much easier to write up a PyTorch dataloader, for example, than if you had to parse the labels JSON yourself. Actually training a model on your data is out of the scope of this post but there are plenty of examples for both PyTorch object detection training and even a TensorFlow Object Detection API to help you along.
If you are just starting and want to see how some pretrained models would behave on your dataset, the easiest way to generate some predictions is with the FiftyOne Model Zoo. It contains over 70 models, many of which are object detection models.
fiftyone zoo models list
Note: If you’re tired of configuring TensorFlow/PyTorch models to use your GPU, check out my blog post on Conda.
Since your data is stored in COCO format, it can be loaded into FiftyOne, model predictions can be generated on it, and then visualized in the App:
The primary evaluation metric for object detection models is mean average precision (mAP). This is a fairly complex metric that is explained in more detail in other posts. In summary, it is computed by:
Matching predicted detections with ground truth objects if they overlap above some intersection-over-union (IoU) valueComputing the number of true positives, false positives, and false negatives for all detections of each classUsing these TP/FP/FN to generate a precision-recall curveComputing the average precision for each classTaking the mean of all of the classwise average precision values
Matching predicted detections with ground truth objects if they overlap above some intersection-over-union (IoU) value
Computing the number of true positives, false positives, and false negatives for all detections of each class
Using these TP/FP/FN to generate a precision-recall curve
Computing the average precision for each class
Taking the mean of all of the classwise average precision values
The COCO evaluation protocol introduces one additional step: mAPs are averaged across a range of 10 IoU thresholds. Additionally, COCO object detection evaluation also includes calculating the mAP for things like small, medium, and large bounding boxes, and varying thresholds of detections per image.
Note: When evaluating an object detection model, the categories of the predictions must match those of the dataset. This means you must either follow the labels of the COCO dataset to use an out-of-the-box model trained on COCO, or you must fine-tune a model on your dataset so it will predict your custom categories.
This COCO mAP value can be computed with either the COCO API or with FiftyOne. Assuming you have a dataset in COCO format with model predictions stored in the predictions field of the dataset, the following will compute the COCO mAP in FiftyOne:
Even though mAP is the most popular single value to compare model performances, this metric does have drawbacks. If you really want to know how well your model is performing, you need to dig into your data and look at model predictions on individual samples.
The best way to build intuition about how your model performs is by looking at predictions that it was confident about but got wrong. With FiftyOne, this is easy. For example, let’s create a view into our dataset looking at the samples with the most false positives:
The example above is in this view and shows a crowd of objects with the false positives not annotated in the ground truth! In the COCO format, ground truth objects can have an iscrowd attribute that specifies that the bounding box is drawn around a crowd of objects. This is one of many examples where this iscrowd box is either missing or incorrectly labeled resulting in false positives.
This is something that would have been impossible to find just by looking at the mAP of the model and shows the importance of sample-level analysis to understand dataset quality.
High-quality, intentionally-curated data is critical to training great computer vision models. At Voxel51, we have over 25 years of CV/ML experience and care deeply about enabling the community to bring their AI solutions to life. That’s why we developed FiftyOne, an open-source tool that helps engineers and scientists to build high-quality datasets and models.
Want to learn more? Check us out at fiftyone.ai.