How to work with object detection datasets in COCO format
By: Eric Hofesmann
Tags: Object Detection, Coco, Getting Started, Machine Learning, Computer Vision

How to work with object detection datasets in COCO format

Image 001298.jpg from the COCO dataset visualized in FiftyOne (Image by author)

Microsoft's Common Objects in Context dataset (COCO) is the most popular object detection dataset at the moment. It is widely used to benchmark the performance of computer vision methods.

Due to the popularity of the dataset, the format that COCO uses to store annotations is often the go-to format when creating a new custom object detection dataset. While the COCO dataset also supports annotations for other tasks like segmentation, I will leave that to a future blog post. For now, we will focus only on object detection data.

The “COCO format” is a specific JSON structure dictating how labels and metadata are saved for an image dataset.

Many blog posts exist that describe the basic format of COCO, but they often lack detailed examples of loading and working with your COCO formatted data. This post will walk you through:

In order to do all of this, I’ll be using the open-source machine learning developer tool, FiftyOne, that I have been working on. It’s designed to let researchers and engineers easily work with and visualize image and video datasets with annotations and model predictions stored in various formats.

You can easily install FiftyOne through pip:

pip install fiftyone

COCO 2017 validation split visualized in FiftyOne (Image by author)

Update: A new way to work with COCO

As of 06/29/2021: With support from the COCO team, COCO has been integrated into FiftyOne to make it easy to download and evaluate on the dataset. You can now specify and download the exact subset of the dataset that you want, load your own COCO-formatted data into FiftyOne, and evaluate your models with COCO-style evaluation enhanced by the visualization capabilities of FiftyOne.

See this post or this documentation for more details!

COCO file format

If you are new to the object detection space and are tasked with creating a new object detection dataset, then following the COCO format is a good choice due to its relative simplicity and widespread usage. This section will explain what the file and folder structure of a COCO formatted object detection dataset actually looks like.

At a high level, the COCO format defines exactly how your annotations (bounding boxes, object classes, etc) and image metadata (like height, width, image sources, etc) are stored on disk.

Files on disk

The folder structure of a COCO dataset looks like this:

<dataset_dir>/

data/

<filename0>.<ext>

<filename1>.<ext>

...

labels.json

The dataset is stored in a directory containing your raw image data and a single json file that contains all of the annotations, metadata, categories, and other information that you could possibly want to store about your dataset. If you have multiple splits of data, they would be stored in different directories with different json files.

JSON format

If you were to download the COCO dataset from their website, this would be the instances_train2017.json and instances_val2017.json files. (Note: The official test set annotations are unavailable to the public)



"info": {

"year": "2021",

"version": "1.0",

"description": "Exported from FiftyOne",

"contributor": "Voxel51",

"url": "https://fiftyone.ai",

"date_created": "2021-01-19T09:48:27"

},

"licenses": [

{

"url": "

"id": 1,

"name": "Attribution-NonCommercial-ShareAlike License"

},

...

],

"categories": [

...

{

"id": 2,

"name": "cat",

"supercategory": "animal"

},

...

],

"images": [

{

"id": 0,

"license": 1,

"file_name": "<filename0>.<ext>",

"height": 480,

"width": 640,

"date_captured": null

},

...

],

"annotations": [

{

"id": 0,

"image_id": 0,

"category_id": 2,

"bbox": [260, 177, 231, 199],

"segmentation": [...],

"area": 45969,

"iscrowd": 0

},

...

]

} "info": {"year": "2021","version": "1.0","description": "Exported from FiftyOne","contributor": "Voxel51","url": "https://fiftyone.ai","date_created": "2021-01-19T09:48:27"},"licenses": ["url": " http://creativecommons.org/licenses/by-nc-sa/2.0/ ","id": 1,"name": "Attribution-NonCommercial-ShareAlike License"},...],"categories": [..."id": 2,"name": "cat","supercategory": "animal"},...],"images": ["id": 0,"license": 1,"file_name": " . ","height": 480,"width": 640,"date_captured": null},...],"annotations": ["id": 0,"image_id": 0,"category_id": 2,"bbox": [260, 177, 231, 199],"segmentation": [...],"area": 45969,"iscrowd": 0},...

Info — Description and versioning information about your dataset.

— Description and versioning information about your dataset. Licenses — List of licenses with unique IDs to be specified by your images.

— List of licenses with unique IDs to be specified by your images. Categories — Classification categories each with a unique ID. Optionally associated with a supercategory that can span multiple classes. These categories can be whatever you want, but note that if you’d need to follow the COCO classes if you want to use a model pretrained on COCO out of the box (or follow other dataset categories to use other models).

— Classification categories each with a unique ID. Optionally associated with a supercategory that can span multiple classes. These categories can be whatever you want, but note that if you’d need to follow the COCO classes if you want to use a model pretrained on COCO out of the box (or follow other dataset categories to use other models). Images — List of images in your dataset and relevant metadata including unique image ID, filepath, height, width, and optional attributes like license, URL, date captured, etc.

— List of images in your dataset and relevant metadata including unique image ID, filepath, height, width, and optional attributes like license, URL, date captured, etc. Annotations — List of annotations each with a unique ID and the image ID it relates to. This is where you will store the bounding box information in our case or segmentation/keypoint/other label information for other tasks. This also stores bounding box area and iscrowd indicating a large bounding box surrounding multiple objects of the same category which is used for evaluation.

Creating a COCO format dataset

This section will outline how to take your raw or annotated dataset and convert it to the COCO format depending on what data you currently have and the format it is in.

Case 1: I have annotated data

In this case, you already have a dataset with images and annotations but want to convert it to the COCO format.

If your dataset happens to follow a different common format that is supported by FiftyOne, like CVAT, YOLO, KITTI, Pascal VOC, TF Object detection, or others, then you can load and convert it to COCO format in a single command.

# Convert a COCO detection dataset to CVAT image format

fiftyone convert \

--input-dir /path/to/cvat-image-dataset \

--input-type fiftyone.types.CVATImageDataset \

--output-dir /path/to/coco-detection-dataset \

--output-type fiftyone.types.COCODetectionDataset

If your data is not stored in a supported format, then it is still easy to load it into FiftyOne using Python and export it in COCO format. The idea is to load each image and associated labels as a FiftyOne Sample and add them to a FiftyOne Dataset:

You can then export this dataset in COCO format with one line:

And there you have it! /path/to/coco-detection-dataset now contains your images and labels in COCO format. Check out the next section to see how to easily load it back into Python.

Case 2: I only have raw image data

If you only have unlabeled images, then you will first need to generate object labels. You can generate either ground truth labels with an annotation tool or provider (like CVAT, Labelbox, MTurk, or one of many others) or predicted labels with an existing pretrained model.

If, for example, you used CVAT to annotate your raw data, then you can now convert it to COCO format using the FiftyOne command just like in the above section:

# Convert a COCO detection dataset to CVAT image format

fiftyone convert \

--input-dir /path/to/cvat-image-dataset \

--input-type fiftyone.types.CVATImageDataset \

--output-dir /path/to/coco-detection-dataset \

--output-type fiftyone.types.COCODetectionDataset

Alternatively, if you want to use a model to generate predictions, you can load your unlabeled data into FiftyOne and generate predictions with the FiftyOne Model Zoo, then save your dataset in COCO format.

Loading a COCO dataset into Python

This section assumes that you have gathered images and annotated them, storing your dataset in the COCO format, either following the previous section or manually building the labels JSON through custom scripting.

In order to load your COCO formatted dataset, you could write a parser for the JSON labels file, but really you should just use one of the various tools out there that will load it for you. Two of the best tools for this are the official COCO APIs and FiftyOne.

There are official COCO APIs for Python, Lua, and Matlab. These APIs are commonly used and provide basic functionality to load and compute dataset-wide evaluation on your dataset.

If you are using Python, I would recommend trying out FiftyOne, since it provides similar functionality to the cocoapi, along with a powerful API and GUI designed specifically to make it as easy as possible for you to explore, analyze, and work with your data.

If your dataset correctly follows the COCO format outlined in the previous sections, you can load it into a FiftyOne Dataset in Python with a single command:

Visualizing and exploring

Now that your dataset is in Python, you can use the FiftyOne API to easily access all of the different information and labels associated with your data and visualize it in the App.

To visualize your dataset, launch the FiftyOne App:

COCO 2017 validation split visualized in FiftyOne (Image by author)

With the API, you can use aggregations to get statistics about your dataset, like the number of detections for each category:

The primary way of interacting with your dataset is through views. Every query you make will give you a different view into your dataset, like sorting by samples with the most number of objects:

Samples with most objects visualized in FiftyOne (Image by author)

You can also make a view that filters a label field based on a more complex value like small bounding box area:

Small object view visualized in FiftyOne (Image by author)

The FiftyOne Brain contains various methods that allow you to analyze the quality of your ground truth data. For example, you can find the most unique samples in your dataset which can help you get a better idea of what kind of additional data you should add: