Getting started with COCO dataset | by Jakub Adamczyk | Towards Data Science

COCO (official website) dataset, meaning “Common Objects In Context”, is a set of challenging, high quality datasets for computer vision, mostly state-of-the-art neural networks. This name is also used to name a format used by those datasets.
Quoting COCO creators:
COCO is a large-scale object detection, segmentation, and captioning dataset. COCO has several features:
- Object segmentation
- Recognition in context
- Superpixel stuff segmentation
- 330K images (>200K labeled)
- 1.5 million object instances
- 80 object categories
Format of this dataset is automatically understood by advanced neural network libraries, e. g. Facebook’s Detectron2 (link). There are even tools built specifically to work with datasets in COCO format, e. g. COCO-annotator and COCOapi. Understanding how this dataset is represented will help with using and modifying the existing datasets and also with creating the custom ones. Specifically, we are interested in annotations files, since complete dataset consists of images directory and annotation file, providing metadata used by machine learning algorithms.
There are actually multiple COCO datasets, each one made for a specific machine learning task, with additional data. The 3 most popular tasks are:
object detection — model should get bounding boxes for objects, i. e. return list of object classes and coordinates of rectangles around them; objects (also called “things”) are discrete, separate objects, often with parts, like humans and cars; the official dataset for this task also contains additional data for object segmentation (see below)
object/instance segmentation — model should get not only bounding boxes for objects (instances/“things”), but also segmentation masks, i. e. coordinates of polygon closely around the object
stuff segmentation — model should do object segmentation, but not on separate objects (“things”), but on background continuous patterns like grass or sky
In computer vision, those tasks have tremendous usage, e. g. for self-driving vehicles (detection of people and other vehicles), AI-based security (human detection and/or segmentation) and object re-identification (object segmentation or removing background with stuff segmentation helps with checking object identity).
File format used by COCO annotations is JSON, which has dictionary (key-value pairs inside braces, {...}) as a top value. It can also have lists (ordered collections of items inside brackets, [...]) or dictionaries nested inside.
Basic structure is as follows:
{  "info": {...},  "licenses": [...],  "images": [...],  "categories": [...],  "annotations": [...]}
Let’s take a closer look at each one of those values.
This dictionary contains metadata about the dataset. For the official COCO datasets, it is as follows:
{  "description": "COCO 2017 Dataset",  "url": "http://cocodataset.org",  "version": "1.0",  "year": 2017,  "contributor": "COCO Consortium",  "date_created": "2017/09/01"}
As you can see, it contains only basic information, with "url" value pointing to the dataset official website (e. g. UCI repository page or in a separate domain). This is a common thing in machine learning datasets to point to their websites for additional information, e. g. how and when was data acquired.
Here are links to licenses for images in the dataset, e. g. Creative Commons licenses, with the following structure:
[  {    "url": "http://creativecommons.org/licenses/by-nc-sa/2.0/",     "id": 1,     "name": "Attribution-NonCommercial-ShareAlike License"  },  {    "url": "http://creativecommons.org/licenses/by-nc/2.0/",     "id": 2,     "name": "Attribution-NonCommercial License"  },  ...]
The important thing to note here is the "id" field — each image in "images" dictionary should have the ‘id’ of its licence specified.
When making use of images, make sure that you don’t violate its license — full text can be found under the URL.
If you decide to create your own dataset, assign the appropriate license to each image — if you’re not sure, it’s better not to use the image.
Arguably the second most important one, this dictionary contains metadata about the images:
{  "license": 3,  "file_name": "000000391895.jpg",  "coco_url": "http://images.cocodataset.org/train2017/000000391895.jpg",  "height": 360,  "width": 640,  "date_captured": "2013–11–14 11:18:45",  "flickr_url": "http://farm9.staticflickr.com/8186/8119368305_4e622c8349_z.jpg",  "id": 391895}
Let’s go through the fields one by one:
"license": the the ID of the image license from the "licenses" section
"file_name": the the name of the file in the images directory
"coco_url", "flickr_url": URLs to the online hosted image copy
"height", "width": size of image, very handy in low-level languages like C where getting size of matrix is impossible or hard
"date_captured": when the photo was taken
The most important field is the "id" field. This is the number that is used in "annotations" to identify the image, so if you wanted e. g. to identify the annotations for the given image file, you would have to check the "id" for the appropriate image document in "images", and then cross-reference it in "annotations".
In the official COCO dataset the "id" is the same as the "file_name" (after removing the leading zeros). Note that this may not necessarily be the case for custom COCO datasets! This is not an enforced rule, e. g. datasets made from private photos may have the original photo names which have nothing in common with "id".
This section a bit different for object detection and segmentation task and for stuff segmentation task.
Object detection / object segmentation:
[  {"supercategory": "person", "id": 1, "name": "person"},  {"supercategory": "vehicle", "id": 2, "name": "bicycle"},  {"supercategory": "vehicle", "id": 3, "name": "car"},  ...  {"supercategory": "indoor", "id": 90, "name": "toothbrush"}]
These are classes of objects that may be detected on images ("categories" in COCO are another name for classes, which you may know from supervised machine learning).
Each category has a unique "id" and they should be in range [1, number of categories]. Categories are also grouped in “supercategories”, which you may use in your programs e. g. to detect vehicles in general, when you don’t care whether it’s a bicycle, car or truck.
Stuff segmentation:
[  {"supercategory": "textile", "id": 92, "name": "banner"},  {"supercategory": "textile", "id": 93, "name": "blanket"},  ...  {"supercategory": "other", "id": 183, "name": "other"}]
Categories number start high to avoid conflicts with object segmentation, since sometimes those tasks can be performed together (so called Panoptic segmentation task, which also has a very challenging COCO dataset). IDs from 92 to 182 are actual background stuff, while ID 183 represents all other background textures that don’t have separate classes.
“annotations” section
This is the most important section of the dataset, which contains information vital for each task for specific COCO dataset.
{  "segmentation":  [[    239.97,    260.24,    222.04,    ...  ]],  "area": 2765.1486500000005,  "iscrowd": 0,  "image_id": 558840,  "bbox":  [    199.84,    200.46,    77.71,    70.88  ],  "category_id": 58,  "id": 156}
"segmentation": a list of segmentation mask pixels; this is a flattened list of pairs, so you should take the first and the second value (x and y at the picture), then the third and the fourth etc. to get coordinates; note that those are not image indices, since they are floating numbers — they are created and compressed by tools like COCO-annotator from raw pixel coordinates
"area": number of pixels inside segmentation mask
"iscrowd": whether the annotation is for a single object (value 0), or for the multiple objects close to each other (value 1); for stuff segmentation this field is always 0 and is ignored
"image_id": the ‘id’ field from the ‘images’ dictionary; warning: this value should be used to cross-reference the image with other dictionaries, not the "id" field!
"bbox": bounding box, i. e. the coordinates (top left x, top left y, width, height) of the rectangle around the object; it’s very useful to extract individual objects from images, since in many languages like Python it can be done by accessing the image array like cropped_object = image[bbox[0]:bbox[0] + bbox[2], bbox[1]:bbox[1] + bbox[3]]
"category_id": class of the object, corresponding to the "id" field in "categories"
"id": unique identifier for annotation; warning: this is only annotation ID, this does not point to the particular image in other dictionaries!
While working with crowd images ("iscrowd": 1), the "segmentation" part may be a little different:
"segmentation":{  "counts": [179,27,392,41,...,55,20],  "size": [426,640]}
This is because for a lot of pixels explicitly listing all pixels creating segmentation mask would take a lot of space. Instead, COCO uses custom Run-Length Encoding (RLE) compression, which is very efficient, since segmentation masks are binary and RLE for only 0s and 1s may decrease size many times.
We’ve explored the COCO dataset format for the most popular tasks: object detection, object segmentation and stuff segmentation. Official COCO datasets are high quality, large and suitable for beginner projects, production environment and state-of-the-art research. I hope this article helped you understand how to interpret this format and work with it for your ML applications.