Google’s Open Images: Now easier to download and evaluate with FiftyOne | by Eric Hofesmann | Towards Data Science

Google’s Open Images is a behemoth of a dataset. With over 9 million images, 80 million annotations, and 600 classes spanning multiple tasks, it stands to be one of the leading datasets in the computer vision community.
Working with Open Images is now easier than ever with the its integration into the open-source ML developer tool, FiftyOne.
Computer vision models have become more data-hungry than ever before. Developing your own model requires you to either collect and annotate massive amounts of data yourself or turn to public datasets hoping that they contain relevant data for your task. The team behind Open Images set out to provide a massive data lake for anyone to access. By only curating images under a Creative Commons Attribution (CC-BY) license, Open Images can be used by anyone who is interested.
With image-level labels, segmentations, visual relationships, localized narratives, and 15x more object detections than the next largest detection dataset, Open Images can be tempting to add to your data lake and training workflows. However, learning and incorporating a new dataset format into your workflow is often tedious and time-consuming. Previously, the procedure to download Open Images consisted of writing custom scripts to download and parse the images and labels you want. This has spawned multiple tools to aid in loading the dataset, but they each have serious drawbacks like not supporting segmentations and relationships, no visualization capabilities, and not being able to specify the download format.
At Voxel51, we have collaborated with Google to create an easy-to-use source for downloading Open Images by incorporating it into the Dataset Zoo of our open-source ML tool, FiftyOne. With FiftyOne, you can specify exactly the subset of Open Images you want to download, export it into dozens of different formats, visualize it in the FiftyOne App, and even evaluate your models with Open Images-style object detection evaluation.
This blog post walks you through how to use FiftyOne to:
Download Open Images
Export Open Images in a different dataset format
Evaluate an object detection model with Open Images-style evaluation
Before we begin, make sure to install FiftyOne:
pip install fiftyone
You can run the examples in this blog post directly in your browser in this Google Colab notebook. Follow the link and click “Run in Google Colab”.
It now only takes one command or one line of Python code to download exactly the samples you want from Open Images. You can then explore the dataset in the FiftyOne App with one more line of code.
The integration of Open Images into FiftyOne provides multiple parameters that you can use to specify exactly which and how many samples you want to download:
label_types: a list of types of labels to load. Values are ("detections", "classifications", "relationships", "segmentations"). By default, all labels are loaded but not every sample will include each label type. If max_samples and label_types are both specified, then every sample will include the specified label types.
split and splits: either a string or list of strings dictating the splits to load. Available splits are ("test", "train", "validation").
classes: a list of strings specifying required classes to load. Only samples containing at least one instance of a specified class will be downloaded.
attrs: a list of strings for relationship attributes to load. This parameter is only useful if "relationships" is in label_types.
max_samples: a maximum number of samples to import. By default, all samples are imported.
shuffle: boolean dictating whether to randomly shuffle the order in which the samples are imported.
seed: a random seed to use when shuffling.
image_ids: a list of specific image IDs to load. The IDs can be specified either as <split>/<image-id> or <image-id>
image_ids_file: the path to a newline-separated .txt, .json, or .csv file containing a list of image IDs to load. The IDs can be specified either as <split>/<image-id> or <image-id>. If image_ids is provided, this parameter is ignored
Some of these parameters allow you to specify specific classes and attributes that you want in your subset. You can use the following commands to see the list of options available in Open Images:
Putting this functionality to use, we can curate datasets for specific tasks without needing to download the entire dataset or parse any raw annotation files. For example, if we wanted to train a detector for wooden objects, we can specify that we only want samples from Open Images containing an object with the visual relationship annotated as Wooden:
As another example, let’s build a dataset for detecting cats and dogs. We can get both detections as well as the image-level classifications for the classes Dog and Cat:
When working with a dataset containing object detections, FiftyOne lets you create a patches view wherein every sample contains a single object. This can be extremely useful when attempting to analyze specific object classes or attributes, especially in cluttered scenes.
For years, the COCO dataset has been the most prominent object detection dataset resulting in a sizable percentage of the computer vision (CV) community adopting the COCO format for their object detection problems. Choosing a dataset format for a project dictates the parsers needed to load and work with your dataset, making it difficult to work with new data in a different format. FiftyOne lets you export datasets into dozens of different formats letting you integrate it easily into existing workflows.
For example, you can export into formats like COCO, Pascal VOC, CVAT, YOLO, TFRecords, directory trees for classifications, and more. Many of these formats also support exporting segmentation masks.
In practice, exporting the subset of Open Images that we loaded previously to disk in COCO format is as easy as:
If you then want to import your data back into FiftyOne and the dataset follows one of the supported formats, that can also be done in one line of code:
Even if your dataset is in a completely custom format, it is still easy to load it into FiftyOne.
FiftyOne not only makes it easy to load and export Open Images and custom datasets, but it also lets you visualize your data and evaluate model results.
Open Images-style object detection evaluation was created for the Open Images challenges. It differs from COCO-style evaluation in a few notable ways:
Image-level labels dictate which object classes are evaluated in each image. If a class was predicted but not annotated as a positive or negative image-level label, it is ignored.
A semantic class hierarchy is used to expand predicted and ground truth object lists to include and evaluate all parent classes. By default, the ground truth detections only include leaf classes and must be expanded before evaluation takes place. This is performed automatically in FiftyOne.
Like COCO, “crowd” ground truth detections are able to be matched with multiple predictions. But unlike COCO, only one prediction will count as a true positive per crowd.
COCO-style evaluation averages AP over a sweep of 10 different IoU values, Open Images-style evaluation only uses an IoU of 0.5 to compute mAP.
Click these links if you want to see the detailed steps for the evaluation protocol for Open Images-style evaluation or COCO-style evaluation.
In order to try out Open Images-style evaluation, we can use a model from the FiftyOne Model Zoo to generate detection predictions on the cat and dog subset of Open Images we downloaded. We will add a new predictions field to the dataset:
Note: Running this model requires TensorFlow 1.14.
pip install tensorflow==1.14
For this example, we are only interested in predictions for the classes “Cat” and “Dog”, so we can utilize the flexible dataset representations of FiftyOne to create a view into the dataset containing only those classes. Additionally, the model we ran was trained on COCO which uses lower-case class names so we will need to remap them to match the upper-case class names of Open Images:
We can now run Open Images-style evaluation on our dataset and predictions:
The results object that is returned contains everything we need from mAP, to PR curves, to confusion matrices.
It should be noted that by default, these detection protocols will only match predicted objects to ground truth objects of the same class. That means that if we want an interesting confusion matrix, we will need to evaluate with the parameter classwise equal to False :
We can now view an interactive confusion matrix where if we click on a cell, the FiftyOne App will be updated showing only samples existing in that cell. So, for example, if we click on the cell containing ground truth “Dog” detections matched with predicted “Cat” detections, we can gain insight into the failure modes of our model:
Note: Interactive plots are currently only supported in Jupyter Notebooks. Support for other environments will be added soon.
When evaluating an object detection dataset in FiftyOne, after running evaluate_detections and storing the results on the dataset with an eval_key , we are then able to create what is called an evaluation view. This is a view into the dataset where every sample is an individual ground truth or predicted object patch allowing you to sort by true positives, false positives, and false negatives, the IoU of matches, and other evaluation metadata at the click of a button.
If we want to get an even more nuanced understanding of our model, it can be useful to visualize object embeddings in a low-dimensional space allowing us to quickly pick out patterns, clusters, and anomalies in our data. The FiftyOne Brain provides exactly this kind of workflow.
Note: The default dimensionality reduction method used by the Brain is umap which we need to install:
pip install umap-learn
We can again use the FiftyOne Model Zoo to generate embeddings that we will then visualize:
This kind of visualization can be invaluable for a multitude of reasons, but especially for datasets like Open Images that contain machine-generated labels. Visualizing and interactively exploring embeddings lets you quickly spot-check which machine-generated labels may need to be reviewed and sent off for reannotation.
We can also use these embeddings to automatically find similar and duplicate samples with the click of a button.
Once we have thoroughly explored our dataset and found aspects of it that we want to analyze further or improve, it is easy to tag individual samples or labels in order to isolate them for downstream use cases.
Open Images is a massive yet thoroughly labeled dataset that can make a useful addition to your data lake and model training workflows. The easiest way to download and explore Open Images is with FiftyOne. With datasets as large as Open Images, hands-on evaluation of your model results can be difficult. FiftyOne makes it easy to understand your dataset, find failure modes in your model, and reveal patterns in your data by visualizing embeddings.
Headquartered in Ann Arbor, Michigan, and founded in 2016 by Dr. Jason Corso and Dr. Brian Moore, Voxel51 is an AI software company that is democratizing access to software 2.0 by providing the open core software building blocks that enable computer vision and machine learning engineers to rapidly engineer data-powered workflows.
Want to learn more? Check us out at fiftyone.ai.