How to Annotate and Improve Datasets with CVAT and FiftyOne | Towards Data Science

Everyone wants to train great models. The machine learning community has had a model-centric focus for quite a while with everyone trying to create the next best model. Recently, the importance of switching gears to data-centric workflows has become more and more prominent through ideas like Software 2.0. When training a model for your task, you will likely find that the biggest improvements to your model will come from carefully curating and improving the datasets you use to train your models rather than worrying about the specific model architecture you are using.
In the dataset curation and annotation space, FiftyOne and CVAT are two leading open-source tools tackling different parts of the dataset curation and improvement workflows each with tens of thousands of users. Moreover, these tools are now tightly integrated, allowing for faster and more efficient workflows. FiftyOne is a dataset curation and model analysis tool providing a flexible API and intuitive App serving as the open-source “IDE for your datasets.” CVAT is one of the leading open-source annotation tools for images and videos that has been widely adopted due to its broad feature set and easy-to-use interface. The integration between FiftyOne and CVAT allows you to curate and explore datasets in FiftyOne and then send off samples or existing labels for annotation in CVAT with just one line of code.
This post covers two example workflows showing how to use the integration between FiftyOne and CVAT. The first workflow intelligently selects a subset of unlabeled data and sends it to CVAT for annotation. The second workflow evaluates an existing dataset to find annotation mistakes and uses CVAT to improve the quality of the dataset. Both of these examples show how easy it is to work with FiftyOne and CVAT together for annotation workflows, leading to fine-tuned computer vision datasets that are needed to develop better models, faster.
Disclaimer: I am an engineer at Voxel51 working on FiftyOne!
The examples shown in the post can be run directly in your browser through Google Colab. Click this link to check it out!
In order to follow along with the examples in this post, you need to set up FiftyOne and CVAT.
For FiftyOne, you just need to install the open-source Python package:
pip install fiftyone
For CVAT, you need to make an account on cvat.org (or set up CVAT locally). The primary benefits of setting up CVAT locally is to avoid the 10 tasks and 500 Mb limits of cvat.org.
When running the example in this post, FiftyOne will need to connect to your CVAT account. The easiest way to pass your CVAT username and password to FfityOne is to store them in environment variables.
export FIFTYONE_CVAT_USERNAME=<YOUR_USERNAME>export FIFTYONE_CVAT_PASSWORD=<YOUR_PASSWORD>
If you do not want to use these environment variables, check out the alternate ways to provide your CVAT username and password to FiftyOne.
This post also runs some deep models that require TensorFlow:
pip install tensorflow
Check out this post if you have a GPU and/or want to set up Tensorflow or PyTorch with Conda instead.
The basic workflow for using this integration between FiftyOne and CVAT on your own data includes loading your data (and possibly existing labels) into FiftyOne, exploring your dataset to find subsets that need to be reviewed or annotated, uploading those subsets to CVAT, annotating them in CVAT, and loading the updated labels back into FiftyOne.
For most machine learning projects, the first step is to collect a dataset needed for a specific task. For computer vision projects specifically, this will generally result in thousands of images or videos that have been gathered from internet sources like Flickr or captured by new footage from a data acquisition team.
With collections containing thousands or millions of samples, the cost to annotate every single sample can be astronomical. It thus makes sense to ensure that only the most useful and relevant data is being sent to annotation. One measure for how “useful” data is in training a model is how unique the example is with respect to the rest of the dataset. Multiple similar examples will not provide the model with as much new information to learn as visually unique examples.
FiftyOne provides a set of methods to calculate dataset properties like sample hardness, label mistakenness, and, most importantly for this workflow, visual similarity/uniqueness.
We begin by loading a set of unlabeled images into FiftyOne. This can be done in just one line of code. For example, if you are using your own data you can run the following:
However, in this blog, we'll download some images from the Open Images V6 dataset using the built-in FiftyOne Dataset Zoo.
Either way, let’s make the dataset persistent so that we can close out of the Python session and load the dataset back in if we need to.
Now that the data is loaded, let’s visualize it in the FiftyOne App.
Let’s run the compute_similarity() method on the dataset in order to index all samples in the dataset by their visual similarity to all other samples in the dataset. Once this is done, we can then use the index to find the most unique samples based on visual similarity. There are other measures of uniqueness but we focus on this similarity-based measure in this blog.
We can also visualize the exact samples that were selected. By default, this visualization is computed with the dimensionality reduction package umap-learn:
pip install umap-learn
Now let’s create a view into the dataset containing only the unique samples that were selected and visualize them.
Having reduced the number of samples that need to be annotated, the time and cost of annotating this dataset have also been reduced.
Whether you are annotating the data yourself or have a team of annotators, the workflow of uploading data from FiftyOne to CVAT is nearly the same. The annotate() method on a collection of samples lets you specify the name, type, and classes for the labels you are annotating.
For example, if we are annotating segmentation masks for the classes “person,” “vehicle,” and “animal” we can run the following. Assuming you want to annotate these samples yourself, you can automatically launch a browser window with the CVAT editor once the data has been loaded. For this example post, let's only load a few of the samples in our view.
The results object can be used to get the current status of the tasks that were created.
Status for label field 'segmentations':	Task 386 (FiftyOne_example_dataset_segmentations):		Status: annotation		Assignee: None		Last updated: 2021-08-20T21:22:37.928988Z		URL: http://cvat.org/tasks/386		Job 441:			Status: annotation			Assignee: None			Reviewer: None
Once the annotation is complete and saved in CVAT, we can download the annotations and automatically update our dataset.
If you want to upload the dataset to a team of annotators, you can provide the CVAT usernames of the annotators and reviewers that will be assigned round-robin style. The segment_size parameter is used to define the maximum number of images in each job in your CVAT task.
For larger datasets, the annotation process may take some time. The anno_key that we provided stores the relevant information about this annotation run on the dataset itself. When the annotations are ready to be imported back into FiftyOne, we can easily do so:
In many projects, a dataset already exists and is being used to train models. In such cases, the best use of time is likely to improve the quality of the dataset, which often provides greater performance gains than a similar effort into optimizing the model architecture.
FiftyOne provides a powerful API and App workflows to identify the samples/annotations that need to be updated, and the tight integration with CVAT allows you to take the necessary actions to improve your dataset’s quality.
For this example workflow, let’s load the COCO object detection dataset from the FiftyOne Dataset Zoo.
In this case, we’ll use a model from the FiftyOne Model Zoo, but you can easily add your own model predictions to your dataset as well. To run this specific model, we also need to install the TensorFlow Model Garden with:
eta install models
Note: It is recommended to run this on a machine with a GPU!
Let’s visualize these model predictions.
In order to find specific cases of how this model performed, let’s evaluate the model:
0.3957238101325776
Using FiftyOne’s sophisticated query language, we can construct different views into the dataset. Specifically, let’s find the samples where the model was confident in its prediction but the prediction was labeled incorrect (false positive). This indicates that the model may have been correct but the ground truth annotation was incorrect.
Browsing through some of these results, we can see a pattern emerge. The COCO dataset contains a iscrowd boolean attribute on labels that indicates if the bounding boxes contain a crowd of multiple objects or just a single instance of the object. In many of the situations where the model was incorrect, the iscrowd attribute is incorrectly annotated or missing entirely.
We can tag some of these samples for annotation by clicking selecting the relevant samples and clicking the tag button.
We can now use the annotate() method to upload these samples and labels to CVAT for reannotation. The following code creates a new task in your account on cvat.org containing only the samples with the requires_annotation tag:
After updating the relevant annotations in all of the samples, make sure to hit the save button in CVAT. Now that the reannotation is complete, let’s load the updated labels back into FiftyOne and clean up the tasks that were created in CVAT.
As we can see, the ground_truth labels on the dataset have been updated. Let's evaluate the same model again on these updated labels.
0.3984999388520894
The mAP of the model has improved from 39.57% to 39.85% just by updating a single label in one sample! The next step is to spend more time exploring the dataset in FiftyOne finding more annotation mistakes and sending them to a team of annotators in CVAT for review and re-annotation. Specific users can be assigned to annotate or review the tasks that are created through this API.
This workflow has shown how to improve the validation split of a dataset and the subsequent performance of a model on that split. We never actually retrained the model at this point! The next step is to iterate this workflow over the entire dataset one split at a time until it is refined in its entirety. Then, when it’s time to train the model, other workflows demonstrating integrations with PyTorch dataloaders or PyTorch Lightning Flash are a great place to start.
Building a high-quality dataset is the most surefire way to produce high-performing models. Open-source tools for building and improving datasets are what allow the computer vision community as a whole to develop better models. FiftyOne and CVAT are two open-source tools that are now tightly integrated that you can use to curate and annotate datasets from scratch, as well as explore and re-annotate existing datasets to improve them.
Headquartered in Ann Arbor, Michigan, and founded in 2016 by University of Michigan professor Dr. Jason Corso and Dr. Brian Moore, Voxel51 is an AI software company that is democratizing access to software 2.0 by providing the open core software building blocks that enable computer vision and machine learning engineers to rapidly engineer data-powered workflows.
Learn more at fiftyone.ai!