Title: DETR Breakdown Part 2: Methodologies and Algorithms - PyImageSearch

URL Source: https://pyimagesearch.com/2023/06/12/detr-breakdown-part-2-methodologies-and-algorithms/

Published Time: 2023-06-12T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [DETR Breakdown Part 2: Methodologies and Algorithms](#h2BPTitle)

*   [The DETR Model 👁️](#h3DETR)

*   [Object Detection Set Prediction Loss 📉](#h2Loss)

*   [Optimal Bipartite Matching 🔄](#h3Bipartite)
*   [Optimal Bipartite Matching for Objects 🌐](#h3Objects)
*   [Optimize Object Specific Losses 🔧](#h3Optimize)

*   [Quiz Time! 🤓](#h2Quiz)
*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, we’ll learn about the methodologies applied in **DETR**.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/06/detr-2-featured.png)

This lesson is the 2nd of a 3-part series on **DETR Breakdown**:

1.  [**_DETR Breakdown Part 1: Introduction to DEtection TRansformers_**](https://pyimg.co/fhm45)
2.  [**_DETR Breakdown Part 2: Methodologies and Algorithms_**](https://pyimg.co/slx2k) **(this tutorial)**
3.  _DETR Breakdown Part 3: Architecture and Details_

**To learn about the methodologies applied in Detection Transformers,** **_just keep reading._**

* * *

[**DETR Breakdown Part 2: Methodologies and Algorithms**](#TOC)
---------------------------------------------------------------

Welcome back to Part 2 of this tutorial series on Detection Transformers. In the previous tutorial [DETR Breakdown Part 1: Introduction to DEtection TRansformers](https://pyimg.co/fhm45), we looked at what factors led to the birth of DETR, what components were added, and what really is the Chemical X that made DETR into the super object detector it is today.

As scientists, learners, and builders, we must seek to understand what causes excellent performances. One of the key components of DETR is a secret sauce called Set Prediction Loss. Today we place this “Set Prediction Loss” under the microscope.

* * *

### [**The DETR Model 👁️**](#TOC)

First, let us revisit the DETR Model architecture in the interactive diagram below.

Ironically, we will not be demystifying the above architecture in this tutorial. Set Prediction Loss comes in where this architecture ends and outputs predictions. The predictions are matched with the ground truth objects to yield a loss. The model iterates on that and gets better and better and eventually at par with past models like Faster R-CNN.

How? Just keep reading to find out!

* * *

[**Object Detection Set Prediction Loss 📉**](#TOC)
---------------------------------------------------

DETR infers a **fixed-size** set of ![Image 2: N](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2011'%3E%3C/svg%3E) predictions. Note that ![Image 3: N](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2011'%3E%3C/svg%3E) is significantly larger than the typical number of objects in an image.

One of the main difficulties of training is to score predicted objects with respect to the ground truth. Therefore, [Carion et al. (2020)](https://arxiv.org/abs/2005.12872) propose the following algorithm.

1.  A loss that produces an **optimal** bipartite matching between _predicted_ and _ground truth_ objects
2.  Optimize the object-specific (bounding box) losses

* * *

### [**Optimal Bipartite Matching 🔄**](#TOC)

Here let us pause for a moment to understand what is **bipartite matching**.

> Bipartite matching is a mathematical concept from graph theory, often used in computer science and optimization problems. It refers to the process of finding a subset of edges in a bipartite graph such that each vertex in the graph is incident to at most one edge in the subset and the size of the subset is maximized. This subset of edges is called a maximum cardinality matching or simply a maximum matching.
> 
> [L.R. Ford and D.R. Fulkerson, “Maximal Flow Through a Network,” _Canadian Journal of Mathematics_, vol. 8, pp. 399-404, 1956](https://doi.org/10.4153/CJM-1956-045-5).

A bipartite _graph_ is a special type whose vertices can be divided into two disjoint sets such that all edges connect a vertex from one set to another. In other words, no edges connect vertices within the same set.

Bipartite _matching_ is the process of pairing vertices from the two sets so that each vertex is paired with at most one vertex from the other set, and the total number of paired vertices is maximized.

Think of it as finding the best way to match items from two categories, like connecting workers to jobs or students to projects. Each item from one category is paired with at most one item from the other category, and the cost of the pairs is optimal. This is shown in the demo below.

* * *

### [**Optimal Bipartite Matching for Objects 🌐**](#TOC)

Before we go into how the above concept of bipartite matching can be applied to ground truth and predicted objects, let us first familiarize ourselves with the _terminologies_ and _nomenclature_.

The process of finding matching pairs is similar to the methods used in previous object detectors for connecting proposed or anchor boxes to actual objects in an image. The main difference is that, for direct set prediction, we need to find unique one-to-one matches without any duplicates. Let’s look at the demo below to understand the pointers better.

We already know that ![Image 4: N](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2011'%3E%3C/svg%3E) is much larger than an image’s actual set of objects. So, to align the ground truth with predictions, we pad ground truth with ![Image 5: \phi](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2014'%3E%3C/svg%3E) (no objects token).

The task now is to find an **optimal** bipartite matching between these two sets: ground truth and prediction.

Let ![Image 6: \mathfrak{S}_{N}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2025%2016'%3E%3C/svg%3E) be a set of all the possible permutations of ![Image 7: N](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2015%2011'%3E%3C/svg%3E). If ![Image 8: N = 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2045%2012'%3E%3C/svg%3E), ![Image 9: \mathfrak{S}_{N}={ {1, 2}, {2, 1}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20100%2016'%3E%3C/svg%3E). This means that our ground truth set and the prediction set have two elements each. To find the optimal bipartite matching between the two sets, we search for a specific permutation (order) of the predictions, which gives the lowest cost when matched with the ground truth.

This specific order is defined by ![Image 10: \hat{\sigma}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2013'%3E%3C/svg%3E), where ![Image 11: \sigma \in \mathfrak{S}_{N}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2055%2016'%3E%3C/svg%3E). First, let’s define the cost function to find the optimal permutation (![Image 12: \hat{\sigma}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2013'%3E%3C/svg%3E)).

[![Image 13](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20498%20144'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/06/image.png)

Let us see an example, ![Image 14: N=2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2045%2012'%3E%3C/svg%3E) the cost function would then look something like this:

We will now choose the ![Image 15: \sigma](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) that gives us the lowest cost. Upon selecting the permutation that lowers the cost, we eventually get the **optimal** bipartite matching of the ground truth and the predicted objects. Let’s see this in action in the video below.

* * *

### [**Optimize Object Specific Losses 🔧**](#TOC)

Now that we have matched the predictions to the ground truth, we apply the Hungarian loss for all matched pairs.

[![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%2094'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/06/image-1.png)

Notice the usage of ![Image 17: \hat{\sigma}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2013'%3E%3C/svg%3E) here that denotes the already optimal permutation for the predictions.

Think of this as the only set of lines in the above diagram that will give you the lowest cost. For example, this is shown in the video below.

**_Note from the paper:_** In practice, we down-weight the log-probability term when ![Image 18: c_{i} = \phi](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2044%2014'%3E%3C/svg%3E) by a factor of 10 to account for class imbalance. In the matching cost, we use probabilities instead of log-probabilities. This makes the class prediction term commensurable to ![Image 19: \mathcal{L}_{\text{box}}(.,.)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2059%2018'%3E%3C/svg%3E), and we observed better empirical performances.

The L1 loss is commonly used in object detection to measure the difference between the predicted box coordinates and the ground-truth box coordinates. However, this loss can cause issues when dealing with boxes of different sizes.

For instance, consider two boxes with the same relative error (i.e., the same proportion of error compared to the box size) but different sizes. The larger box will have a larger absolute error, which could lead the model to prioritize larger boxes over smaller ones.

A combination of L1 loss and a scale-invariant loss called generalized Intersection over Union (IoU) can address this issue.

The generalized IoU loss measures the overlap between the predicted box and the ground-truth box, considering their sizes. The full expression is shown in the video below.

Using a linear combination of L1 loss and IoU loss, the model can balance the importance of accurately predicting box coordinates and their relative sizes, leading to more consistent performance across boxes of different sizes.

* * *

[**Quiz Time! 🤓**](#TOC)
-------------------------

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

As we know by now, DETR has two main components — pièce de résistance, if you will: Set Prediction Loss and Transformer Architecture. In this tutorial, we primarily focused on Set Prediction Loss.

*   First, we familiarized ourselves with Object Detection Set Prediction Loss.
*   Next, we took a detour to comprehend the workings of Optimal Bipartite Matching.
*   Subsequently, we explored how Optimal Bipartite Matching applies to objects.
*   Finally, we delved into the mathematics behind optimizing object-specific losses.

In the upcoming and final part of this series, we will reflect on the architecture and intricacies of DETR. We will examine its building blocks, connections, and what makes this model so efficient.

Stay tuned for the upcoming parts of this series, and be sure to share your thoughts and learnings from DETR and what you’re looking forward to. Connect with us on Twitter by tagging [@pyimagesearch](https://twitter.com/PyImageSearch).

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty and R. Raha.** “DETR Breakdown Part 2: Introduction to DEtection TRansformers,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2023, [https://pyimg.co/slx2k](https://pyimg.co/slx2k)

@incollection{ARG-RR\_2023\_DETR-breakdown-part2,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {{DETR} Breakdown Part 2: Introduction to DEtection TRansformers},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2023},
  url = {https://pyimg.co/slx2k},
}

* * *

![Image 20: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 21](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.