Title: DETR Breakdown Part 1: Introduction to DEtection TRansformers - PyImageSearch

URL Source: https://pyimagesearch.com/2023/05/22/detr-breakdown-part-1-introduction-to-detection-transformers/

Published Time: 2023-05-22T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [DETR Breakdown Part 1: Introduction to DEtection TRansformers](#h2BPTitle)

*   [Why DETR and What’s So Special? 🤔](#h3Special)
*   [What Is Object Detection? 😯](#h3What)
*   [So What’s New in DETR? 🎁](#h3New)
*   [Life Is Not a Bed of Roses 🌹: Past Challenges](#h3Roses)
*   [The Key Players 🤾: Main Components of DETR](#h3Players)
*   [In a Nutshell 🔩](#h3Nutshell)

*   [The Problem Statement Reimagined 🧠](#h2Statement)

*   [Object Detection from the Old Days 🧓](#h3Days)
*   [Set Prediction for Object Detection 🕵️](#h3Prediction)
*   [Transformers and Parallel Decoding for Object Detection ⚡](#h3Decoding)
*   [A Niche Solution 🥳](#h3Niche)

*   [Quiz Time! 🤓](#h2Quiz)
*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, we’ll learn about **DETR**, an **end-to-end** trainable deep learning architecture for **object detection** that utilizes a **transformer** block.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20575'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/05/detr-part1-featured.png)

This lesson is the 1st of a 3-part series on **DEtection TRansformers (DETR) Explained: A Comprehensive Guide to DETR in Computer Vision**:

1.  [**_DETR Breakdown Part 1: Introduction to DEtection TRansformers_**](https://pyimg.co/fhm45) **(this tutorial)**
2.  _DETR Breakdown Part 2: Methodologies and Algorithms_
3.  _DETR Breakdown Part 3: Architecture and Details_

**To learn about Detection Transformers, _just keep reading._**

* * *

[**DETR Breakdown Part 1: Introduction to DEtection TRansformers**](#TOC)
-------------------------------------------------------------------------

In this blog post, we explore the revolution in object detection with [DETR](https://arxiv.org/abs/2005.12872) (the entire architecture is presented in the interactive Figure shown below), a unique approach employing Transformers and set prediction for parallel decoding that reimagines the problem statement, bringing an alternative to traditional methods.

We also uncover how [Carion et al. (2020)](https://doi.org/10.48550/arXiv.2005.12872) present a niche solution that transcends the old days of object detection.

As we progress through the tutorial series, we aim to understand the components of the above interactive Figure. But, for now, let us focus on the topics at hand:

*   why DETR
*   where it came from
*   what’s the big deal about it

* * *

### [**Why DETR and What’s So Special? 🤔**](#TOC)

In a world plagued by hand-crafted features and time-consuming processes, a new architecture emerged from the lab: DETR, the end-to-end object detector that could battle the almighty CNNs in its own field! By combining the strengths of two unlikely parents CNNs and transformers, DETR streamlined the training process, bidding farewell to laborious manual work.

With its unique superpower of attention mechanisms, DETR could identify objects and their relationships with remarkable accuracy.

As news of this extraordinary architecture spread, labs turned to newer possibilities inspired by components from DETR, leading to better image segmentation and object detection models. And so, the legacy of DETR, the end-to-end super object detector, was born.

Today we shall delve into its past, the factors that led to its birth, and how this miracle was even achieved in the first place.

* * *

### [**What Is Object Detection? 😯**](#TOC)

Alright, so object detection is basically guessing where objects are in an image by drawing boxes around them and figuring out what each object is called.

Previously object detection methods solved the problem of predicting object locations in an image by creating **related sub-problems**. These sub-problems involve estimating object properties using the image’s numerous predefined regions, points, or centers.

* * *

### [**So What’s New in DETR? 🎁**](#TOC)

The authors want us to see that object detection is pretty straightforward, but they also point out that people used to tackle it more roundaboutly.

The authors view object detection as a **set prediction problem**. A set prediction problem is when you try to guess a group of items based on some information. Think of it like trying to figure out which movies your friends might like based on the movies they’ve already watched.

* * *

### [**Life Is Not a Bed of Roses 🌹: Past Challenges**](#TOC)

**Just a heads up:** Thinking of object detection as a set prediction problem has some challenges. The biggest one is getting _rid of duplicate predictions_. Keep this in mind as you go through the rest of the tutorial. It’ll help you better understand why the authors made certain choices and decisions.

* * *

### [**The Key Players 🤾: Main Components of DETR**](#TOC)

Their paper primarily consists of **two** main components:

1.  A unique set-based global loss, which utilizes bipartite matching (an algorithm that facilitates set prediction) to encourage distinct set predictions.
2.  A transformer-based architecture, which includes both an encoder and a decoder.

* * *

### [**In a Nutshell 🔩**](#TOC)

By treating object detection as a set prediction problem, the need for manually designed components previously required in object detection tasks to incorporate prior knowledge is eliminated. This approach simplifies the process and streamlines the task.

* * *

[**The Problem Statement Reimagined 🧠**](#TOC)
-----------------------------------------------

It is important to understand the contribution of this paper in the field of Computer Vision. To make sense of this, we first **reimagine** the **problem statement** by understanding object detection as it was in the old days. Next, we look at the two new **methodologies** that the authors proposed to solve **End-to-End Object Detection**. And finally, we look at the simple solution provided by the authors.

1.  Object Detection from the Old Days (problem statement)
2.  Set Prediction for Object Detection (methodology 1)
3.  Transformers and Parallel Decoding for Object Detection (methodology 2)
4.  A Simple Solution (solution)

* * *

### [**Object Detection from the Old Days 🧓**](#TOC)

Previously object detection methods made predictions relative to some initial guesses.

*   **Two-stage detectors:** predict boxes with respect to proposals (shown in **Figure 1**)  
    [](https://pyimagesearch.com/2020/07/13/r-cnn-object-detection-with-keras-tensorflow-and-deep-learning/)

*   **Single-stage detectors:** predict with respect to anchors or a grid of possible object centers (shown in **Figure 2**)  
    [](https://pyimagesearch.com/2018/11/12/yolo-object-detection-with-opencv/)

**The Reimagined Problem Statement:** The quality of the predictions relied a lot on how the initial guesses were made. So, instead of using handcrafted methods, we need a way to directly predict the set of detections.

* * *

### [**Set Prediction for Object Detection 🕵️**](#TOC)

In image processing applications, the basic set prediction task is multilabel classification. But first, let us quickly recap what multilabel and multiclass mean.

[**Multiclass**](https://pyimagesearch.com/2021/07/26/pytorch-image-classification-with-pre-trained-networks/)**:** An image belonging to one class out of several possible classes (an example of Multiclass object detection is shown in **Figure 3**).

[![Image 2: Figure 3: Multiclass detection (source: image by the authors).](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20768'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/05/dog-tree-multiclass_resized.png)

**Figure 3:** Multiclass detection (source: image by the authors).

[**Multilabel**](https://pyimagesearch.com/2018/05/07/multi-label-classification-with-keras/)**:** An image with multiple labels assigned to it simultaneously (an example of Multilabel object detection is shown in **Figure 4**).

[![Image 3: Figure 4: Multilabel detection (source: image by the authors).](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20768'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2023/05/dog-tree-multilabel_resized.png)

**Figure 4:** Multilabel detection (source: image by the authors).

The baseline approach for multilabel classification is called **one-vs-rest**. Here we train a binary classification model for each label and then try to predict which labels are present in the image and which are not. Of course, this technique works well only when the labels are **mutually exclusive**.

In the object detection pipeline, **near-identical boxes** are quite common. This fails the above-explained **one-vs-rest** approach. Instead, the authors use direct set prediction with a twist to achieve set prediction in such a scenario.

**_Note:_** _In direct set prediction, we need a way to consider the relationships between all the predicted elements to avoid duplicates._

This is where the authors use the Hungarian algorithm (covered in upcoming tutorials of this series), which helps them match the ground truth (actual data) with their predictions smartly.

* * *

### [**Transformers and Parallel Decoding for Object Detection ⚡**](#TOC)

Attention mechanisms aggregate information from the entire input sequence. These are more suitable for long sequence representations.

To gain confidence and a better understanding of Transformers and their underlying principles, we recommend watching **Video 1**, which provides a comprehensive overview and intuitive explanation of Transformers.

**Video 1:** A Deep Dive into Transformers | PyImageSearch | Live Lesson ([source](https://www.youtube.com/live/FJAJKPRHZUg?feature=share)).

Let’s pause and ponder on this for a second. What could go wrong if we input an image, and the object detection model predicts the objects (bounding boxes) in parallel?

While a model outputs all the objects in parallel, each prediction does not have the context of all the other predictions, unlike in autoregressive methods. This can create two problems:

*   The parallel model outputs duplicates.
*   The parallel model can output objects in a different order each time.

* * *

### [**A Niche Solution 🥳**](#TOC)

DETR addresses its challenges by combining bipartite matching loss and transformers. The matching loss function helps to pair each prediction with a unique ground truth object, so duplicates aren’t a concern.

Additionally, the bipartite matching loss doesn’t care about the order of predicted objects, meaning we don’t need to worry about the sequence in which predictions are made.

* * *

[**Quiz Time! 🤓**](#TOC)
-------------------------

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

Let’s take a moment to recap what we’ve learned in this tutorial so far:

1.  We explored the **challenges** and **pain points** of **object detection** before the advent of DETR.
2.  We briefly introduced **DETR’s innovative features** and main components, including **Set Prediction Loss** and the **Transformer-based architecture**.
3.  We **reformulated the problem statement** to better align with DETR’s approach.
4.  We familiarized ourselves with the two key components of DETR and **how they address** the problem statement.  
    

In the **next two tutorials of this series**, we’ll delve into the **tools**, **methodologies**, and **architecture** of **DETR** in greater detail, providing a more comprehensive understanding of this groundbreaking approach.

_However, valuable research is about more than just reading papers and admiring the brilliance of their authors. It’s also about asking ourselves what we would do in their position and how we would design an architecture to tackle these challenges if DETR didn’t exist._

Stay tuned for the upcoming parts of this series, and be sure to share your thoughts and learnings from DETR and what you’re looking forward to. Connect with us on Twitter by tagging [@pyimagesearch](https://twitter.com/PyImageSearch).

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty and R. Raha.** “DETR Breakdown Part 1: Introduction to DEtection TRansformers,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2023, [https://pyimg.co/fhm45](https://pyimg.co/fhm45)

@incollection{ARG-RR\_2023\_DETR-breakdown-part1,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {{DETR} Breakdown Part 1: Introduction to DEtection TRansformers},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2023},
  url = {https://pyimg.co/fhm45},
}

* * *

![Image 4: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.