Paper Review — End-to-End Detection With Transformers
By: Krut Patel
Tags: Transformers, Machine Learning, Computer Vision, Object Detection, Deep Learning

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko

ArXiv link — https://arxiv.org/abs/2005.12872

Contributions

In this paper —

the authors present DETR — an object detection method based on transformers and bipartite matching loss that views the problem as a direct set prediction problem.

their method removes the need for hand-designed components that require prior knowledge about the task like non-maximum suppression and anchor generation.

the source code is made available at — https://github.com/facebookresearch/detr

DETR Architecture

DETR Architecture (src — https://arxiv.org/pdf/2005.12872.pdf)

The model consists of three components as shown in the above figure

1. CNN Backbone

The CNN takes in the image and learns a 2D representation of it by generating a low resolution activation map.

2. Encoder-Decoder Transformer

Architecture of DETR’s transformer (src — https://arxiv.org/pdf/2005.12872.pdf)

Each encoder layer follows the standard architecture of the transformer consisting of a multi-head self-attention module and a feed forward network.

The decoder also follows the standard architecture of the transformer consisting of multi-headed self- and encoder-decoder attention module

3. Feed Forward Network

3-layer perceptron with RELU activations, a hidden dimension d and a linear projection layer are used for the final prediction.

FFN predicts the normalized center, coordinates, height and width of the box w.r.t the input image

Linear layer predicts the class label using softmax function.

Approach

The backbone takes an image with 3 channels as input and outputs a lower-resolution activation map of it with 2048 channels

A 1x1 convolution reduces the channel dimension of this activation map creating a smaller dimension feature map. Then the spatial dimension of this feature map is collapsed since the transformer’s encoder expects a sequence as input.

A positional encoding is passed along with the collapsed feature vector as input to the encoder.

The decoder then takes in the output of the encoder along with object queries which are a small fixed number of learned positional embedding. They are added to the input of each attention layer

The decoder decodes N objects in parallel outputting N embedding, which are then independently passed to a shared FFN after normalizing them by a shared layer-norm.

The FFN independently predicts the class and bounding box (or no object) for each of those N decoder embeddings.

While using the whole image as context, the model globally reasons about all objects together using pair-wise relations between them by using self- and encoder-decoder attention over these embeddings.

An auxiliary decoding loss is used during training to help the model output the correct number of objects of each class.

A bipartite matching loss is used to train DETR.

Loss Function — Set Prediction Loss

It forces unique matching between predicted and ground truth boxes. DETR outputs a fixed-size of N predictions, where N is set to a number larger

than the typical number of objects in an image. The loss function produces an optimal bipartite matching between predicted and ground truth objects and then optimizes object-specific bounding box losses.

Step 1: Finding pair wise matching

All permutations of N elements are searched to find a bipartite matching between the predicted and the ground truth set of objects as seen below:

The matching cost takes into account both the class prediction and the similarity of predicted and ground truth boxes as seen below:

Step 2: Hungarian Loss

For all pairs matched in the previous step, hungarian loss is defined as a linear combination of a negative log-likelihood for class prediction and box loss.

Bounding box loss

Class prediction loss

Experiments

The authors show that their approach achieves competitive results compared to Faster R-CNN in quantitative evaluation on COCO dataset.

Comparison with Faster R-CNN (src — https://arxiv.org/pdf/2005.12872.pdf)

They also provide a detailed ablation study with insights and qualitative results for the architecture and loss function.

They also present results of extending DETR for the task of panoptic segmentation by training a small extension on a fixed DETR model.

DETR extended by a panoptic head (src — https://arxiv.org/pdf/2005.12872.pdf)

Results for panoptic segmentation generated by DETR-R101 (src — https://arxiv.org/pdf/2005.12872.pdf)

Short Comings

Though DETR seems to predict large objects accurately, it has trouble regarding training, optimization and performance on small objects.

References

https://arxiv.org/pdf/2005.12872.pdf

https://arxiv.org/abs/2005.12872

https://github.com/facebookresearch/detr

https://ai.facebook.com/research/publications/end-to-end-object-detection-with-transformers