[WIP] Adding DETR
Author: NielsRogge
Pull request: yes

# What does this PR do?

I've made quite a lot of progress on implementing HuggingFace's version of [DETR](https://arxiv.org/abs/2005.12872). However, there are some remaining things to be discussed, mainly regarding `DetrFeatureExtractor`, which can be used to prepare images + annotations for the model. 

## What I currently have
There are 3 models defined: 

- `DetrModel`, which consists of a convolutional backbone + encoder-decoder Transformer, without any head on top.
- `DetrForObjectDetection`, which is `DetrModel` with 2 heads on top, namely a class labels classifier and a bounding box regressor.
- `DetrForSegmentation`, which is `DetrForObjectDetection` (yes you read that right, not `DetrModel`) with a mask head on top, for predicting segmentation masks.

Available notebooks:

- [inference notebook](https://colab.research.google.com/drive/1RWzoQHkGSfztcRcgTRcd3FJUDY4GVXVB?usp=sharing) of `DetrForObjectDetection`
- [fine-tuning notebook](https://drive.google.com/file/d/1NbG_DEPh2A87bpyQYvuutFXDvczkYAJ8/view?usp=sharing) - fine-tuning `DetrForObjectDetection` on a custom dataset (balloon dataset) - obtaining very good results!
- [inference notebook](https://colab.research.google.com/drive/1P-bz2ZBPNciT86gFQTl_qiD2LVPKqrSW?usp=sharing) of `DetrForSegmentation` (panoptic segmentation)

There's the feature extractor:

- `DetrFeatureExtractor`, which can be used to prepare images and annotations for the model. The API is is similar to `ViTFeatureExtractor` and `DeiTFeatureExtractor`: the input are image(s) + annotation(s), and the output is `pixel_values` and `pixel_mask`. 
- Currently, it only supports preparing data for object detection, not for panoptic segmentation. It is based on [this code](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/datasets/coco.py#L17) in the original implementation. Given an image and annotations in COCO format, it turns the annotations into the format expected by DETR, followed by normalization + resizing of the image and corresponding annotations. 

## Questions

### 1: Supporting panoptic segmentation for DetrFeatureExtractor (done)
The problem is that if we also want to support panoptic segmentation, we rely on an external package named `panopticapi`, as it is used when preparing the annotations as can be seen [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/datasets/coco_panoptic.py#L9). I don't know how we can add this dependency, because I assume that people don't want to install this package if they want to use DetrFeatureExtractor for object detection. How can I handle an optional dependency?

I think there are 2 options here: either 1) add a `task` argument to the feature extractor and raise an error if task = "panoptic segmentation" and panopticapi is not available, or 2) create two different feature extractors (one for object detection, and one for panoptic segmentation). The first option would look something like

```
if task == "panoptic_segmentation":
  if not is_panopticapi_available():
     raise ImportError("Panopticapi is required for the feature extractor.")
```
also, this can be at the init of the feature extractor, or at the call

### 2: DetrForPanopticSegmentation
`DetrForPanopticSegmentation` is a bit special in the sense that there are 2 ways to train this model, either 1) end-to-end, in which you train `DetrForObjectDetection` and the mask head altogether, or 2) in a 2-step process, in which you first train a `DetrForObjectDetection` model to predict bounding boxes + classes, and then in a second step you provide this model to DetrForPanopticSegmentation, freeze it and only train the mask head further for about 25 epochs. That's why there's a `box_model` argument in the `init` of `DetrForPanopticSegmentation`. 

Also, `DetrForObjectDetection` itself only uses the last feature map of the convolutional backbone, but `DetrForPanopticSegmentation` does not, it also uses layers 2, 3 and 4 of a ResNet in the mask head. There's an attribute `return_intermediate_layers` of `DetrConfig`, which should be set to `False` for `DetrForObjectDetection` and `True` for `DetrForPanopticSegmentation`. Currently, I set `config.return_intermediate_layers` to `True` no matter what at the `init` of `DetrForPanopticSegmentation`, but I don't know if hard coding this value is allowed.

### 3: timm support
My implementation supports any convolutional backbone of the [timm](https://github.com/rwightman/pytorch-image-models) package. Should I add a `is_timm_available` check for the model (instead of `is_torch_available`)? 

Supporting only object detection would make life easier, but as DETR also obtains very good results on panoptic segmentation, it would be good to support that too. 

Would love to hear your opinions @sgugger @patrickvonplaten @LysandreJik @patil-suraj 


Comments:
Thanks for the review, yes you've made it a lot more clear for me now:
* it's only `rgb2id` and `id2rgb` which are used from `panopticapi`, and it's only about 20 lines of code I see now. So indeed, we can just copy that code into the library (and cite the authors).
* however, I think it's still useful to have a `task` attribute at the init of `DetrFeatureExtractor`, because then we can do input type checking depending on the task, and it will also be useful regarding postprocessing the outputs of DETR (the task attribute is also done in `LukeTokenizer` for example). 
* regarding `config.return_intermediate_layers`, there's indeed to reason for it to be user-facing anymore (it is in the original repo - but for other reasons), so let's remove that from the config.

Addressed most comments. Once the draft is done, I will create a new branch, squash all commits and open up a new PR, with the remaining comments copied.

@sgugger I've also added dummies for timm. But CI doesn't seem to be happy, as timm is not installed on it.