Fix RT-DETR inference with float16 and bfloat16
Author: qubvel
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fix positional embeddings and anchor data types for RT-DETR.

Running the model in `float16` or `bfloat16` with the following code crashes with the error:

```python
import torch
import requests

from PIL import Image
from transformers import RTDetrForObjectDetection, RTDetrImageProcessor

device = "cuda"
dtype = torch.float16

url = 'http://images.cocodataset.org/val2017/000000039769.jpg' 
image = Image.open(requests.get(url, stream=True).raw)

image_processor = RTDetrImageProcessor.from_pretrained("PekingU/rtdetr_r50vd")
model = RTDetrForObjectDetection.from_pretrained("PekingU/rtdetr_r50vd", attn_implementation="eager", torch_dtype=dtype, device_map=device)

inputs = image_processor(images=image, return_tensors="pt").to(device)
pixel_values = inputs.pixel_values.to(device).to(dtype)

with torch.no_grad():
    outputs = model(pixel_values)

assert outputs.logits.dtype == dtype

results = image_processor.post_process_object_detection(outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.3)

for result in results:
    for score, label_id, box in zip(result["scores"], result["labels"], result["boxes"]):
        score, label = score.item(), label_id.item()
        box = [round(i, 2) for i in box.tolist()]
        print(f"{model.config.id2label[label]}: {score:.2f} {box}")

```
Error:
```
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half
```

One approach to fix is to use `torch.autocast()`, but it could also be fixed with proper dtypes in the modeling file.

Steps:

1. Add test that fails with float16/bfloat16 dtypes
2. Add fix

## Before submitting
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
[Added test failed](https://github.com/huggingface/transformers/actions/runs/9680176397/job/26707756431?pr=31639) for float16 and bfloat16 as expected:

<img width="1456" alt="Screenshot 2024-06-26 at 14 14 00" src="https://github.com/huggingface/transformers/assets/31920396/d1c3efa4-4d90-41dd-8fc8-b2096f6d497f">


Tests are green with the fix

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31639). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review!