HuggingFace GroundingDINO inference execution time is slower than the original groundingDINO (~100ms)
Author: lucasMichelis
Pull request: no
Labels: name: Vision

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-5.10.0-30-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install the env : 
```
python3 -m venv ./.venv
source .venv/bin/activate
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install git+https://github.com/IDEA-Research/GroundingDINO.git#egg=groundingdino

# Get GroundingDINO weights
wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth
```

2. Run the below python script : 
```python
import os
import time
import torch
import sys
import PIL
import requests
import numpy as np
from groundingdino.util.inference import load_image, predict, load_model
from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor

TEST_IMAGE_URL_LIST = [
    'http://images.cocodataset.org/val2017/000000039769.jpg',
    'http://images.cocodataset.org/val2017/000000039769.jpg',
    'http://images.cocodataset.org/val2017/000000397133.jpg',
    'http://images.cocodataset.org/val2017/000000397133.jpg',
]
PATH_TO_ENV = "<path_to_the_installed_environment_.venv>"
PATH_TO_GROUNDINGDINO_WEIGHTS = "groundingdino_swinb_cogcoor.pth"

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device : {device}")

# CLASSIC GROUNDINGDINO MODEL
config_path = os.path.join(
    PATH_TO_ENV,
    f".venv/lib/python{sys.version_info.major}.{sys.version_info.minor}/site-packages/groundingdino/config/GroundingDINO_SwinB_cfg.py",
)
model = load_model(config_path, PATH_TO_GROUNDINGDINO_WEIGHTS)

# Predict
for image_url in TEST_IMAGE_URL_LIST:
    # Load the test image
    image = PIL.Image.open(requests.get(image_url, stream=True).raw)
    image.save("test_image.jpg")
    
    # Preprocess load + preprocess image using groundingdino load_image function
    start = time.time()
    _, preprocessed_image = load_image("test_image.jpg")
    preprocess_time = np.round(time.time() - start, 3)
    
    # Predict execution time
    start = time.time()
    boxes, logits, phrases = predict(
        model=model,
        image=preprocessed_image,
        caption="cat . human",
        box_threshold=0.3,
        text_threshold=0.25,
        remove_combined=True,
        device=device
    )
    inference_time = np.round(time.time() - start, 3)
    print(f"Classic GroundingDINO : Preprocess: {preprocess_time} // Inference: {inference_time}")
    

# HUGGINGFACE
print("############################")
# Load processor and model
hf_groundingDINO_processor = AutoProcessor.from_pretrained(
    "IDEA-Research/grounding-dino-base",
    torch_dtype=torch.float32
)
hf_groundingDINO_model = AutoModelForZeroShotObjectDetection.from_pretrained(
    "IDEA-Research/grounding-dino-base",
    torch_dtype=torch.float32
).to(device)

# Predict bounding boxes et get usable outputs
with torch.no_grad():
    for image_url in TEST_IMAGE_URL_LIST:
        # Load the test image
        image = PIL.Image.open(requests.get(image_url, stream=True).raw)
        image.save("test_image.jpg")
        
        # Load image + preprocess image as shown on HuggingFace example
        start = time.time()
        image = PIL.Image.open("test_image.jpg")
        model_inputs = hf_groundingDINO_processor(
            images=image, 
            text="shoes . dress", 
            return_tensors="pt"
        ).to(device)
        preprocess_time = np.round(time.time() - start, 3)
        
        # Predict execution time
        start = time.time()
        hf_groundingDINO_model(**model_inputs)
        inference_time = np.round(time.time() - start, 3)
        
        print(f"HuggingFace GroundingDINO : Preprocess: {preprocess_time} // Inference: {inference_time}")
```

### Expected behavior

Device : cuda
final text_encoder_type: bert-base-uncased

Classic GroundingDINO : Preprocess: 0.017 // Inference: 3.175
Classic GroundingDINO : Preprocess: 0.022 // Inference: 0.293
Classic GroundingDINO : Preprocess: 0.021 // Inference: 0.342
Classic GroundingDINO : Preprocess: 0.018 // Inference: 0.329
############################
HuggingFace GroundingDINO : Preprocess: 0.051 // Inference: 0.528
HuggingFace GroundingDINO : Preprocess: 0.046 // Inference: 0.378
HuggingFace GroundingDINO : Preprocess: 0.052 // Inference: 0.426
HuggingFace GroundingDINO : Preprocess: 0.051 // Inference: 0.426



Comments:
Hi @lucasMichelis, thanks for opening this issue! 

Regarding the processing times, this isn't that surprising. The image processors we currently have are unfortunately pretty slow, as they do everything through list comprehensions and in numpy. Their main purpose is to enable users to quickly go from image -> model in a single step. We're working on improving this and will hopefully be introducing a "fast" version of grounding dino's processor soon c.f. #28847 Until then, I'd recommend using a torchvision pipeline is inference time is important. 

For the different in model inference time, it's something we can look into! As the differences aren't huge i.e. not indicative of a bug, it's not something which I can guarantee will be addressed immediately, but we'd like to have comparable numbers. If you or anyone in the community would like to do some investigation and open a PR to make things faster I'd be very happy to review! 

cc @EduardoPach 

@lucasMichelis do you have `ninja` installed as well? If I'm not mistaken the original implementation uses the custom kernels for the `MultiScaleDeformableAttention`, but the `transformers` implementation will only use if you have `ninja` installed 

@amyeroberts Thank you for your answer !
I will probably use a torchvision pipeline for preprocessing if it can make it quicker !

@EduardoPach You were right ! it resolves the issue !
Once ninja is installed, the inference time is almost exactly the same ! 

Thank you both for your quick answers ! 👍 

[UPDATE] Why is that ninja library not in the requirement of the pyproject.toml of transformers ?

> Why is that ninja library not in the requirement of the pyproject.toml of transformers ?

@lucasMichelis The reason is it's used for a small number of our 200+ models in the library. Every dependency we add as part of the default installation adds installation time, maintenance requirements and package size. Instead, we prefer to specify a minimal set and let users optionally install other libraries according to their needs e.g. we don't require flash attention to be installed. 

One thing we could do is add a note on the model's doc page suggesting this install and emitting a logger.info message if we don't use the custom cuda kernels. Would you like to open a PR to add these? 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @lucasMichelis, what happens to the first image that has completely different inference time between the 2 approaches? 
```
Classic GroundingDINO : Preprocess: 0.017 // Inference: 3.175
############################
HuggingFace GroundingDINO : Preprocess: 0.051 // Inference: 0.528
```
And does using the torchvision pipeline for preprocessing reduce the inference time? Have you tested it?