GroundingDINO cannot work with MiniGPT4
Author: pspdada
Pull request: no
Labels: name: bug

name: Vision

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-5.15.0-120-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: 1
- GPU type: NVIDIA A100-PCIE-40GB

### Who can help?

@zucchini-nlp  @amyeroberts 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I use the minigpt4 model from the repository ·https://github.com/Vision-CAIR/MiniGPT-4·, I find that grounding dino cannot be used together with it.

Specifically, when I import some necessary content from the minigpt4 repository into my project (without doing anything else about the minigpt4 repo) and use transformers grounding dino model, dino crashes the program directly at the `model(**encoded_inputs)` call with an error code of SIG(117), and no traceback or other information is provided.

Other models, such as flan-t5-base-VG-factual-sg, do not crash during their forward pass even when minigpt4 is imported.

After commenting out the four import lines related to minigpt4, there are no issues anymore.
```python
import torch
from PIL import Image
from transformers import (
    GroundingDinoForObjectDetection,
    GroundingDinoProcessor,
)

# imports modules for registration
from minigpt4.datasets.builders import *  # noqa
from minigpt4.models import *  # noqa
from minigpt4.processors import *  # noqa
from minigpt4.tasks import *  # noqa

image_path = "/root/llm-project/LVLM/eval/Extended_CHAIR/images/chair-500/000000006763.jpg"
image: Image.Image = Image.open(image_path)
model: GroundingDinoForObjectDetection = (
    GroundingDinoForObjectDetection.from_pretrained(
        "IDEA-Research/grounding-dino-base",
        cache_dir="/root/llm-project/utils/models/hub",
        torch_dtype="auto",
        low_cpu_mem_usage=True,
    )
    .to("cuda")
    .eval()
)

processor: GroundingDinoProcessor = GroundingDinoProcessor.from_pretrained(
    "IDEA-Research/grounding-dino-base",
    cache_dir="/root/llm-project/utils/models/hub",
)

text = "man.umbrella.top hat."

with torch.inference_mode():
    encoded_inputs = processor(
        images=image,
        text=text,
        max_length=200,
        return_tensors="pt",
        padding=True,
        truncation=True,
    ).to("cuda")
    outputs = model(**encoded_inputs) # Crash here
    results = processor.post_process_grounded_object_detection(
        outputs=outputs,
        input_ids=encoded_inputs["input_ids"],
        box_threshold=0.25,
        text_threshold=0.25,
    )
    print(results)
```

### Expected behavior

Since this issue is related to other repositories, I would like to ask if you can help resolve this problem? Or kindly just guide me on how to find the deeper cause? Combining multiple models is significant for my project, but this issue does not provide any traceback, leaving me without a starting point.

Comments:
Hi @pspdada, thanks for reporting the issue! Does it work fine without any imports from minigpt4 in your env?

> Hi @pspdada, thanks for reporting the issue! Does it work fine without any imports from minigpt4 in your env?

Yes

@zucchini-nlp @qubvel 

I have found where the problem lies: when I execute the `forward` method, there's an issue in the sequence of methods `GroundingDinoEncoder -> GroundingDinoEncoderLayer -> GroundingDinoDeformableLayer -> GroundingDinoMultiscaleDeformableAttention -> forward()`. Specifically, the following code causes the problem:
``` python
try:
    # custom kernel
    output = MultiScaleDeformableAttentionFunction.apply(
        value,
        spatial_shapes,
        level_start_index,
        sampling_locations,
        attention_weights,
        self.im2col_step,
    )
except Exception:
    # PyTorch implementation
    output = multi_scale_deformable_attention(value, spatial_shapes, sampling_locations, attention_weights)
```
The part `try: MultiScaleDeformableAttentionFunction.apply()` crashes the program. If I modify this part of the code to directly use:
``` python
# PyTorch implementation
output = multi_scale_deformable_attention(value, spatial_shapes, sampling_locations, attention_weights)
```
then there is no issue, and the entire grounding DINO inference process works without problems.

This is a very strange phenomenon because the error does not get caught by the `try-except` block but instead causes the entire Python program to crash without providing any traceback. Do you have any ideas about this?

I can try to dive deeper later

@pspdada hmm interesting. Thanks for investigating this further. If the error arises from custom kernels, a workaround can be to save the config with `config.disable_custom_kernels = True` value so you do not have to manually edit the source code.

Feel free to dive deeper, as out team might take longer to investigate this issue. Looking forward to your insights!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.