Error loading model with device_map="auto" for AutoModelForVisualQuestionAnswering in visual-question-answering pipeline
Author: chakravarthik27
Pull request: no
Labels: name: bug

### System Info


- `transformers` version: 4.44.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.9.13
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Rocketknight1

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import pipeline

pipe = pipeline("visual-question-answering", model=path, device_map="auto")
 ```

### Expected behavior

it should be support for all type of models under **visual-question-answering** pipeline

Comments:
```python
ValueError: Could not load model Salesforce/blip-vqa-base with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForVisualQuestionAnswering'>, <class 'transformers.models.blip.modeling_blip.BlipForQuestionAnswering'>). See the original errors:

while loading with AutoModelForVisualQuestionAnswering, an error is thrown:
Traceback (most recent call last):
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\pipelines\base.py", line 286, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 3875, in from_pretrained
    no_split_modules = model._get_no_split_modules(device_map)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 1979, in _get_no_split_modules
    raise ValueError(
ValueError: BlipForQuestionAnswering does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.

while loading with BlipForQuestionAnswering, an error is thrown:
Traceback (most recent call last):
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\pipelines\base.py", line 286, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 3875, in from_pretrained
    no_split_modules = model._get_no_split_modules(device_map)
  File "c:\Users\KALYAN\OneDrive\Documents\JSL_Developements\langtest_2.5.0\langtest\.venv\lib\site-packages\transformers\modeling_utils.py", line 1979, in _get_no_split_modules
    raise ValueError(
ValueError: BlipForQuestionAnswering does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.

```

cc @qubvel @zucchini-nlp @yonigozlan maybe

Rather than a bug, this is more of a feature request. Do we have a guide to add support big model inference for specific models @SunMarc ?

I think it comes to just adding the `_no_split_modules` attribute in the model class, similar to what we did with some other VLMs. Not sure if there was any specific reason for not supporting it in the first place so we need to enable/run the multi-gpu tests and see if everything works

@chakravarthik27 would you like to open a PR?

> I think it comes to just adding the `_no_split_modules` attribute in the model class, similar to what we did with some other VLMs. Not sure if there was any specific reason for not supporting it in the first place so we need to enable/run the multi-gpu tests and see if everything works
> 
> @chakravarthik27 would you like to open a PR?

Sorry, I didn't worked on this error.

oke no worries, I'll submit a PR soon then :)