Kosmos2 device_map and batch processing issues.
Author: rabiulcste
Pull request: no

### System Info


- `transformers` version: 4.36.0.dev0
- Platform: Linux-4.15.0-213-generic-x86_64-with-glibc2.27
- Python version: 3.10.11
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: RTX8000
- Using distributed or parallel set-up in script?: No




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Batch processing raises batch size mismatch errors.
```
from transformers import AutoProcessor, Kosmos2ForConditionalGeneration
from PIL import Image
import requests


def initialize_grounding_model(model_name: str):
    print(f"Initializing {model_name} model")
    model = Kosmos2ForConditionalGeneration.from_pretrained(model_name, device_map="auto")
    processor = AutoProcessor.from_pretrained(model_name)
    processor.tokenizer.padding_side = "left"
    return model, processor


def test_kosmos2_batch(model, processor, batching=False):
    print("=========================================")
    print(f"Running test with Batching = {batching}")
    print("=========================================")

    prompt = "<grounding>An image of"
    url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.png"
    image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

    if batching:
        prompts = [
            "<grounding>An image of",
            "<grounding>A photo of",
            "<grounding>Describe this photo in details",
            "<grounding>What is in this photo?",
            "<grounding>What is this a picture of?",
            "<grounding>What is the image about?",
        ]
        batch_images = [image] * 6
    else:
        prompts = prompt
        batch_images = image

    print(f"Input: {prompts}")
    print(f"Image: {batch_images}")
    inputs = processor(text=prompts, images=batch_images, padding=True, return_tensors="pt")
    device = model.device
    print(f"Running on device: {device}")
    generated_ids = model.generate(
        pixel_values=inputs["pixel_values"].to(device),
        input_ids=inputs["input_ids"].to(device),
        attention_mask=inputs["attention_mask"].to(device),
        image_embeds=None,
        image_embeds_position_mask=inputs["image_embeds_position_mask"].to(device),
        use_cache=True,
        max_new_tokens=64,
    )

    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
    processed_text = [
        processor.post_process_generation(out, cleanup_and_extract=False) for out in generated_text
    ]
    print(f"Output: {processed_text}")


if __name__ == "__main__":
    model_name = "microsoft/kosmos-2-patch14-224"
    model, processor = initialize_grounding_model(model_name)
    print("Model initialized successfully!")
    test_kosmos2_batch(model, processor, batching=False)
    test_kosmos2_batch(model, processor, batching=True)

```    
```   
Traceback (most recent call last):
  File "/home/mila/r/rabiul.awal/.venv/dgx-machines/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 748, in convert_to_tensors
    tensor = as_tensor(value)
  File "/home/mila/r/rabiul.awal/.venv/dgx-machines/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 720, in as_tensor
    return torch.tensor(value)
ValueError: expected sequence of length 74 at dim 1 (got 75)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mila/r/rabiul.awal/vqazero-private/local_files/test_kosmos2.py", line 64, in <module>
    test_kosmos2_batch(model, processor, batching=True)
  File "/home/mila/r/rabiul.awal/vqazero-private/local_files/test_kosmos2.py", line 39, in test_kosmos2_batch
    inputs = processor(text=prompts, images=batch_images, padding=True, return_tensors="pt")
  File "/home/mila/r/rabiul.awal/.venv/dgx-machines/lib/python3.10/site-packages/transformers/models/kosmos2/processing_kosmos2.py", line 257, in __call__
    BatchEncoding(
  File "/home/mila/r/rabiul.awal/.venv/dgx-machines/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 223, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/home/mila/r/rabiul.awal/.venv/dgx-machines/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 764, in convert_to_tensors
    raise ValueError(
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
```

2. To reproduce the device_map issue please run the same command with 2 GPUs. It will raise the device allocation issue.


### Expected behaviour

I want the device_map to assign available GPUs automatically. Also, this is the only model failing in newer versions of Transformers. I've used it fine previously! 
- fix device map or .cuda() for multiple GPUs
- fix batch processing as expected

Comments:
cc @ydshieh 

Hi @rabiulcste 

Thank you (again) for opening this issue. I will take a look. In order to ease the process, could you provide all the necessary imports and variable definition.

The goal is to have a self-complete code snippet that could be run, so we can debug it straightforward.

> Hi @rabiulcste
> 
> Thank you (again) for opening this issue. I will take a look. In order to ease the process, could you provide all the necessary imports and variable definition.
> 
> The goal is to have a self-complete code snippet that could be run, so we can debug it straightforward.

Thanks for the prompt response! I've now updated it with a complete script. 

Very nice @rabiulcste I appreciated a lot - will take a look!

@rabiulcste Regarding batch issue, it is fixed in #27323.


@ydshieh thanks so much for the fix! any thoughts on the multi-GPU issue?



@rabiulcste taking a look today for auto map

@rabiulcste Fixed!