How to set pad_token of Llava for batched generation and training?
Author: TideDra
Pull request: no

Hello, @younesbelkada I'm trying to use Llava for batched generation, using the default pad_token. here is the script:
```python
import json
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration,AutoTokenizer
from torch.utils.data import Dataset,DataLoader
import torch
import os
from tqdm import tqdm
DATA_ROOT = "/mnt/gozhang/code/LLaVA/playground/data/eval/mm-vet"
processor = AutoProcessor.from_pretrained("/mnt/gozhang/ckpts/llava-1.5-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("/mnt/gozhang/ckpts/llava-1.5-7b-hf")

class MMVetDataset(Dataset):
    def __init__(self,data_root) -> None:
        super().__init__()
        self.data_root = data_root
        with open(os.path.join(data_root, "mm-vet.json"), "r") as f:
            data = json.load(f)
        self.data = [(k,v) for k,v in data.items()]
    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        return {'id':self.data[index][0],
                'image':os.path.join(self.data_root,'images',self.data[index][1]['imagename']),
                'question':"USER: <image>\n"+self.data[index][1]['question']+" ASSISTANT:"}

def collator(batch):
    ids = [b['id'] for b in batch]
    questions = [b['question'] for b in batch]
    images = [Image.open(b['image']) for b in batch]
    inputs = processor(text=questions,images=images,return_tensors="pt",padding=True)
    return ids,inputs

model = LlavaForConditionalGeneration.from_pretrained("/mnt/gozhang/ckpts/llava-1.5-7b-hf",torch_dtype=torch.float16)
model.to('cuda')
#model.to(torch.float16)
dataset = MMVetDataset(DATA_ROOT)
dataloader = DataLoader(dataset,batch_size=16,collate_fn=collator)
results = {}
bar = tqdm(total=len(dataset))
model.eval()
with torch.inference_mode():
    for ids, inputs in dataloader:
        inputs.to('cuda')
        inputs['pixel_values'] = inputs['pixel_values'].half()
        outputs = model.generate(**inputs,temperature=0.2,do_sample=True,max_new_tokens=1024,use_cache=True)
        input_token_len = inputs['input_ids'].shape[1]
        responses=tokenizer.batch_decode(outputs[:, input_token_len:], skip_special_tokens=True, clean_up_tokenization_spaces=False)
        for id,res in zip(ids,responses):
            results[id]=res
        bar.update(len(responses))
with open('mmvet_result.json','w') as f:
    json.dump(results,f,indent=4)
```
But when generating the fifth batch, it reports `RuntimeError: probability tensor contains either inf, nan or element < 0`. Then I try different pad_token, setting `processor.tokenizer.pad_token = processor.tokenizer.unk_token` (following the raw llava codebase), or `processor.tokenizer.pad_token = processor.tokenizer.eos_token`(following the common setting), or `processor.tokenizer.pad_token = processor.tokenizer.bos_token`(following this [issue](https://discuss.huggingface.co/t/llama2-pad-token-for-batched-inference/48020)). And I find that only setting pad_token to eos_token can avoid the error. 
I wonder what's the effect of different pad_token during batched generation, and what's the root cause of this error, and how to set the correct pad_token for training the model?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey 🤗  Sorry for the late reply here, and thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!