ValueError: If `eos_token_id` is defined, make sure that `pad_token_id` is defined.
Author: TomasAndersonFang
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-4.18.0-425.19.2.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I finished the fine-tuning of CodeGen2 on my own dataset, but when I performed inference, I met the problem shown in the title. Here is my code:

```python
import json
from typing import Optional
from dataclasses import dataclass, field
from pathlib import Path

import torch
import transformers
from peft import PeftModel
from transformers import (
    AutoTokenizer, 
    AutoModelForCausalLM, 
    AutoConfig,
    GenerationConfig, 
    HfArgumentParser, 
    BitsAndBytesConfig,
)
from tqdm import tqdm


@dataclass
class ModelArguments:
    base_model_path: Optional[str] = field(default="elinas/llama-7b-hf-transformers-4.29")
    lora_path: Optional[str] = field(default="elinas/llama-7b-hf-transformers-4.29")
    max_length: int = field(default=512, metadata={"help": "Maximum length of the input sequence."})


@dataclass
class DataArguments:
    data_path: str = field(default=None, metadata={"help": "Path to the training data."})
    output_file: str = field(default=None, metadata={"help": "Output file name."})


@dataclass
class GenerationArguments:
    max_new_tokens: int = field(
        default=256,
        metadata={"help": "Maximum number of new tokens to generate."},
    )
    is_lora: bool = field(default=True, metadata={"help": "Whether to use LORA."})
    do_sample: bool = field(default=True, metadata={"help": "Whether to use sampling."})
    num_beams: int = field(default=1, metadata={"help": "Number of beams for beam search. 1 means no beam search."})
    temperature: float = field(default=1.0, metadata={"help": "Temperature for sampling."})
    top_k: int = field(default=50, metadata={"help": "Top-k for sampling."})
    top_p: float = field(default=1.0, metadata={"help": "Top-p for sampling."})
    request_num: int = field(default=1, metadata={"help": "Number of requests."})


def main():
    parser = HfArgumentParser((ModelArguments, DataArguments, GenerationArguments))
    model_args, data_args, generation_args = parser.parse_args_into_dataclasses()

    if generation_args.is_lora:
        model = AutoModelForCausalLM.from_pretrained(
            model_args.base_model_path,
            torch_dtype=torch.float16,
            load_in_8bit=True,
            trust_remote_code=True,
            quantization_config=BitsAndBytesConfig(
                load_in_8bit=True,
                llm_int8_threshold=6.0
            ),
        )

        model = PeftModel.from_pretrained(
            model,
            model_args.lora_path,
            torch_dtype=torch.float16,
        )

    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_args.base_model_path,
            torch_dtype=torch.float16,
        )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.base_model_path,
        trust_remote_code=True,
    )

    model.config.pad_token_id = tokenizer.pad_token_id = tokenizer.unk_token_id

    device = "cuda:0" if torch.cuda.is_available() else "cpu"
    model.to(device)
    
    if generation_args.do_sample:
        if generation_args.num_beams > 1:
            # Beam search
            generation_config = GenerationConfig(
                do_sample=generation_args.do_sample,
                num_beams=generation_args.num_beams,
                # max_new_tokens=generation_args.max_new_tokens,
                # num_return_sequences=generation_args.request_num,
            )
        else:
            # Temperature sampling
            generation_config = GenerationConfig(
                do_sample=generation_args.do_sample,
                # max_new_tokens=generation_args.max_new_tokens,
                # num_return_sequences=generation_args.request_num,
                temperature=generation_args.temperature,
                top_k=generation_args.top_k,
                top_p=generation_args.top_p,
            )

    data_path = Path(data_args.data_path)
    
    for original_file in tqdm(data_path.rglob('original.*'), desc="Generating..."):
        with open(original_file, 'r') as of:
            original_code = of.read()

        inputs = tokenizer(original_code, truncation=True, max_length=model_args.max_length, return_tensors='pt')
        inputs_len = inputs.input_ids.shape[1]
        input_ids = inputs.input_ids.to(device)

        outputs = model.generate(
            input_ids=input_ids,
            max_new_tokens=generation_args.max_new_tokens,
            num_return_sequences=generation_args.request_num,
            generation_config=generation_config,
        )
        output_ids = outputs[:, inputs_len:]

        output_diff = tokenizer.batch_decode(output_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
        original_outputs = tokenizer.batch_decode(outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False)

        output_dict = {}

        for i in range(len(output_diff)):
            output_dict[i] = {
                "original_output": original_outputs[i],
                "output_diff": output_diff[i],
            }
        
        output_file = original_file.parent / data_args.output_file

        with open(output_file, 'w') as pd_file:
            json.dump(output_dict, pd_file, indent=4)


if __name__ == "__main__":
    main()
```

I use the following bash script to run my code:

```bash
python3 codegen2_pred.py \
    --base_model_path <my_path> \
    --lora_path <my_path> \
    --data_path <my_path> \
    --output_file output.json \
    --is_lora True \
    --max_length 512 \
    --max_new_tokens 256 \
    --do_sample True \
    --num_beams 1 \
    --temperature 0.95 \
    --top_k 50 \
    --top_p 0.8 \
    --request_num 10 \
```

I got the following error information:

```
Traceback (most recent call last):
  File "codegen2_pred.py", line 141, in <module>
    main()
  File "codegen2_pred.py", line 115, in main
    outputs = model.generate(
  File "/python3.10/site-packages/peft/peft_model.py", line 977, in generate
    outputs = self.base_model.generate(**kwargs)
  File "/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/python3.10/site-packages/transformers/generation/utils.py", line 1572, in generate
    return self.sample(
  File "/python3.10/site-packages/transformers/generation/utils.py", line 2660, in sample
    raise ValueError("If `eos_token_id` is defined, make sure that `pad_token_id` is defined.")
ValueError: If `eos_token_id` is defined, make sure that `pad_token_id` is defined.
```

I'm confused why has this error since I set the `pad_token_id` in `codegen2_pred.py` at line 80: `model.config.pad_token_id = tokenizer.pad_token_id = tokenizer.unk_token_id`

### Expected behavior

The script will successfully generate results for my input.

Comments:
Pinging @gante here! 

I found this problem also occurs in Bloom and GPT-J. Additionally, I printed `tokenizer.pad_token_id` and `model.config.pad_token_id` and found that I successfully setted the `pad_token`.

@TomasAndersonFang the immediate fix would be to set the pad token in the generation config. Because a generation config is passed, `generate` assumes all parameterization is there (and it isn't).

On our end, we'll add better model config<>generation config arg management :)

> @TomasAndersonFang the immediate fix would be to set the pad token in the generation config. Because a generation config is passed, `generate` assumes all parameterization is there (and it isn't).
> 
> On our end, we'll add better model config<>generation config arg management :)

Thanks for your help, my problem is solved!

> @TomasAndersonFang the immediate fix would be to set the pad token in the generation config. Because a generation config is passed, `generate` assumes all parameterization is there (and it isn't).
> 
> On our end, we'll add better model config<>generation config arg management :)

I observed this error today with `TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ`:

```
            gene_config = transformers.GenerationConfig(
                max_new_tokens=args.max_new_tokens, 
                do_sample=True, 
                temperature=0.6, 
                top_p = 0.9,)       
            
            for i in range(args.n_tests):
                prompt = dataset[i]                
                input_ids = spec_generator.tokenizer.encode(prompt, return_tensors="pt").to(device)
                big_model.generate(input_ids, generation_config=gene_config)
```