No way to get ONLY the generated text, not including the prompt.
Author: monsieurpooh
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.15.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.5
- PyTorch version (GPU?): 1.10.2+cu113 (True)
- Tensorflow version (GPU?): 2.5.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@Narsil @patrickvonplaten 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

At first I thought I could just substring by the prompt's length. This doesn't work because there's a bug where it converts every instance of " ," to "," in the generated text. 

For example, "Characters in this scene: King , Gertrude." becomes "Characters in this scene: King, Gertrude."

In https://github.com/huggingface/transformers/blob/main/src/transformers/generation_utils.py there are tons of options but not a single one of them allows us to specify it to ONLY return the generated text, not including the prompt.

I could do a workaround where I replace all the " ," with "," myself, but I'm sure this is a code smell which could lead to future problems.

Example code:
```
 gen_tokens = model.generate(input_ids, do_sample=specifiedDoSample, temperature=specifiedTemperature, max_length=calculated_max_length, min_length=calculated_min_length, repetition_penalty=specifiedRepetitionPenalty, bad_words_ids=badWordsTokens)

        #gen_text = tokenizer.batch_decode(gen_tokens)[0]
```


### Expected behavior

Two possibilities: Either don't modify the prompt at all so I can substring by the prompt's length, or have an option where we get only the generated text not including the prompt.

Comments:
Hi @monsieurpooh ,

`generate` will not change, since it's a relatively low level function, it really does exactly what it should do to the relative tensors (`encoder-decoder` and `decoder-only` don't work the same for instance).

Two suggestions:

- Simple modification `gen_text = tokenizer.batch_decode(gen_tokens[input_ids.shape[0]:])[0]`  (Ignore the first ids you sent)
- Use a pipeline:
```python
from transformers import pipeline

# This will remove the text for you.
pipe = pipeline(model="gpt2", return_full_text=False)
print(pipe("This is a test"))
```

Does that solve your issue ?

Thanks so much for your help Narsil! After a tiny bit of debugging and learning how to slice tensors, I figured out the correct code is: `tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0]`
It returns the correct tokens even when there's a space after some commas and periods.

Thank you for giving the correct code here, will help other users for sure ! :)

is there a fix for this?

i'm using an workaround like this:
`encoding = tokenizer(batch['prompt'], return_tensors='pt', padding=True).to(device)
        with torch.no_grad():
            generated_ids = model.generate(**encoding)
        generated_ids = generated_ids[:, encoding.input_ids.shape[1]:]
        generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)`

> Thanks so much for your help Narsil! After a tiny bit of debugging and learning how to slice tensors, I figured out the correct code is: `tokenizer.batch_decode(gen_tokens[:, input_ids.shape[1]:])[0]` It returns the correct tokens even when there's a space after some commas and periods.

Small observation. This works only if 
* batch size = 1, or
* all elements of the batch have the same input context length.

I encountered the same problem! and write a simple code here:

```
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

# Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m", device_map="cuda")

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m", padding_side="left")

tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
model_inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")
generated_ids = model.generate(**model_inputs,do_sample=True, temperature=0.8, top_p=0.95,length_penalty=1.0,repetition_penalty=1.0,max_new_tokens=16)
full_outputs=tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

generated_texts = [] 
for input_text, full_output in zip(prompts, full_outputs): 
    generated_text = full_output[len(input_text):].strip() 
    generated_texts.append(generated_text)

print(f"full_outputs = {full_outputs}")
print(f"generated_texts = {generated_texts}")
```

This method did not work in case of having special tokens such as bos, eos etc in the input prompt, any idea how to handle those?

The problem seems to be the special tokens. 
![image](https://github.com/user-attachments/assets/0e8485e8-c775-484e-a47a-a222f1560d22)
I tried passing `skip_special_tokens=False` to have the correct calculation but that turned my decoded text into gibberish. 
![image](https://github.com/user-attachments/assets/e3560e52-ec1b-4ae6-b4a4-1a0cbbbcacc6)


Will not work with TextStream streaming !