Stripping last some words from output of model.generate() method
Author: ancil009
Pull request: no

### System Info

Hi All,

Am using pretrained "gec-t5_small" for grammar error correction. But the output from model.generate() method stripping the output. Please any one suggest a solution for the same.

**Code**

```
model = T5ForConditionalGeneration.from_pretrained("Unbabel/gec-t5_small", torch_dtype="auto")
tokenizer = T5Tokenizer.from_pretrained('t5-small',model_max_length=1024, torch_dtype="auto")

sentence  = "600 character length sentence"

sentence = sentence.strip()
tokenized_sentence = tokenizer('gec: ' + sentence , max_length=1024, truncation=True,  return_tensors='pt',add_special_tokens=True)

model_output = model.generate(
                input_ids = tokenized_sentence.input_ids,
                attention_mask = tokenized_sentence.attention_mask, 
                max_new_tokens = 1024, (max_length also tried)
                use_cache=True,
                num_beams=3, (tried  num_beams=5 as well)
                early_stopping=True,
                do_sample=False,        
            )

corrected_sentence = tokenizer.decode(
            model_output[0],
            skip_special_tokens=True, 
            clean_up_tokenization_spaces=True
        )
```

**Output**

sentence : "Manager Hi sir I am Srikanth so all ready the team is discussed to develop a new feature of XYZ so that's why we discussed a that new features about that project so in that case in my team to learn about the new projects topics to based on our discussion the project so it's take's time to learn to that features and to implement and as off you know my team members are very fast to learn about new things and in work purpose also my team members are very fast and so please give additional to complete about new features of our project and sorry for the delay but you give me additional time my team members are to give our more best about to our new features."

corrected_sentence :  "Manager Hi sir I am Srikanth so all ready the team is discussed to develop a new feature of XYZ so that's why we discussed those new features about that project so in that case in my team to learn about the new projects topics to based on our discussion the project so it's take's time to learn to those features and to implement and as of you know my team members are very fast to learn about new things and in work purpose also my team members are very fast and so please give additional information to complete"

Please suggest any solution for the same? Is there any way to increase length of output.

Note:- platform Azure databricks

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps:

1) Use the above mentioned code .
2) with sentence as input .
3) check output is deletion some word from end of the string.

### Expected behavior

Corrected should give fully inputted sentence with correct grammar.

Comments:
cc @gante 

Hi @ancil009 👋 

I've spent some time debugging, and here's what's happening:
The output is shorter because the model outputs `eos_token_id` at that point. In other words, it thinks it is done. In fact, if I modify the code to ignore `eos_token_id`, the output is as follows.

```
Manager Hi sir I am Srikanth so all ready the team is discussed to develop a new feature of XYZ so that's why we discussed those new features about that project so in that case in my team to learn about the new projects topics to based on our discussion the project so it's take's time to learn to those features and to implement and as of you know my team members are very fast to learn about new things and in work purpose also my team members are very fast and so please give additional information to complete Manager Hi sir I am Srikanth. I am ready to discuss the team is discussed to develop a new feature of XYZ. The team is discussed. So, I am ready to discuss the new features about that project, so in that case in my team to learn about the new projects topics to based on our discussion the project so it's take's time to learn about the new features and in that case in the Manager Hi sir. Hi sir I am Sri Lankan. Hi sir. I am Sri Lankan. Hi sir. I am Sri Lankan. So, I am Sri Lankan. So, I am Sri Lankan. So, I am Sri Lankan. So, I am very fast and so please give me the team members are very fast and Manager Hi sir. Hi sir, Hi sir, Hi sir, Hi sir, Hi sir, Hi sir, I am Sri Lankan. Hi sir, I am Sri Lankan. Hi sir, I am Sri Lankan. Hi sir, I am Sri Lankan. Hi sir, I am Sri Lankan. Hi sir, I am Sri Lankan. So, I am very fast and so please give me additional information about the new features of XYZ. Manager Hi sir. [...]
```

In other words, the model starts repeating itself, which isn't helpful. From this, we can rule out code-related problems.

Despite accepting infinite sequences, T5 has a relatively small attention window. Depending on the dataset the model was fine-tuned with, it might also be biased towards short sequences. Can you try splitting your input into multiple (smaller) sequences, to see if it helps?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.