ValueError: The following `model_kwargs` are not used by the model: ['length']
Author: CrackerHax
Pull request: no
Labels: name: wontfix

### System Info

4.22.2

### Who can help?

@SaulLu

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```
            prompt = tokenizer(user_input, return_tensors='pt', return_length=True)
            prompt = {key: value.to(device) for key, value in prompt.items()}
            out = gpt.generate(**prompt, ...)
```

When using "return_length=True" with the tokenizer, the error is given. This is from a change in a recent version and did not happen in older versions.

`ValueError: The following `model_kwargs` are not used by the model: ['length'] (note: typos in the generate arguments will also show up in this list)`

### Expected behavior

Model should not produce an error when "return_length" is set to True
Downgrade to 4.21.0 fixes the problem and according to my googling this is what people are doing

Comments:
Hi @CrackerHax,

Thanks for the issue. 

Could I ask you to share also the lines of code that you used to initialize the model, tokenizer and user_input which work with the `4.21.0` version of transformers and not the newer versions?

FYI, I'm also triggering this error when I use the latest transformers, when running the [BLIP captioning model example](https://github.com/salesforce/LAVIS) on the saleforce lavis library which uses transformers:

``` py
import torch
from lavis.models import load_model_and_preprocess
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# loads BLIP caption base model, with finetuned checkpoints on MSCOCO captioning dataset.
# this also loads the associated image processors
model, vis_processors, _ = load_model_and_preprocess(name="blip_caption", model_type="base_coco", is_eval=True, device=device)
# preprocess the image
# vis_processors stores image transforms for "train" and "eval" (validation / testing / inference)
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
# generate caption
model.generate({"image": image})
# ['a large fountain spewing water into the air']
```

In my case I was getting the following error:

```
 The following `model_kwargs` are not used by the model: ['encoder_hidden_states', 'encoder_attention_mask'] (note: typos in the generate arguments will also show up in this list)
```

> Hi @CrackerHax,
> 
> Thanks for the issue.
> 
> Could I ask you to share also the lines of code that you used to initialize the model, tokenizer and user_input which work with the `4.21.0` version of transformers and not the newer versions?

config = transformers.GPTJConfig.from_pretrained("./gpt-j-6B-8bit)
tokenizer = AutoTokenizer.from_pretrained("./gpt-j-6B-8bit",add_prefix_space=True)
vocab           = tokenizer.get_vocab()
gpt = GPTJForCausalLM.from_pretrained("./gpt-j-6B-8bit",low_cpu_mem_usage=True)

prompt = tokenizer(user_input, return_tensors='pt', return_length=True)
prompt = {key: value.to(device) for key, value in prompt.items()}
out = gpt.generate(**prompt, max_length=max_length, do_sample=True, temperature=temperature, repetition_penalty=2.0)
  out = tokenizer.decode(out[0], skip_special_tokens=True)

> FYI, I'm also triggering this error when I use the latest transformers, when running the [BLIP captioning model example](https://github.com/salesforce/LAVIS) on the saleforce lavis library which uses transformers:
> 
> ```python
> import torch
> from lavis.models import load_model_and_preprocess
> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
> # loads BLIP caption base model, with finetuned checkpoints on MSCOCO captioning dataset.
> # this also loads the associated image processors
> model, vis_processors, _ = load_model_and_preprocess(name="blip_caption", model_type="base_coco", is_eval=True, device=device)
> # preprocess the image
> # vis_processors stores image transforms for "train" and "eval" (validation / testing / inference)
> image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
> # generate caption
> model.generate({"image": image})
> # ['a large fountain spewing water into the air']
> ```
> 
> In my case I was getting the following error:
> 
> ```
>  The following `model_kwargs` are not used by the model: ['encoder_hidden_states', 'encoder_attention_mask'] (note: typos in the generate arguments will also show up in this list)
> ```

Same here. Can anyone help fix this?

@zzxslp 

Change these line at https://github.com/salesforce/BLIP/blob/main/models/med.py#L932 as following:

from
```python
    def prepare_inputs_for_generation(self, input_ids, past=None, attention_mask=None, **model_kwargs):
        input_shape = input_ids.shape
        # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
        if attention_mask is None:
            attention_mask = input_ids.new_ones(input_shape)

        # cut decoder_input_ids if past is used
        if past is not None:
            input_ids = input_ids[:, -1:]

        return {
            "input_ids": input_ids, 
            "attention_mask": attention_mask, 
            "past_key_values": past,
            "encoder_hidden_states": model_kwargs.get("encoder_hidden_states", None),
            "encoder_attention_mask": model_kwargs.get("encoder_attention_mask", None),
            "is_decoder": True,
        }
```

to
```python
    def prepare_inputs_for_generation(self, input_ids, past=None, attention_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, **model_kwargs):
        input_shape = input_ids.shape
        # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
        if attention_mask is None:
            attention_mask = input_ids.new_ones(input_shape)

        # cut decoder_input_ids if past is used
        if past is not None:
            input_ids = input_ids[:, -1:]

        return {
            "input_ids": input_ids, 
            "attention_mask": attention_mask, 
            "past_key_values": past,
            "encoder_hidden_states": encoder_hidden_states,
            "encoder_attention_mask": encoder_attention_mask,
            "is_decoder": True,
        }
```

Why: https://github.com/huggingface/transformers/blob/v4.23.1/src/transformers/generation_utils.py#L899

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still needs to be addressed

Would you like to take a look at this @ArthurZucker?

Sure! 


@ArthurZucker  I can look at this issue, But I am setup for reproduction due to size of model, how do you folks setup for reproducing bug for larger models ? 

Well, in that case I just downloaded the model. You can usually initialize a random tiny model using a different configuration. 
Could you tell me why you need to set `return_length=True`. This has the effect of adding `length` to the list of inputs, and is not useful when generating. This argument is mostly used when training a fast tokenizer.

> Well, in that case I just downloaded the model. You can usually initialize a random tiny model using a different configuration. Could you tell me why you need to set `return_length=True`. This has the effect of adding `length` to the list of inputs, and is not useful when generating. This argument is mostly used when training a fast tokenizer.

Yes, in my use case I need the length when generating. Unless you have a really good reason, it shouldn't be removed but fixed.

The thing is that the `generate` function does not take the `length` argument, and there is no reason to add it as it is not used. Which is why  I don't understand why you would use it? 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> The thing is that the `generate` function does not take the `length` argument, and there is no reason to add it as it is not used. Which is why I don't understand why you would use it? 😉

Yes it does, it takes min and max length as arguments. The only other option is to use an older version of huggingface. Why would it work in earlier versions and not now? This is useful for adjusting max_length dynamically such as:

```
            max_t=max_length+prompt['length'][0]
            if(max_length<min_length):
                max_length=min_length
            prompt = {key: value.to(device) for key, value in prompt.items()}
            out = gpt.generate(**prompt, min_length=min_length, max_length=max_t, do_sample=do_sample)
```

> > FYI, I'm also triggering this error when I use the latest transformers, when running the [BLIP captioning model example](https://github.com/salesforce/LAVIS) on the saleforce lavis library which uses transformers:
> > ```python
> > import torch
> > from lavis.models import load_model_and_preprocess
> > device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
> > # loads BLIP caption base model, with finetuned checkpoints on MSCOCO captioning dataset.
> > # this also loads the associated image processors
> > model, vis_processors, _ = load_model_and_preprocess(name="blip_caption", model_type="base_coco", is_eval=True, device=device)
> > # preprocess the image
> > # vis_processors stores image transforms for "train" and "eval" (validation / testing / inference)
> > image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
> > # generate caption
> > model.generate({"image": image})
> > # ['a large fountain spewing water into the air']
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > In my case I was getting the following error:
> > ```
> >  The following `model_kwargs` are not used by the model: ['encoder_hidden_states', 'encoder_attention_mask'] (note: typos in the generate arguments will also show up in this list)
> > ```
> 
> Same here. Can anyone help fix this?

transformers==4.25.1 will be ok?

No, the `generate` function take a `max_length` argument, and almost always has. 
We don't adapt the library to external ones like `lavis`  so this will not be adressed

I feel like you did not read my reply. I KNOW the generate function uses max_length.

Oh sorry, I was just responding to the question of whether we will change this or not! 
THe `max_new_token` should do what you want, it takes into account the length of the input