GPT2.generate() with custom input_embeds argument returning tensor (1*max_length) instead of (batch_size*max_length)
Author: JustABiologist
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.17.0
- Platform: Linux-4.15.0-175-generic-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Nope
```


### Who can help?

@patrickvonplaten, @patil-suraj

Hi everybody I have a problem/bug to report regarding the .generate() function, when using GPT2 with custom embeddings instead of embedding ID's. 

When using it like stated below with custom input_embeds, the output shape of the longtensor is [1,maxlength] instead of [batch_size, maxlength]. I am confused as to why this happens. 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

import torch.nn as nn
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BartForConditionalGeneration

MOD = AutoModelForCausalLM.from_pretrained("nferruz/ProtGPT2").to('cuda:0')

Out = MOD.generate(input_embeds=torch.rand(3,314,1280).to('cuda:0'), max_length=40, temperature=1.0, repetition_penalty=1.2, top_k=950, num_return_sequences=1, do_sample=True, top_p=1.0)

Out.shape

### Expected behavior

```shell
Expected shape of Out = [batch_size, maxlength]
```


Comments:
Hey @JustABiologist,

I sadly can't run the above code as I don't have access to `"GPT_2 filepath"` - could you make sure that the code snippet is reproducible?

There are two issues here.

1. The argument is called `inputs_embeds` and not `input_embeds`
2. `generate` doesn't support passing `inputs_embeds` for auto-regressive models, cf 
https://github.com/huggingface/transformers/blob/74814574aeab5256ab3c6e428c247739aa0c869d/src/transformers/generation_utils.py#L436-L442

> There are two issues here.
> 
>     1. The argument is called `inputs_embeds` and not `input_embeds`
> 
>     2. `generate` doesn't support passing `inputs_embeds` for auto-regressive models, cf
>        https://github.com/huggingface/transformers/blob/74814574aeab5256ab3c6e428c247739aa0c869d/src/transformers/generation_utils.py#L436-L442

Hey @patil-suraj , sorry for not mentioning, when I use input_embeds the output is the bugish matrix, the decoded output  makes sense though. It just is outputting one sample instead of batch size amount of samples...
When using inputs_embeds it does throw the afformentioned error. 


@patrickvonplaten the filepath is "nferruz/ProtGPT2" I corrected it in the issue! :)

Thanks Guys!

> Sorry for not mentioning, when I use input_embeds the output is the bugish matrix, the decoded output makes sense though. It just is outputting one sample instead of batch size amount of samples.

This is because `input_embeds` is not a valid argument and is ignored. And since there are no `input_ids` provided, `generate` creates one `input_ids` with `bos_token_id` hence the batch size of 1.

@patil-suraj 
Okay so to circumvent  this I should do it like this ?

`config = AutoConfig.from_pretrained("nferruz/ProtGPT2")
model = GPT2LMHeadModel(config)
TRAIN = model.from_pretrained("nferruz/ProtGPT2")
OUT = TRAIN(inputs_embeds=torch.rand(2, 314, 1280))
OUT['logits'].shape`

And write my own sampling/CLM script ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Okay whatever solved the problem on my own ciao cacao.

@JustABiologist could you please share how you made it work with using input_embeds to a decoder-based model like ProtGPT2? I'm trying to implement the same, but I also read that decoder-only models cannot generate with "input_embeds", so I'm wondering how you made it work.
Thanks :)

cc: @patrickvonplaten  @patil-suraj 

@JustABiologist could you please share how you made it work with using input_embeds to a decoder-based model like ProtGPT2? I'm trying to implement the same, but I also read that decoder-only models cannot generate with "input_embeds", so I'm wondering how you made it work.
Thanks :)

cc: @patrickvonplaten @patil-suraj

@hunarbatra sure I basicially just wrote my own generation class on top of the model. That is the most straightforward approach in my opinion. Sorry I can't share the exact code it's for a startup idea.. 

If you want pseudocode I can write that out if needed :)

No worries, I can understand that! Sure, it'll be really helpful if you could share pseudocode for it! Thank you soo much! :) @JustABiologist 

def generate(model, start_embs, generation_parameters, tokenizer):
  for i in range(whatever_iterations)  
    ### raw logit output of the mode 
    out = model(input_embeds = start_embeds)
     ### modify logit output with whatever metod you want beam search whatever
    out = modify_function(generation_parameters
    ### sample token from modified logits
    tok = sample_function(out, generation_parameters)
    ### turn tok into long tensor
    ### turn tok into embedded tok with embedding layer of the model or whatever upstream model you might use
    ### concatenate start_embeds and tok_embeds
    ### feed concat start_embeds + tok_embeds back into model with input_embeds
    ### break when endofline or max toks

You can streamline that alot though :) Do you mind connecting on linkedin? I saw you did some work with proteins aswell so we could discuss ideas!


Thank you so much for sharing this! This is helpful :)

Sure, you can add me on [LinkedIn](https://linkedin.com/in/hunarbatra)