greedy beam search generates same sequence N times
Author: rajarsheem
Pull request: no

## ❓ Questions & Help

I am doing greedy beam search (without sampling to avoid randomness) using GPT-2. However, all the returned sequences are same. Why is that the case? Shouldn't it give N best and different sequences?

```python
model = GPT2LMHeadModel.from_pretrained('gpt2-medium').cuda()
tokenizer = GPT2Tokenizer.from_pretrained('gpt2-medium')

input_context = 'The dog'
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0).cuda()
outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3)
for i in range(3):
    print('Generated {}: {}'.format(i, tokenizer.decode(outputs[0][i], skip_special_tokens=True)))
```

The resulting output is:
```
Generated 0: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
Generated 1: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
Generated 2: The dog was taken to a veterinary clinic for treatment.

The dog's owner said the!
````

Comments:
Took me most of the day to figure this out, set the `do_sample` arg to `True`

```
outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, do_sample=True)
```

Actually, I don't want to do sampling because that is random and would give different results each time I run for the same prompt.

I am looking for greedy beam search which should be able to give the static top sequences (which is not happening)! Sad.

In your case your have 3 parallel beam search going on with a beam of 5 in each case.

But the current beam search only returns the top beam in each case, we don't have an option to return all beams.

Thanks Thomas. Does that mean `num_return_sequences` is only useful when `do_sample` is `True`?

That's a good point.

We could probably take the `num_return_sequences` top beams in the case of having beam search + greedy decoding otherwise this option is not useful in this case.


Thanks @thomwolf for the clarification. So, in case of greedy decoding, you would do beam search only once and take the top ```num_return_sequences``` ones. 

Any rough ETA you have?


No ETA, but if you need it now, feel free to make a PR and I or Lysandre will give a look

Hi, I'm also interested in this feature - did anyone attempt to implement greedy beam search that returns multiple sequences? (Alternatively, has a an idea of which part of the function should be fixed, so I can try it myself)? Thanks!

See PR #3078 for how this feature is implemented.

The following example: 
```
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

input_context = 'The dog'
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0)
outputs = model.generate(input_ids=input_ids, num_beams=20, num_return_sequences=3, do_sample=False)
for i in range(3):
    print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True)))
```

would produce:

```
Generated 0: The dog was taken to a local hospital, where he was pronounced dead.

The dog was
Generated 1: The dog was taken to a local hospital, where it was treated and released.

The dog
Generated 2: The dog was taken to a local hospital where he was pronounced dead.

The dog's owner
```



Thanks @patrickvonplaten for your effort in this. 

@patrickvonplaten It helps a lot!!

> Took me most of the day to figure this out, set the `do_sample` arg to `True`
> 
> ```
> outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, do_sample=True)
> ```

what about for HF pipelines? 



related: https://stackoverflow.com/questions/78466376/how-to-generate-multiple-text-completions-per-prompt-like-vllm-using-huggingfa
related: https://discuss.huggingface.co/t/how-to-generate-multiple-text-completions-per-prompt-like-vllm-using-huggingface-transformers-pipeline-without-triggering-an-error/86297

cc @gante maybe a slight doc update would be nice if this is still cause for problems!