What does "is_beam_sample_gen_mode" mean 
Author: huhk-sysu
Pull request: no

Hi, I find there are many ways for generating sequences in `Transformers`(when calling the `generate` method).
According to the code there:
https://github.com/huggingface/transformers/blob/01f8e639d35feb91f16fd3c31f035df11a726cc5/src/transformers/generation_utils.py#L947-L951
As far as I known:
`is_greedy_gen_mode` stands for Greedy Search.
`is_sample_gen_mode` stands for Sampling(with top_k and top_p).
`is_beam_gen_mode` stands for Beam Search .

But what does `is_beam_sample_gen_mode` mean?

Besides, I want to know how do I choose the correct way for generating. I have tried serval ways, but:
1. I find the sequences out from "beam search" mode becomes too similar.
2. I also find the sequences out from "sample" mode, while being diverse, are lacking context coherence.

Thank you!

Comments:
Hi,

A good overview can be found in [this blog post](https://huggingface.co/blog/how-to-generate).

It explains the most prominent decoding methods, mainly Greedy search, Beam search, Top-K sampling and Top-p sampling.

I'm not sure "beam sampling" exists.

> Hi,
> 
> A good overview can be found in [this blog post](https://huggingface.co/blog/how-to-generate).
> 
> It explains the most prominent decoding methods, mainly Greedy search, Beam search, Top-K sampling and Top-p sampling.
> 
> I'm not sure "beam sampling" exists.

Thanks for your response. 

First, to be clear, according to the code there:
https://github.com/huggingface/transformers/blob/01f8e639d35feb91f16fd3c31f035df11a726cc5/src/transformers/generation_utils.py#L947-L951
`is_greedy_gen_mode` stands for Greedy Search.
`is_sample_gen_mode` stands for Sampling(with top_k and top_p).
`is_beam_gen_mode` stands for Beam Search .

But what does `is_beam_sample_gen_mode` mean?

In fact, I have already read the post, and learned concepts like Beam Search and Sampling(along with top_k and top_p) from it.
And that's why I'm confused: the post showed me that Beam Search and Sampling is two different way for generating. And I can't image what's "beam sample gen mode" looks like.

Hi,

Looking at the [docs](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.beam_sample) of beam sample, it means beam search with multinomial sampling.



> Hi,
> 
> Looking at the [docs](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.beam_sample) of beam sample, it means beam search with multinomial sampling.

Hi, I have checked the docs, but I need something more specific. 
Assumes `num_beams`= 5,  and there are 100 words in the vocab.
Does `beam sample` means at each time step:
1. Firstly generate 5 * 100 = 500 possible sequences, calculate and normalize their scores.
2. Secondly use Multinomial Sampling to randomly choose 5  sequences as output.

BTW I read the source code but fail to understand.
In the following code, why does it sample `2 * num_beams`(instead of `num_beams`) times in line 2152?

https://github.com/huggingface/transformers/blob/69e16abf98c94b8a6d2cf7d60ca36f13e4fbee58/src/transformers/generation_utils.py#L2150-L2159


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> > Hi,
> > Looking at the [docs](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.beam_sample) of beam sample, it means beam search with multinomial sampling.
> 
> Hi, I have checked the docs, but I need something more specific. Assumes `num_beams`= 5, and there are 100 words in the vocab. Does `beam sample` means at each time step:
> 
> 1. Firstly generate 5 * 100 = 500 possible sequences, calculate and normalize their scores.
> 2. Secondly use Multinomial Sampling to randomly choose 5  sequences as output.
> 
> BTW I read the source code but fail to understand. In the following code, why does it sample `2 * num_beams`(instead of `num_beams`) times in line 2152?
> 
> https://github.com/huggingface/transformers/blob/69e16abf98c94b8a6d2cf7d60ca36f13e4fbee58/src/transformers/generation_utils.py#L2150-L2159

halo，do u understand details about this "is_beam_sample_gen_mode" now ?

What exactly is beam search with multinomial sampling? Googling returns no relevant results (except on Huggingface).

Ok. I think beam sample is almost the same as beam search, except that in each step, instead of picking the `num_beams` most-likely sequences, it chooses these sequences by sampling.

> Ok. I think beam sample is almost the same as beam search, except that in each step, instead of picking the `num_beams` most-likely sequences, it chooses these sequences by sampling.

I think @shunzh is right. And another difference is that when setting `num_return_sequences > 1`, the `beam_sample` will be done for  `num_return_sequences` times and will sample 1 sequence every time, while the `beam_search` will be done for only 1 time and will return top-`num_return_sequences` sequences.

> 2 * num_beams

refer to [https://github.com/huggingface/transformers/issues/16095#issuecomment-1071123601](https://github.com/huggingface/transformers/issues/16095#issuecomment-1071123601)