Can we generate multiple possible sentences using GPT?
Author: zhaoxy92
Pull request: no

Hi,
Is there any way to generate multiple candidate text sequences using the pretrained generators?

Comments:
@zhaoxy92 I happen to have a use case for this as well. I'll add in this feature to the ```run_generation.py```

@zhaoxy92  Added this functionality in ```run_generation.py```. You can set the number of candidates generated by setting the argument ```num_samples``` which is set to 1 by default.

I think you need to change `top_k_top_p_filtering()` as well.

@s-js not sure why we'd have to change```top_k_top_p_filtering()``` since the sampling only happens at ```sample_sequence()```. ```top_k_top_p_filtering()``` only filters the logits, so we can still generate multiple candidate sequences (independent of the filtered distribution).

@enzoampil Sorry, I meant repetition penalty (https://github.com/enzoampil/transformers/blob/7facbbe9871fe458b530ae8ce1b4bfefabd47c74/examples/run_generation.py#L142). Each sample has a different set of seen tokens.
At first I thought you were doing it inside `top_k_top_p_filtering()`.

Hi, thanks very much for adding this functionality – I'm trying to implement this into my own notebook and hitting a tensor mismatch error I can't figure out. I hope this is the right forum to post this question, since I'm using the new functionality you created.

At line 150: `generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)`

I'm getting this error:
```RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 1 and 3 in dimension 0 at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/generic/THCTensorMath.cu:71```

The debugger shows me the sizes:
generated = (3,37)
next_token.unsqueeze(0) = (1,3)

So I figure that next_token tensor shape ought to be (3,1) instead, so I tried changing the line to `next_token.unsqueeze(1)` instead. When I do that I get a `CUDA error: device-side assert triggered`. Did that change fix my problem or just cause a new one?

Any ideas are greatly appreciated, thank you!

hi @buttchurch , did you run ```run_generation.py``` (with the multiple sentence functionality) as a CLI? It should work if you run it from my [fork](https://github.com/enzoampil/transformers/blob/7facbbe9871fe458b530ae8ce1b4bfefabd47c74/examples/run_generation.py#L142). Can you please post here the exact script you ran and the complete error message. 

Also, my pull [request](https://github.com/enzoampil/transformers/blob/7facbbe9871fe458b530ae8ce1b4bfefabd47c74/examples/run_generation.py#L142) shows ```generated = torch.cat((generated, next_token.unsqueeze(1)), dim=1)``` in the last line of ```sample_sequence``` so I'm not sure where you got that line 150 code snippet.

@s-js noted on repetition penalty support. I'll try to find time for this within the next week.

hi @enzoampil, thanks for such a quick response! I still don't understand navigating git forks and branches and the different versions of git projects very well, so I have been just going off the main code I find in the transformers github.

It's probably not the 'right' way to do it, but I've pulled my own jupyter notebook together from a couple of the transformer example.py files, rather than using the run_generation.py. I think it might be way too long to post here, but I will now try implementing the changes in your fork to my notebook. I'll report back – thanks again for your help, and for creating this new functionality :)

Edit: It works! Seems like the important bit I was missing was `replacement=True` on the previous line.

@buttchurch glad it works for you :) Very welcome!

@s-js Latest [commit](https://github.com/huggingface/transformers/commit/17dd64ed939e09c1c9b1fa666390dd69a4731387) now implements _repetition penalty_ and ```top_k_top_p_filtering``` separately per candidate sequence generated.

We just merged https://github.com/huggingface/transformers/pull/1333 to master (+ subsequent fixes), can you check that it does what you guys want?

I'll close the issue for now, re-open if needed.