Add feature #1572 which gives support for multiple candidate sequences
Author: enzoampil
Pull request: yes

**Multiple candidate sequences can be generated by setting ```num_samples  >  1``` (still 1 by default).**

EXAMPLE with ```num_samples == 2``` for a GPT2 model:

```
INPUT:
Why did the chicken

OUTPUT:
 cross the road <eoq>  To go to the other side. <eoa>  
 eat food <eoq>  Because it was hungry <eoa>

```
(above is illustrative w some words changed from the actual output)

**UPDATE:** Multiple candidate sequences can now be generated with _repetition penalty_ and ```top_k_top_p_filtering``` applied separately to each candidate. This allows for independent probability distributions across candidate sequences.

~Samples are generated with replacement to allow for sequences that have similar tokens at the same index (e.g. [CLS], stopwords, punctuations).~

~When ```temperature == 0```, the tokens returned are the top ```num_samples``` logits (first sample gets top 1, second gets top 2, and so on). I realize this might not be the best implementation because it doesn't allow for similar tokens at the same index across samples. I will consider later changing this to just returning ```num_samples``` copies of the top 1 logits (argmax).~

Comments:
Please note that main() now returns a list with ```num_samples``` elements inside.

Because of this, the test for run_generation.py should be updated to test for ```length``` for each element within the list. This explains why **build_py3_torch** test failed.

I will update ```ExamplesTests.test_generation``` to reflect the new output format.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1577?src=pr&el=h1) Report
> Merging [#1577](https://codecov.io/gh/huggingface/transformers/pull/1577?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ef1b8b2ae5ad1057154a126879f7eb8de685f862?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1577/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1577?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1577   +/-   ##
=======================================
  Coverage   86.17%   86.17%           
=======================================
  Files          91       91           
  Lines       13595    13595           
=======================================
  Hits        11715    11715           
  Misses       1880     1880
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1577?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1577?src=pr&el=footer). Last update [ef1b8b2...17dd64e](https://codecov.io/gh/huggingface/transformers/pull/1577?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Latest commit 17dd64e now applies repetition penalty and ```top_k_top_p_filtering``` to each candidate sequence separately.

Thanks @enzoampil 

Superseded by https://github.com/huggingface/transformers/pull/1333 which was just merged to master.

Let me know if this works for your use case.