CTRL does not react to the "seed" argument
Author: igorwolford
Pull request: no

Using latest master branch and run_generation.py.
When passing different seeds to GPT-2 it produces different results, as expected,
however CTRL does not.

**GPT-2** 
python run_generation.py **--seed=42** --model_type=gpt2 --length=10 --model_name_or_path=gpt2 --prompt="Transformers movie is"
**>>> a great example of bringing early, action-packed**

python run_generation.py **--seed=22** --model_type=gpt2 --length=10 --model_name_or_path=gpt2 --prompt="Transformers movie is"
**>>> a lot of fun. It's a decent one**

**CTRL**
python run_generation.py -**-seed=42** --model_type=ctrl --length=10 --model_name_or_path=ctrl --temperature=0 --repetition_penalty=1.2 --prompt="Links Transformers movie is"
**>>> a big deal for the company, and it has been**

python run_generation.py **--seed=22** --model_type=ctrl --length=10 --model_name_or_path=ctrl --temperature=0 --repetition_penalty=1.2 --prompt="Links Transformers movie is"
**>>> a big deal for the company, and it has been**

Comments:
Noticed the same thing on the Salesforce version as well. 

Do you test with [this url](https://github.com/salesforce/ctrl/blob/master/generation.py)?
@tanselmi If not, please specify which script do you try out

N.B. if you see this code, the lines 40-41-42-43 assign the seed passed as argument (default value is 1337) for NumPy, PyTorch and CUDA. It's strange that the internal seed doesn't modify the CTRL's model output.

> Noticed the same thing on the Salesforce version as well.

In general, is it a bug in the code implementation of the CTRL model or a "model property" (i.e. the CTRL model is not affect by a random seed set at the start of the process) ?

Hi, this is not due to a bug, this is due to your argument `temperature=0`, which implies greedy sampling via argmax, rather than polynomial sampling like it is usually done.

You can try it out with a temperature of 0.2:

With seed 42 - Links Transformers movie is **a huge hit in the box office and has been**
With seed 22 - Links Transformers movie is **a big deal for the studio. The first film was**

Thanks for the update and explanation. I will give it a go.

On Mon, Nov 4, 2019, 08:37 Lysandre Debut <notifications@github.com> wrote:

> Hi, this is not due to a bug, this is due to your argument temperature=0,
> which implies greedy sampling via argmax, rather than polynomial sampling
> like it is usually done.
>
> You can try it out with a temperature of 0.2:
>
> With seed 42 - Links Transformers movie is *a huge hit in the box office
> and has been*
> With seed 22 - Links Transformers movie is *a big deal for the studio.
> The first film was*
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/1710?email_source=notifications&email_token=ABO5AYW55TLAXKGM2PODQJTQSAXSZA5CNFSM4JIMFK42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC7OTJY#issuecomment-549382567>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABO5AYWK6JXQUIHOLWNNXG3QSAXSZANCNFSM4JIMFK4Q>
> .
>


@LysandreJik I confirm setting temperature to non-zero makes the seed argument work fine.
Thank you!