Repetitive sampling generations from opt1.3b but not from opt350m
Author: ZhangShiyue
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.4.0-104-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@LysandreJik, @patrickvonplaten 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import OPTForCausalLM, GPT2Tokenizer

prompt = ["Hey, are you consciours? Can you talk to me?",
          "Hey, are you consciours? Can you talk to me?",
          "Hey, are you consciours? Can you talk to me?",
          "Hey, are you consciours? Can you talk to me?",
          "Hey, are you consciours? Can you talk to me?"]

print("=====OPT 1.3b Sampling=====")
model = OPTForCausalLM.from_pretrained("facebook/opt-1.3b")
tokenizer = GPT2Tokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer(prompt, return_tensors="pt", padding=True)
generate_ids = model.generate(inputs=inputs.input_ids, attention_mask=inputs.attention_mask,
                              max_new_tokens=100, do_sample=True, temperature=1.0)
print(tokenizer.batch_decode(generate_ids, skip_special_tokens=True))
'''
Generations are repetitive, e.g.,
["Hey, are you consciours? Can you talk to me? Please?\nI'm sorry but I'm afraid I'm incapable of talking to you :(", 
'Hey, are you consciours? Can you talk to me?\nYeah sure thing buddy! Whats your discord?', 
"Hey, are you consciours? Can you talk to me?\nI'm sorry I'm not consciours :(", 
'Hey, are you consciours? Can you talk to me? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please?', 
'Hey, are you consciours? Can you talk to me? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please? Please?']
'''

print("=====OPT 350m Sampling=====")
model = OPTForCausalLM.from_pretrained("facebook/opt-350m")
tokenizer = GPT2Tokenizer.from_pretrained("facebook/opt-350m")
inputs = tokenizer(prompt, return_tensors="pt", padding=True)
generate_ids = model.generate(inputs=inputs.input_ids, attention_mask=inputs.attention_mask,
                              max_new_tokens=100, do_sample=True, temperature=1.0)
print(tokenizer.batch_decode(generate_ids, skip_special_tokens=True))
'''
Generations look normal, e.g.,
["Hey, are you consciours? Can you talk to me?\nThis is a repost, stop doing it lol\nI'm not trying to repost.... it's just happened to me like that, so I thought I'd do it, too.", 
"Hey, are you consciours? Can you talk to me? A few months ago there seemed to be a large influx of new and existing consciourns around Melbourne. All we see at the top are people wanting to participate. They've got no skills, no money, no interest and they're not interested in anything other than the show and being popular. I'm not even kidding about that!\n\nI understand that the show is quite difficult and is full of competition, but does your hobby really need such an appeal?\n\nFor example, my", 
"Hey, are you consciours? Can you talk to me?\nThe reason I can't talk to him is because I'm in a hurry. He's at the airport, waiting for his driver to come back (he doesn't drive, and he's not comfortable being in someone's car and trying to take it with him). I think I'm overthinking this one.\nIf you need someone to talk to, please PM me.", 
"Hey, are you consciours? Can you talk to me?\nNo I'm not\nMy mistake, I've been looking around for a little while and don't have a chance to look up the exact wording.  I just noticed your 'cognizant'.\nno worries and yea it's a common one so let's move on haha", 
"Hey, are you consciours? Can you talk to me? I'm not a cop, but I've dealt with one or two\nOf course. I am in Texas, but we're not all like that."]
'''
```

### Expected behavior

```shell
Sampling generations from opt1.3b are often very repetitive (see the example above). 
It does not seem to happen by chance. If you run the code multiple times, similar patterns will always appear. 
It is unexpected because usually standard sampling should produce diverse results. 
Interestingly, I did not see this issue when using opt350m.
```


Comments:
Interesting, thanks for opening the issue! 

@stephenroller @suchenzang have you see similar behavior with the fairseq model ? Could this be due to a bug in porting the model?

One thing I'd like to add here is that we enable `topk=50` by default -> does changing this value maybe help? But it indeed looks like a modeling issue

Related issue https://github.com/facebookresearch/metaseq/issues/136

Should be fixed now in https://github.com/huggingface/transformers/releases/tag/v4.20.1

Thanks for the update! After this patch, it should be able to pass the end-to-end regression test between metaseq and huggingface (https://github.com/facebookresearch/metaseq/issues/136)? I dug through the convo and it seems this is on @stephenroller @ArthurZucker @thomasw21 's radar? ❤️ 

It would be great if we can load the model into metaseq directly without merging if possible, so that we can catch subtle bug in conversion.

thanks so much for fixing it! @patrickvonplaten