Illegal Instruction Error on  `prepare_inputs_for_generation`  -> gpt neo/ j
Author: chris-aeviator
Pull request: no

## Environment info

- `transformers` version: 4.10.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.32
- Python version: 3.8.10
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
Models:

@StellaAthena 

Library:

- text generation: @patrickvonplaten






## Information

Model I am using (Bert, XLNet ...): GPT-neo-2.7B / GPT-J-6B

The problem arises when using:
* [ ] the official example scripts: (give details below)


## To reproduce

Steps to reproduce the behavior:


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


```
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B")

input_ids = tokenizer.encode("the universe is most dense at", return_tensors="pt")
output = model.generate(input_ids, max_length=50)
output = tokenizer.decode(output[0], skip_special_tokens=True)
```
```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Fatal Python error: Illegal instruction

Current thread 0x00007f3ce013e640 (most recent call first):
  File "/home/korny/.local/lib/python3.8/site-packages/transformers/models/gpt_neo/modeling_gpt_neo.py", line 929 in prepare_inputs_for_generation
  File "/home/korny/.local/lib/python3.8/site-packages/transformers/generation_utils.py", line 1288 in greedy_search
  File "/home/korny/.local/lib/python3.8/site-packages/transformers/generation_utils.py", line 989 in generate
  File "/home/korny/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 15 in decorate_context
  File "try.py", line 20 in worker
  File "/usr/lib/python3.8/threading.py", line 870 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007f426b06b740 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 302 in wait
  File "/usr/lib/python3.8/queue.py", line 89 in join
  File "try.py", line 36 in <module>
Illegal instruction (core dumped)
```

I'm getting the same error when using 

```
>>> from transformers import pipeline
>>> generator = pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B')
>>> generator("EleutherAI has", do_sample=True, min_length=50)
```
## Expected behavior
No errors


Comments:
this seems connected to `torch==1.6.0` - the generator works fine with `torch==1.9.0`

## BTW

> the universe is most dense at the center of the galaxy, and the density decreases with distance from the center.

> The density of the universe is measured in units of the critical density, which is the density at which the universe would collapse

## EDIT

feel free to close this if the version requirement was stated in the docs

cc @patil-suraj 

Hi there! I just tried this on colab with torch 1.6 but couldn't reproduce, here's the colab if you want to take a look
https://colab.research.google.com/drive/1weoJYgJaVme7LgpiWRxprr-BY71GG3Cq?usp=sharing

I‘ll try to reproduce in a fresh conda environment - not the first time having strange errors with it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.