causalLM Text generation with OPT models give weird results
Author: SteffenBauer
Pull request: no

### System Info

* Platform: NVIDIA Jetson Orin 32GB development kit
* CPU `ARMv8 Processor rev 1 (v8l)`
* Linux `5.10.104-tegra #1 SMP PREEMPT Sun Mar 19 07:55:28 PDT 2023 aarch64 aarch64 aarch64 GNU/Linux`
* Python 3.8.10
* transformers 4.29.2
* torch `2.0.0a0+fe05266f.nv23.4` (compiled package for Jetson platform provided by Nvidia)

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

In my first attempts to run causalLM text generation with huggingface transformer models, I get weird and most surely wrong results with the OPT model family.

I run the official example from the huggingface OPT documentation:

https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTForCausalLM.forward.example

```
~$ python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoTokenizer, OPTForCausalLM
>>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m")
Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 644/644 [00:00<00:00, 239kB/s]
Downloading pytorch_model.bin: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 663M/663M [00:51<00:00, 12.8MB/s]
Downloading (…)neration_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 58.7kB/s]
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
Downloading (…)okenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 685/685 [00:00<00:00, 600kB/s]
Downloading (…)olve/main/vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 899k/899k [00:00<00:00, 2.33MB/s]
Downloading (…)olve/main/merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 1.61MB/s]
Downloading (…)cial_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 441/441 [00:00<00:00, 324kB/s]
>>> prompt = "Hey, are you consciours? Can you talk to me?"
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
'Hey, are you consciours? Can you talk to me?














'
```

For OPT-1.3b I get the following, repeatedly:
```
>>> model = OPTForCausalLM.from_pretrained("facebook/opt-1.3b")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
>>> inputs = tokenizer(prompt, return_tensors="pt")
>>> generate_ids = model.generate(inputs.input_ids, max_length=30)
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
'Hey, are you consciours? Can you talk to me? it it it it it it it it it it it it it it it'
```

Any other prompts give the same result always, just `
` tokens or repeated `it`.

### Expected behavior

OPT model causalLM text generation should return text that not consists of only `
` tokens or repeated `it`, like in the given documentation example:

```
>>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"Hey, are you consciours? Can you talk to me?\nI'm not consciours, but I can talk to you."
```

Comments:
cc @younesbelkada @gante 

hi @SteffenBauer 
thanks for the issue!
I just checked on our daily CI tests that also checks the generations on OPT and there seem to be no issue on our side. What I can see is that you are using a version of torch that sounds a bit exotic `2.0.0a0+fe05266f.nv23.4`
Does this behavior happen with OPT only or with other models as well?

Hi @younesbelkada 
thanks for checking this! 

The torch version is a special package provided by Nvidia, compiled with platform-dependent patches for the Jetson platform. (https://developer.nvidia.com/embedded/downloads)

Nvidia released yesterday an update to torch-2.0.0 for Jetson, but the problem still persists unchanged.

As it works on your environment, it strongly hints that the problem is Jetson-platform specific. I will investigate this further.

Could also be connected with issue #23413, where an issue in torch might be the problem.

I will close this now, as it doesn't seem to be an issue with the transformers library.


Thank you very much @SteffenBauer , please let us know how it goes

Hey @SteffenBauer 👋 

I can confirm that things seem to be fine on our end, using the latest release (v4.30). Have a look at [this notebook](https://colab.research.google.com/drive/11iSgaa6j0S9jbIMg2C9iqUTfhHEtFebq?usp=sharing) :)

Hi @gante ,

thanks for confirming, your colab notebook also works here fine with expected result. 
As I copied the example code verbatim to a python shell on the Jetson Orin, the problem now is confirmed to be on the side of the platform.
I hadn't time to investigate this further, no clue so far if it caused by the ARM64 architecture, or the Nvidia provided PyTorch library. As the Nvidia Jetson platform is an important tool for DL practitioners, this should indeed be looked into more deeply.

Will update here once I have results.

@SteffenBauer can you set the device with PyTorch's `.to()`, using Jetson? If so, you could try running the model on CPU and GPU to determine whether the problem is exclusive to a particular device or at a software level.

Opening an issue with Nvidia may help too :)

Hi @gante ,
you might have found something! There is indeed a difference between CPU and GPU.

(by the way, I just upgraded the transformer library to 4.30.2 before doing this test)

I use this code, directly copied from the notebook you supplied:

```python
from transformers import OPTForCausalLM, AutoTokenizer
import torch

device = 'cpu' # 'cuda'

model = OPTForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
prompt = "Hey, are you consciours? Can you talk to me?"

inputs = tokenizer(prompt, return_tensors="pt")
generate_ids = model.generate(inputs.input_ids.to(device), max_length=30)
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
```
`device = 'cpu'`:
 "Hey, are you consciours? Can you talk to me? it it it it it it it it it it it it it it it"

`device = 'cuda'`:
"Hey, are you consciours? Can you talk to me?\nI'm not a conscioure, but I can talk to"


@SteffenBauer On my end, using an x86 CPU, I get the same outputs as your CUDA output above. This seems to point to a PyTorch+ARM64 issue 🤔  

@gante I'm referring to your notebook with opt-350m instead and here is the result of text-generation, which is the model just addedl blank tokens to get equal length to the max_length instead of some real texts. Could you please give me some insights?

![image](https://github.com/user-attachments/assets/8e29f6bf-5709-4802-89d7-471a8f02e4d1)


Hi @daminho 👋 

`opt-350m` is a small and relatively old model, it is not entirely unexpected the behavior you're seeing. I heavily recommend using a newer model.

Nevertheless, I've ran the model on my end, using the latest version of `transformers`, and it seems to be working fine:

```py
from transformers import OPTForCausalLM, AutoTokenizer

model = OPTForCausalLM.from_pretrained("facebook/opt-350m")
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m", device_map="auto")
prompt = "Tell me about the french revolution."

inputs = tokenizer(prompt, return_tensors="pt")
generate_ids = model.generate(inputs.input_ids.to(model.device), max_new_tokens=128, do_sample=True)
print(tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0])
```