Multiple different models returning only `<unk>` tokens in text generation
Author: serenalotreck
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I've tested the following minimal reproducible example for three models: `ausboss/llama-30b-supercot`, `digitous/Alpacino30b`, and `MetaIX/GPT4-X-Alpasta-30b`.

I originally thought that this issue was related to my prompts not being in the correct format for a given model, as referenced in the comments of #23411. However, I've identified the correct prompt formatting for `ausboss/llama-30b-supercot` and `digitous/Alpacino30b` from their model cards. Additionally, while the prompt format is not explicitly stated in the model card for `MetaIX/GT4-X-Alpasta-30b`, it is also based off the alpaca model, so I would expect the same prompt formatting to work as well.

The example (the only thing that I changed for each model was what string was the `checkpoint` variable):

```
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch


# Load model and tokenizer
checkpoint = 'digitous/Alpacino30b'
model = AutoModelForCausalLM.from_pretrained(checkpoint,
    torch_dtype=torch.float16, device_map='auto', load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# Build prompt
prompt = """
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Extract the biological relations from the following text as (Subject, Predicate, Object) triples in the format ("Subject", "predicate", "Object"):

### Input:
Salmeterol is a long-acting beta2-adrenergic receptor (beta 2AR) agonist used clinically to treat asthma.

### Response:

"""

# Generate predictions
inputs = tokenizer(prompt, return_tensors='pt')
inputs = inputs.to(0)
output = model.generate(inputs['input_ids'], max_new_tokens=500)
response = tokenizer.decode(output[0].tolist())

print(response)
```

The output of this script for all three models gives an identical response:

```

 
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Extract the biological relations from the following text as (Subject, Predicate, Object) triples in the format
 ("Subject", "predicate", "Object"):
### Input:
Salmeterol is a long-acting beta2-adrenergic receptor (beta 2AR) agonist used clinically to treat asthma.
### Response:
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
```

I'm immediately suspicious that I'm getting an identical output from multiple models. I can't load these models without `load_in_8bit`, so I can't check whether or not this is related to the quantization of the models. I also tried running this code with a shorter prompt that contained less instruction, in case it was too complex: "Extract the biological relations from the following text:". However, I once again get an identical output.

### Expected behavior

A response that contains normal tokens, and varies between different models at least somewhat.

Comments:
Same issue here. From that kernel revision it seems like @serenalotreck is running on CentOS 7, possibly within an HPC setup. That's the case with me as well. I've tried several methods of loading / merging but not getting anything from the output apart from `<unk>` tokens for quantized models. For reference, all concerned libraries are at their latest versions, including a few attempts with git versions.

I've requested our sysadmins to update the NVIDIA drivers, but other than that, I'm not sure what to do next.

@akhilvaid that's correct about my kernel revision -- I'm so glad to finally have some answer about what's going on, even if it's a frustrating one!

Do you think rolling back versions could possibly help? I don't have a good sense of how recently the features I'm using were added, so I'm not sure if I'd be able to do that from a, still being able to run the code I have, standpoint, but maybe this was a bug that was recently introduced if all the newest versions don't work?

Also want to tag @gante since this is related to `generate`

Hey! Thanks for reporting this. 
I am not entirely sure about what might be going on here, I would suggest to try running a smaller model without `load_in_8bits` and check if the issue persists. If not, then it might be related to `generate` otherwise, it can be a problem with the instabilities

@akhilvaid just wanted to update you that I got the suggestion from someone on the HPC staff to try running CentOS 9 in a Singularity container, so I'm spending some time trying that today in hopes that it works!

@ArthurZucker How small of a model is small? 😄 

@serenalotreck Unless I'm missing something, containers only really have additional / new libraries installed. That said, I tried the same thing using one of the NGC docker images repurposed into a singularity container with v515 drivers - but the error is persisting.

@ArthurZucker I can successfully use/generate responses with a 13B parameter Vicuna in 16bit on an A100 80G. 33B or greater don't fit into a single GPU's memory - and quantization leads to the same issues as earlier. GPU memory and compute utilization jumps - but only `<unk>` tokens are generated.

@ArthurZucker @akhilvaid I found the same thing -- if I could fit the non-quantized model into memory then it was fine, it's definitely related to the quantization process. However, I only have access to GPUs with 32768MB (~33GB) memory, so I'm even more limited in what I can do without being able to quantize the models.

Do you have any specific suggestions for what to do to try and get around this issue?

Hey @serenalotreck @akhilvaid 👋 

This is not a fix per se, as I don't have a similar setup and can't reproduce the issue. Using transformers 4.30, bitsandbytes 0.39.0, pytorch 2.0.0, and **4 bit quantization** on a single RTX 3090, I get

```
### Response:

("Salmeterol", "is a long-acting beta2-adrenergic receptor (beta 2AR) agonist", "used clinically to treat asthma")

```

This means that the error is likely related to 8 bit quantization or to your setup. Using 4 bit quantization may solve the issue 🙌 

Let us know about further developments on your end :)

______________________________________________

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
checkpoint = 'digitous/Alpacino30b'
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map='auto', load_in_4bit=True)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# Build prompt
prompt = """
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Extract the biological relations from the following text as (Subject, Predicate, Object) triples in the format ("Subject", "predicate", "Object"):

### Input:
Salmeterol is a long-acting beta2-adrenergic receptor (beta 2AR) agonist used clinically to treat asthma.

### Response:

"""

# Generate predictions
inputs = tokenizer(prompt, return_tensors='pt')
inputs = inputs.to(0)
output = model.generate(inputs['input_ids'], max_new_tokens=500)
response = tokenizer.decode(output[0].tolist())

print(response)
```

@gante thank you!

@akhilvaid Curious to know what happens when you run this code.

Something whacky is happening on my end -- the code aborts at trying to load the model (after successfully downloading the shards). When I had `load_in_4bit=True`, it didn't print anything, and when I removed `load_in_4bit=True`, it printed out half of a message:

```
lerate` to properly deal with them (`pip install --upgrade acc  .cuda  
```
I'm working in a conda environment so I ran `conda upgrade accelerate` to see if that would help, accelerate was successfully upgraded, but I still got the same weird half-message.

When I change the model to `ausboss/llama-30b-supercot`  and include `load_with_4bit`, I get a different part-message:
```
d(init_empty_weights()) 
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am facing the same issue, for some inputs in the batch. So I tried to generate only the specific input that generated <unk>, and it generated fine. In other words, my problem arises in batched generation.