Pipeline GPT-NeoX only returns "BB" from any prompt then nothing for subsequent calls of the pipeline
Author: jaimu97
Pull request: no
Labels: name: bug

### System Info

CPU: AMD 4750G (Onboard video disabled)
OS: Ubuntu Server 20.04.5 LTS x86_64
RAM: 64GB
GPUs: 2x Tesla M40 24GB
Driver Version: 510.47.03
CUDA Version: 11.6

Pytorch stable 1.12.1 (Installed via Anaconda)
Both accelerate and transformers installed through pip
accelerate @ git+https://github.com/huggingface/accelerate@98823de572246d68cb31db94b60f7328ae9d551e
transformers @ git+https://github.com/huggingface/transformers@cfd623a859890c6d106610d3c688064eadc7bd61

### Who can help?

@patil-suraj @Narsil

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi everyone! 

I'm trying to run GPT-NeoX 20B with accelerate and `device-map="auto"` however, I can't seem to get the model to return anything other than "BB" for the first response and then nothing but an empty string after.

Steps to reproduce the behaviour:

1. Use latest git of `accelerate` and `transformers`
2. Setup a pipeline of `EleutherAI/gpt-neox-20b`
3. Call pipeline with any prompt

<details>
<summary>Code I am using</summary>
<pre>
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch, accelerate

print("Loading generator!")

generator = pipeline('text-generation',
                     model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b", device_map="auto",
                                                                  torch_dtype=torch.float16),
                     tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b"),
                     temperature=0.7, return_full_text=False, max_new_tokens=250)

print("Loaded generator!")

while True:

    prompt = input("Enter prompt: ")
    response = generator(prompt)
    print(response)

</pre>
</details>

<details>
<summary>Example runs</summary>
<pre>

(hf) jai@tesla-server:~$ python gpt-neox-test2.py 
Loading generator!
Loaded generator!
Enter prompt: Test!
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
[{'generated_text': 'BB'}]
Enter prompt: Testing again!
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
[{'generated_text': ''}]
Enter prompt: One more time!
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
[{'generated_text': ''}]


(hf) jai@tesla-server:~$ python gpt-neox-test2.py 
Loading generator!
Loaded generator!
Enter prompt: This is a different prompt. 
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
[{'generated_text': 'BB'}]
Enter prompt: yet every time the responses are still the same. :(
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
[{'generated_text': ''}]
</pre>
</details>

I'm not sure where this issue belongs as I'm still new to huggingface, but if I make a small change and use a model that can fit into a single card such as `EleutherAI/gpt-j-6B` and remove `device_map="auto"` there is no issue with generation (still with piplines).

I have also tried using the `GPTNeoXTokenizerFast` class with the same results.

There is no errors as in python crashing it just doesn't generate anything meaningful.

### Expected behavior

Response should be at the length of `max_new_tokens` (250) or at least more than one token and relevant to prompt provided.
`[{'generated_text': 'BB'}]`

Comments:
I noticed also changing the `eos_token_id` to `187` (`\n`) increases time for a response to about 20 seconds (previously ~5s) and there is an increased load on the cards but the response is the same "BB"

cc @Narsil 

I unfortunately don't have a machine at hand big enough to run that code.

Does this happen with any other (smaller) model that we can try on ?

If not, the ideal thing would be to check if the problem is the `eos_token_id` being generated or not.
Using `pipeline(..., eos_token_id=None)` should deactivate it, and your generation should now actually generate 250 tokens.

The other options is that it DOES generate tokens, but they are somehow removed by the decoding process. In order to check for that I would add a `print` statement directly into the `postprocess` method of `TextGenerationPipeline` and see what's going on here.

Would that help ?

Hi, I actually found that this was a hardware issue and forgot to close this issue. Any time I ran something that required both cards to work together I would get an `IO_PAGE_FAULT` error and to fix it I needed to disable IOMMU in my motherboard settings now it works. 😅