Getting only 
 token  in GPT-NEOX-20B model
Author: pragnakalpdev11
Pull request: no
Labels: name: bug

### System Info

Transformer Version: 4.20.1
Python : 3.8
ubuntu : 18.04

### Who can help?

@patil-suraj  



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

 - We are creating a streaming app using HF transformer generate() but during the token decoding we are only getting " 
 " . when an input prompt is passed, No tokens are generated from the model. The model we are using is GPT-NEOX-20B.

- We have tried 3 Tokenizers, GPTNeoXTokenizerFast, AutoTokenizer, and GPT2TokenizerFast but all of them returned the same output.
``
- Below is the generate parameters.
```
output_sequences = model.generate(
        input_ids=input_ids,
        max_length=200,
        temperature=temperature,
        top_k=k,
        top_p=p,
        repetition_penalty=repetition_penalty,
        do_sample=False,
        num_return_sequences=num_return_sequences,
        filename=filename,
        tokenizer=tokenizer,
        num_beams=1,
        use_cache=False
        
    )
```

-  where we are receiving Empty tokens, is https://github.com/huggingface/transformers/blob/main/src/transformers/generation_utils.py line 1740, 

We are trying to decode the next tokens.

# finished sentences should have their next token be a padding token
          ```
  if eos_token_id is not None:
                if pad_token_id is None:
                    raise ValueError("If `eos_token_id` is defined, make sure that `pad_token_id` is defined.")
                next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
                
            tensor_text = tokenizer.decode(next_tokens, clean_up_tokenization_spaces=True)

```
- A confirmation for the code in file https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py 
and line 146. what logic should be there. if use or if not? 

 `present = None if  use_cache else (key, value)`


### Expected behavior

The above code works well with all the models of the GPT-NEO  but with GPT-NEOX we are facing the 
 tokens issues. Instead of 
 tokens.  we need to generate the correct tokens.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@pragnakalpdev11 (or anyone else running into this issue) when you load the model, are you specifying a `torch_dtype`, and/or converting it to another dtype after loading it (e.g. `.half()`, `.to(torch.float16)`)? And what type of device are you running the generation on (`cuda`, `cpu`, etc.)?

This was actually the first symptom I noticed that eventually led to opening [issue 34692](https://github.com/huggingface/transformers/issues/34692) - just about any GPTNeoX-based model I ran on the CPU in `torch.float16` form would only ever output `
`. If that describes your situation, try loading it as `torch.float32` and not converting it to another dtype and see if that helps.