Llama2 model not loading, stuck in infinite loop
Author: AmritaBh
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-4.18.0-348.el8.0.2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: only via `device_map='auto'`



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

### Trying to load Llama2 in 4-bit using bnb but model load gets stuck in an infinite loop (check attached error log)
Seems like the model config loads properly but the model itself doesn't load at `.from_pretrained(...)`.  No errors are thrown, I had to interrupt execution. 

```python
import torch
import transformers
transformers.logging.set_verbosity_debug()

model_id = 'meta-llama/Llama-2-13b-chat-hf'
hf_auth = <MY-HF-AUTH-TOKEN>

bnb_config = transformers.BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model_config = transformers.AutoConfig.from_pretrained(
    model_id,
    force_download=True,
    token=hf_auth
)

model = transformers.AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    config=model_config,
    quantization_config=bnb_config,
    device_map='auto',
    token=hf_auth
)

```
### Error log file 
[llama2_13b_error.log](https://github.com/huggingface/transformers/files/12424911/llama2_13b_error.log)

### Additional Hardware Details

Executed on two A100s, each with 40G memory.



### Expected behavior

Model should load

Comments:
cc @younesbelkada and @SunMarc 

Hi @AmritaBh 
in the traceback you shared I see:

```bash
Traceback (most recent call last):
  File "llama2_13b_bnb.py", line 277, in <module>
    generate_ids = model.generate(inputs.input_ids, max_new_tokens=max_tokens, **generation_kwargs)
```
Meaning there might be some missing code in the shared snippet as it seems you are generating some text? 

( I am mostly curious about the Config being printed 10x)

Okay I did some more digging and it looks like model **does** load, but the Config gets printed way too many times (I guess multiple times for each call to `model.generate(...)`?) which had me confused that it's stuck in a loop. 

For my use case, I am iterating over `N` data samples for a particular task I have, calling `model.generate(...)` three times for each data sample, and processing the responses. For my previous runs, `N` was too large (around 1000) and my job was timing out without any results. I tried with a smaller `N=50` and it did finish (albeit quite slow), but the Config gets printed in the log way too many times. Please see the attached log file below for this run. 

This is making the error logs difficult to read and understand. I assume this is not the expected behavior?


In addition to the model loading code above, here's the generation code structure I am using:

### Generation code (the basic structure for each inference)

```python
import gc

MAX_TOKENS_CONTEXT = 4096

generation_kwargs = {
    "top_p": 1.0,
    "temperature": 0.4,
    "do_sample": True,
    "repetition_penalty": 1.1
}

tokenizer = transformers.AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_auth)

context = "some long prompt with data and instructions"

context_token_count = count_tokens(context)  ## function that uses tiktoken to get approx count of tokens
max_tokens = MAX_TOKENS_CONTEXT - context_token_count - 128
inputs = tokenizer(context, return_tensors="pt")
generate_ids = model.generate(inputs.input_ids, max_new_tokens=max_tokens, **generation_kwargs)
gen_tokens = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
response = gen_tokens[0][inputs.input_ids.shape[-1]:]

gc.collect()
torch.cuda.empty_cache()
```

### Log file for new run with 150 `model.generate(...)` calls (`N=50`)
[llama2_13b_error_n50.log](https://github.com/huggingface/transformers/files/12433202/llama2_13b_error_n50.log)


Could you share the full script with the loop? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.