[LLAVA-NEXT] Warning: The model weights are not tied. Please use the tie_weightsmethod before using the infer_auto_device function.
Author: aliencaocao
Pull request: no

### System Info

- `transformers` version: 4.40.0.dev0
- Platform: Linux-5.19.0-051900rc6-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.21.1
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0.dev20240326+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import LlavaNextForConditionalGeneration
model = LlavaNextForConditionalGeneration.from_pretrained('panoyo9829/llava-v1.6-mistral-7b-bnb-4bit', low_cpu_mem_usage=True, device_map='auto')
```
Running this will yield a ```The model weights are not tied. Please use the tie_weightsmethod before using theinfer_auto_device function.``` warning. Inference result seems correct though, but I'm not sure if the warning should still be paid attention to.

### Expected behavior

No warning

Comments:
Hi @aliencaocao, thanks for opening this issue! 

I'm unable to replicate this error when running the snippet on `main`. 

The checkpoint in the example [panoyo9829/llava-v1.6-mistral-7b-bnb-4bit](https://huggingface.co/panoyo9829/llava-v1.6-mistral-7b-bnb-4bit/) doesn't appear to be one compatible with a model in the transformers library.

It maps to [LlavaMistralForCausalLM](https://huggingface.co/panoyo9829/llava-v1.6-mistral-7b-bnb-4bit/blob/6c4d511c497a754bca70ecbf7051015bc80271bf/config.json#L4) which isn't defined in transformers.  

When running the snippet, the warning I get is about weights in the model being randomly initialized. 



I can reproduce this with the original untouched fp16 model as long as I use device_map:
```python
from transformers import LlavaNextForConditionalGeneration
model = LlavaNextForConditionalGeneration.from_pretrained('llava-hf/llava-v1.6-mistral-7b-hf', low_cpu_mem_usage=True, device_map='auto')
```

![image](https://github.com/huggingface/transformers/assets/20109683/9e6c2dfe-34f4-4675-9996-d2d8ca50ec53)
I can repro in fresh colab using latest main branch too: https://colab.research.google.com/drive/1pKm8yQ2JBfeQmFQbXgFNoozfSAckEgJu?usp=sharing

Note the notebook cannot be executed as it OOM RAM when loading fp16 weights. Notheless, the warning still came out:
![image](https://github.com/huggingface/transformers/assets/20109683/4be28e51-091d-4a10-ac63-2a76488b25c1)
It might be better for `accelerate` to resolve this issue though, since warning come from it. Feel free to close this if you think it is not a `transformers` issue.

Thanks for sharing the details. 

The error indicates that the [check_tied_parameters_in_config(model)](https://github.com/huggingface/accelerate/blob/7531e8c13ea42be2f1f4f8b27cecd28810a05d52/src/accelerate/utils/modeling.py#L1597C8-L1597C46) for the model is evaluating as True. 

@muellerzr Any suggestion on how to handle this? The warning indicates calling `model.tie_weights()`, but it's not obvious how this is supposed to be done if we're loading with `from_pretrained` and `device_map="auto"`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

issue still there

@SunMarc if you have a second to check this :)

Hi @aliencaocao, the above PR should fix this. This happens because a config attribute was not set correctly but there was no impact apart from triggering a warning message. 
@amyeroberts We shouldn't get this warning at all since in `from_pretained()`, we make sure to call `tie_weights` before `infer_auto_device_map`. In this case, the issue was that the `tie_word_embedding` was incorrectly set in the config of the model. 

> there was no impact apart from triggering a warning message.

Glad to have that confirmation.

Thanks for the fix! Would not have noticed this as I also checked the source code and saw the `tie_weights` call too.