cannot load meta tensor issue when loading llama model saved to s3, Cannot copy out of meta tensor; no data
Author: hugocool
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-4.14.301-224.520.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.10
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

steps:

1. download the model and tokenizer and instantiate pipeline
2. save the model to s3 using `save_pretrained` & cloudpathlib
3. load the model on aws batch using `from_pretrained`, this results in an error

# 1:
```
from transformers import AutoTokenizer,AutoModelForCausalLM,pipeline
import torch

model_name = "meta-llama/Llama-2-70b-chat-hf"
token = 'my_token'

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    use_auth_token=token,
    torch_dtype=torch.float16,
    device_map="auto",
    )

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    use_auth_token=token,
    )

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)
```
# 2, this is performed locally (so a g4dn.2xlarge  EC2 instance)
```
from tempfile import TemporaryDirectory
from cloudpathlib import CloudPath

path = 's3://my-project/data/06_models/llama-70b-chat-hf'
cloud_path = CloudPath(path)

with TemporaryDirectory() as tmp_dir:
    pipe.save_pretrained(tmp_dir)
    cloud_path.upload_from(tmp_dir)
```

# 3, this is performed on a p4d.24xlarge:
```
from tempfile import TemporaryDirectory
from cloudpathlib import CloudPath

path = 's3://my-project/data/06_models/llama-70b-chat-hf'
cloud_path = CloudPath(path)

with TemporaryDirectory() as tmp_dir:
    cloud_path.download_to(tmp_dir)

    model = AutoModelForCausalLM.from_pretrained(
    tmp_dir,
    torch_dtype=torch.float16,
    device_map="auto",
    )

    tokenizer = AutoTokenizer.from_pretrained(
        tmp_dir,
        )

    pipe = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
    )
```

this results in:
```
While copying the parameter named "lm_head.weight", whose dimensions in the model are torch.Size([5120]) and whose dimensions in the checkpoint are torch.Size([5120]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
```
For all the layers in the model. ( so a huge traceback)

I have tried different parameter combinations, for example leaving the device_map out (which resulted in OOM), or different torch_dtype, or setting safetensors, using the cache_dir in from_pretrained instead of explicitly save_pretrained. 
I dont want to waste expensive GPU hours on loading a 150gb model, and the downloading unfortunately times out a lot.
It happens for all the model sizes, 7b, 13b and 70b.
Any ideas why saving and subsequently loading Llama-2 from s3 causes such issues?

### Expected behavior

it should just load and work.

Comments:
did you have any luck so far loading models from s3 via transformers? Devs have been clear they are not interested into supporting any other platforms than the hf hub. https://github.com/huggingface/transformers/issues/19834#issuecomment-1289079562

Yeah, the code I shared works for any HF model. I’ve been using it for quite some time.
Kedro will also release a dataset that will help you save and load to and from any cloud storage provider (I made a custom kedro dataset based on the above code that also works for any cloud provider, not just s3).
You can also use mlflow’s transformer flavor, or kedro-mlflow, this will also support saving to any s3-compatible storage.
Obviously this is not in huggingface’s interest to support since it conflicts with their business model, but for LLM using outside networks for saving and loading makes no sense.

Anyway, yes you can use the above code for any transformer model (you can also use kedro’s pickledataset), but it won’t work LlaMa (also not the pickle dataset).


___

Hugo Evers
On 27 Aug 2023 at 12:09 +0200, chris-aeviator ***@***.***>, wrote:
> did you have any luck so far loading models from s3 via transformers? Devs have been clear they are not interested into supporting any other platforms than the hf hub. #19834 (comment)
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.Message ID: ***@***.***>


I'll be working on supporting a more production friendly version of TGI (loading models from s3, https,.., hot-swapping LORA adapters, back to truly OS licensing) via https://github.com/ohmytofu-ai/tgi-angryface , might have some overlaps with transformers  then :)

Thanks for sharing this!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale, this should be resolved.

___

Hugo Evers
On 25 Sep 2023 at 10:02 +0200, github-actions[bot] ***@***.***>, wrote:
> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> Please note that issues that do not follow the contributing guidelines are likely to be ignored.
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.Message ID: ***@***.***>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.