Model cache does not work
Author: bitcoinvsalts
Pull request: no

### System Info

I am ubuntu, torch = "2.0.0"

The following code always re-download the models instead of re-using the cached files:

`        adapter = T2IAdapter.from_pretrained("TencentARC/t2i-adapter-canny-sdxl-1.0",
                                            torch_dtype=torch.float16, 
                                            varient="fp16").to(accelerator.device)

        model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
        logger.info(f"model loading..")
        euler_a = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
        logger.info(f"model loading...")
        vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

        logger.info(f"model loading....")
        pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
            model_id,
            vae=vae,
            adapter=adapter,
            scheduler=euler_a,
            torch_dtype=torch.float16,
            variant="fp16"
        ).to(accelerator.device)
        pipe.enable_xformers_memory_efficient_attention()
        logger.info(f"model weights loading")
        pipe.load_lora_weights(
            "stabilityai/stable-diffusion-xl-base-1.0", weight_name="sd_xl_offset_example-lora_1.0.safetensors"
        )`
      
I also tried with the `cache_dir` param but same result.  
        
<img width="1377" alt="Screenshot 2024-02-21 at 11 32 22 AM" src="https://github.com/huggingface/transformers/assets/13439477/562c4f1e-b033-4c03-883b-9dcf3b6a579a">

        
        

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`        adapter = T2IAdapter.from_pretrained("TencentARC/t2i-adapter-canny-sdxl-1.0",
                                            torch_dtype=torch.float16, 
                                            varient="fp16").to(accelerator.device)

        model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
        logger.info(f"model loading..")
        euler_a = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
        logger.info(f"model loading...")
        vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

        logger.info(f"model loading....")
        pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
            model_id,
            vae=vae,
            adapter=adapter,
            scheduler=euler_a,
            torch_dtype=torch.float16,
            variant="fp16"
        ).to(accelerator.device)
        pipe.enable_xformers_memory_efficient_attention()
        logger.info(f"model weights loading")
        pipe.load_lora_weights(
            "stabilityai/stable-diffusion-xl-base-1.0", weight_name="sd_xl_offset_example-lora_1.0.safetensors"
        )`

### Expected behavior

use the cached models/files and not download any files from the internet

Comments:
Hi @bitcoinvsalts, thanks for raising this issue! 

Could you isolate the lines which trigger the redownloads - giving us the minimal amount of code to reproduce will help us figure out what's happening. 

A few questions / comments: 
* Can you share your running environment: run `transformers-cli env` in the terminal and copy-paste the output
* Where are you running the code? Is it locally or remote? 
* Are you setting any global params for the cache dir e.g. `HF_HOME`? 


If you run the following in a python session, what do you get as output? 

```py
In [1]: from transformers.utils.hub import HF_MODULES_CACHE

In [2]: HF_MODULES_CACHE
```

Hi @amyeroberts,

Thanks for your reply.

The line that triggers the redownload is this one:
`pipe = StableDiffusionXLAdapterPipeline.from_pretrained(`

`- `transformers` version: 4.31.0
- Platform: Linux-6.2.0-1018-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>`

`>>> HF_MODULES_CACHE
'/home/ubuntu/.cache/huggingface/modules'`

More context about this weird issue: 
on AWS, using a g4dn.xlarge ec2 instance, I am running `from_pretrained` and it will download the model files in `cache/huggingface/hub` 
Next I create an AMI based on this instance and I start a new instance based on this AMI. 
When I run `from_pretrained` again, it will re-download the model files again, even so they already exist in the hf hub folder.

If you create an AMI and start an new instance with it, I'm not sure if folders like `.cache` will be copied along. 

You can easily check this by just running `ls /home/ubuntu/.cache/` in the command line before running any python command. 

If you run `from_pretrained` again on the new instance, does it redownload again? 

yes .cache exists with all correct model files in it at` /home/ubuntu/.cache/`

> If you run from_pretrained again on the new instance, does it redownload again?

Yes, that's my weird issue.

@bitcoinvsalts OK, that's weird. I suggest opening an issue in the diffusers repo and linking to this one, as the problem seems to be coming from the SD pipeline, which isn't handled by transformers 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.